AI Integration for Backend Engineers: A Practical Playbook

Backend engineers are being asked to “add AI” to everything now. And most of the content out there is either too academic (ML research papers) or too simple (call the API and print the result). Neither helps you build production AI features.

Here is the thing: AI integration is fundamentally a backend engineering problem. You are calling an API, handling responses, managing state, and dealing with failure modes. The API just happens to return natural language instead of structured data. That creates some unique challenges, but your existing skills apply directly.

I have built AI features for content platforms, customer support tools, and data analysis pipelines. This playbook covers the patterns that work in production. This builds on my experience mentoring engineers on AI integration and from actual production deployments.

AI Integration is Not AI Research

Let me be clear about what backend engineers need to know vs what they can ignore:

You need to know:

• How to call LLM APIs efficiently (rate limiting, retries, streaming)
• How to manage prompts (versioning, testing, evaluation)
• How to cache responses (cost reduction, latency improvement)
• How to validate outputs (the model can return garbage)
• How to handle costs (model selection, token optimization)

You do NOT need to know:

• How transformers work internally
• Training or fine-tuning processes
• Mathematical foundations of embeddings
• Neural network architectures

If you can build a reliable REST API integration with retry logic and caching, you can build production AI features.

Architecture Patterns

Pattern 1: Synchronous (Request-Response)

The simplest pattern. User sends request, backend calls LLM, returns result:

User → API → LLM API → Response → User
       (200-2000ms round trip)

Use when:

• Response time under 3 seconds is acceptable
• Result is needed immediately
• Low to moderate volume (under 100 concurrent requests)

Implementation:

func (h *Handler) SummarizeText(w http.ResponseWriter, r *http.Request) {
    var req SummarizeRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        writeError(w, http.StatusBadRequest, "invalid request")
        return
    }

    ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
    defer cancel()

    summary, err := h.aiService.Summarize(ctx, req.Text)
    if err != nil {
        slog.Error("summarization failed", "error", err)
        writeError(w, http.StatusInternalServerError, "summarization failed")
        return
    }

    writeJSON(w, http.StatusOK, map[string]string{"summary": summary})
}

Pattern 2: Asynchronous (Queue-Based)

For long-running or high-volume AI tasks:

User → API → Queue → Worker → LLM API → Result Store
       (returns job ID immediately)

User → API → Poll/Webhook → Get Result

Use when:

• Processing takes more than 5 seconds
• High volume requiring rate limit management
• Results can be consumed later
• Batch processing of multiple items

Implementation:

func (h *Handler) AnalyzeDocument(w http.ResponseWriter, r *http.Request) {
    var req AnalyzeRequest
    json.NewDecoder(r.Body).Decode(&req)

    // Create job and queue it
    jobID := uuid.NewString()
    if err := h.queue.Publish(ctx, "ai-analysis", AnalysisJob{
        ID:       jobID,
        Document: req.DocumentID,
    }); err != nil {
        writeError(w, http.StatusInternalServerError, "failed to queue job")
        return
    }

    writeJSON(w, http.StatusAccepted, map[string]string{
        "job_id": jobID,
        "status": "processing",
    })
}

// Worker processes jobs from queue
func (w *Worker) ProcessAnalysis(ctx context.Context, job AnalysisJob) error {
    result, err := w.aiService.Analyze(ctx, job.Document)
    if err != nil {
        return fmt.Errorf("analysis failed: %w", err)
    }

    return w.resultStore.Save(ctx, job.ID, result)
}

Pattern 3: Streaming

For chat interfaces or progressive content generation:

func (h *Handler) ChatStream(w http.ResponseWriter, r *http.Request) {
    flusher, ok := w.(http.Flusher)
    if !ok {
        writeError(w, http.StatusInternalServerError, "streaming not supported")
        return
    }

    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")

    stream, err := h.aiService.ChatStream(r.Context(), req.Messages)
    if err != nil {
        return
    }

    for chunk := range stream {
        fmt.Fprintf(w, "data: %s\n\n", chunk)
        flusher.Flush()
    }
}

OpenAI API: Rate Limiting and Retry Logic

The OpenAI API has aggressive rate limits. You will hit them. Build defensive code from day one:

type AIClient struct {
    client     *http.Client
    apiKey     string
    limiter    *rate.Limiter
    maxRetries int
}

func NewAIClient(apiKey string, rateLimit float64) *AIClient {
    return &AIClient{
        client:     &http.Client{Timeout: 60 * time.Second},
        apiKey:     apiKey,
        limiter:    rate.NewLimiter(rate.Limit(rateLimit), int(rateLimit)),
        maxRetries: 3,
    }
}

func (c *AIClient) Complete(ctx context.Context, req CompletionRequest) (*CompletionResponse, error) {
    // Wait for rate limiter
    if err := c.limiter.Wait(ctx); err != nil {
        return nil, fmt.Errorf("rate limiter: %w", err)
    }

    var lastErr error
    for attempt := 0; attempt <= c.maxRetries; attempt++ {
        if attempt > 0 {
            // Exponential backoff
            backoff := time.Duration(math.Pow(2, float64(attempt))) * time.Second
            select {
            case <-time.After(backoff):
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }

        resp, err := c.doRequest(ctx, req)
        if err == nil {
            return resp, nil
        }

        lastErr = err

        // Only retry on rate limit or server errors
        if !isRetryable(err) {
            return nil, err
        }
    }

    return nil, fmt.Errorf("max retries exceeded: %w", lastErr)
}

func isRetryable(err error) bool {
    var apiErr *APIError
    if errors.As(err, &apiErr) {
        return apiErr.StatusCode == 429 || apiErr.StatusCode >= 500
    }
    return false
}

Token Budget Management

Each API call has token limits. Manage your token budget:

func (s *AIService) buildPrompt(systemPrompt, userInput string, maxTokens int) string {
    // Estimate tokens (rough: 1 token ≈ 4 chars for English)
    systemTokens := len(systemPrompt) / 4
    inputTokens := len(userInput) / 4
    availableForContext := maxTokens - systemTokens - inputTokens - 500 // Reserve for response

    if availableForContext < 0 {
        // Truncate user input to fit
        maxInputChars := (maxTokens - systemTokens - 500) * 4
        userInput = userInput[:maxInputChars] + "... [truncated]"
    }

    return fmt.Sprintf("%s\n\n%s", systemPrompt, userInput)
}

Prompt Management: Version Control for Prompts

Prompts are code. Treat them accordingly:

// prompts/summarize.go
package prompts

const SummarizeV1 = `You are a technical content summarizer.
Given the following text, produce a concise summary in 2-3 sentences.
Focus on key technical decisions and their rationale.
Do not include opinions or recommendations.

Text to summarize:
{{.Text}}

Summary:`

const SummarizeV2 = `Summarize the following technical content in exactly 3 bullet points.
Each bullet should capture one key insight or decision.
Use plain language accessible to junior engineers.

Content:
{{.Text}}

Bullets:`

Prompt Testing

func TestSummarizePrompt(t *testing.T) {
    // Golden test: known input should produce expected output characteristics
    input := loadTestFixture("technical-article.txt")
    
    result, err := aiService.Summarize(context.Background(), input)
    require.NoError(t, err)

    // Validate output characteristics
    assert.Less(t, len(result), 500, "summary should be concise")
    assert.NotContains(t, result, "I think", "should not contain opinions")
    assert.Greater(t, len(result), 50, "summary should be meaningful")
}

RAG: When and How to Add Retrieval

RAG (Retrieval-Augmented Generation) adds domain knowledge to LLM responses without fine-tuning. The next article in this series covers building a complete RAG system in Go, but here is the high-level decision:

Use RAG when:

• The LLM needs to answer questions about your specific data
• Information changes frequently (product docs, knowledge base)
• You need citations and source attribution
• Fine-tuning is too expensive or slow for your iteration speed

Skip RAG when:

• The LLM already knows the answer (general knowledge)
• Your data fits in the context window (under 100K tokens)
• Accuracy requirements are low (creative writing, brainstorming)

Caching Strategies for LLM Responses

LLM calls are expensive and slow. Cache aggressively:

type CachedAIService struct {
    client *AIClient
    cache  *redis.Client
    ttl    time.Duration
}

func (s *CachedAIService) Complete(ctx context.Context, req CompletionRequest) (*CompletionResponse, error) {
    // Generate cache key from prompt content hash
    cacheKey := s.buildCacheKey(req)

    // Check cache
    cached, err := s.cache.Get(ctx, cacheKey).Bytes()
    if err == nil {
        var resp CompletionResponse
        json.Unmarshal(cached, &resp)
        return &resp, nil
    }

    // Cache miss: call LLM
    resp, err := s.client.Complete(ctx, req)
    if err != nil {
        return nil, err
    }

    // Store in cache
    data, _ := json.Marshal(resp)
    s.cache.Set(ctx, cacheKey, data, s.ttl)

    return resp, nil
}

func (s *CachedAIService) buildCacheKey(req CompletionRequest) string {
    // Hash the full request to create a stable cache key
    h := sha256.New()
    h.Write([]byte(req.Model))
    h.Write([]byte(req.SystemPrompt))
    h.Write([]byte(req.UserMessage))
    return fmt.Sprintf("ai:completion:%x", h.Sum(nil))
}

Cache hit rates depend on your use case. For FAQ-style queries: 60-80% hit rate. For unique user inputs: 10-20% hit rate. Even low hit rates save money at scale.

Error Handling for Non-Deterministic Outputs

LLM responses can be structurally valid but semantically wrong. You need output validation:

type StructuredResponse struct {
    Category   string   `json:"category"`
    Confidence float64  `json:"confidence"`
    Tags       []string `json:"tags"`
}

func (s *AIService) Classify(ctx context.Context, text string) (*StructuredResponse, error) {
    raw, err := s.client.Complete(ctx, classifyPrompt(text))
    if err != nil {
        return nil, fmt.Errorf("llm call failed: %w", err)
    }

    // Parse structured output
    var result StructuredResponse
    if err := json.Unmarshal([]byte(raw.Content), &result); err != nil {
        // LLM returned non-JSON despite instructions
        return nil, fmt.Errorf("failed to parse LLM output as JSON: %w", err)
    }

    // Validate output semantics
    if err := validateClassification(&result); err != nil {
        return nil, fmt.Errorf("invalid classification output: %w", err)
    }

    return &result, nil
}

func validateClassification(r *StructuredResponse) error {
    validCategories := map[string]bool{
        "technical": true, "business": true, "support": true,
    }
    if !validCategories[r.Category] {
        return fmt.Errorf("invalid category: %s", r.Category)
    }
    if r.Confidence < 0 || r.Confidence > 1 {
        return fmt.Errorf("confidence out of range: %f", r.Confidence)
    }
    return nil
}

Retry on Malformed Output

Sometimes the LLM produces valid JSON but misses required fields. Retry with a corrective prompt:

func (s *AIService) ClassifyWithRetry(ctx context.Context, text string) (*StructuredResponse, error) {
    for attempt := 0; attempt < 3; attempt++ {
        result, err := s.Classify(ctx, text)
        if err == nil {
            return result, nil
        }

        // If it is a validation error, retry with corrective feedback
        if strings.Contains(err.Error(), "invalid classification") {
            slog.Warn("retrying classification with corrective prompt", "attempt", attempt)
            continue
        }

        return nil, err
    }
    return nil, fmt.Errorf("classification failed after 3 attempts")
}

Testing AI Features

Golden Dataset Testing

Maintain a set of known input/output pairs:

func TestClassification_GoldenDataset(t *testing.T) {
    golden := []struct {
        input    string
        expected string
    }{
        {"How do I reset my password?", "support"},
        {"What are your pricing tiers?", "business"},
        {"Getting 500 error on the API", "technical"},
    }

    for _, tc := range golden {
        result, err := aiService.Classify(context.Background(), tc.input)
        require.NoError(t, err)
        assert.Equal(t, tc.expected, result.Category)
    }
}

Evaluation Metrics

Track these over time:

• Accuracy: Percentage of correct classifications vs golden dataset
• Latency P95: Response time including LLM call
• Cost per request: Tokens consumed per classification
• Cache hit rate: Percentage of requests served from cache
• Retry rate: How often the first attempt fails validation

Cost Optimization

Model Selection by Task Complexity

Not every request needs GPT-4o:

func (s *AIService) selectModel(task TaskType) string {
    switch task {
    case TaskClassification, TaskExtraction, TaskSummarize:
        return "gpt-4o-mini"  // Simple tasks: cheaper model
    case TaskAnalysis, TaskGeneration, TaskReasoning:
        return "gpt-4o"       // Complex tasks: powerful model
    default:
        return "gpt-4o-mini"
    }
}

Token Management

// Trim context to minimize token usage
func trimContext(docs []string, maxTokens int) []string {
    var result []string
    totalTokens := 0

    for _, doc := range docs {
        docTokens := estimateTokens(doc)
        if totalTokens+docTokens > maxTokens {
            break
        }
        result = append(result, doc)
        totalTokens += docTokens
    }

    return result
}

func estimateTokens(text string) int {
    // Rough estimate: 1 token per 4 characters for English
    return len(text) / 4
}

Production Checklist for AI Features

Before shipping an AI feature:

• Rate limiting configured for LLM API
• Retry logic with exponential backoff
• Output validation beyond HTTP status codes
• Caching layer for repeated queries
• Cost monitoring and alerts for token usage
• Timeout configuration (LLM calls can be slow)
• Fallback behavior when LLM is unavailable
• Prompt versioning in source control
• Golden dataset tests passing
• Structured logging for debugging
• Model selection appropriate for task complexity
• User-facing error messages that do not expose LLM internals

Wrapping Up

AI integration is a backend engineering discipline. The same principles that make reliable APIs, idempotent operations, retry logic, caching, and monitoring, apply directly to LLM integration.

Start simple: one synchronous endpoint, one model, one prompt. Get it working in production. Then optimize with caching, async processing, and model routing.

The engineers who will excel at AI integration are not the ones with ML PhDs. They are the ones who understand distributed systems, failure modes, and cost optimization. That is backend engineering.

Need help building AI features into your product? I build custom applications with AI integration for teams that want production-ready AI without hiring a dedicated ML team. Let us talk about what is possible for your use case.

See AI integration applied end-to-end in my MFunnel AI SaaS project and AI agent portfolio work.