Building a RAG System with Go and OpenAI: Step by Step

RAG (Retrieval-Augmented Generation) is the most practical AI pattern for backend engineers. You do not need ML expertise, custom model training, or expensive GPU infrastructure. You need a database, an embedding API, and good chunking logic.

The concept is simple: instead of hoping the LLM knows about your specific data, you retrieve relevant documents and inject them into the prompt. The model answers based on your actual data, not its training knowledge. Hallucinations drop dramatically because the answer is right there in the context.

This tutorial builds a complete RAG system in Go. We will use OpenAI for embeddings and generation, pgvector for vector storage, and PostgreSQL as our only database. The same patterns from my production Go API guide apply here.

What RAG Solves

Without RAG, asking an LLM about your company’s products, internal docs, or domain knowledge gives you:

• Hallucinated answers that sound confident but are wrong
• Generic responses that do not reflect your specific context
• Outdated information from the model’s training cutoff

With RAG:

• Answers are grounded in your actual documents
• You can cite sources (which chunk provided the answer)
• Knowledge updates when you add new documents, no retraining needed
• Costs a fraction of fine-tuning

Architecture Overview

The RAG pipeline has two phases:

Ingestion Phase (Offline)

Documents → Chunking → Embedding → Vector Storage
   │            │           │            │
   ▼            ▼           ▼            ▼
  PDF/MD    500-800 tok   OpenAI API   pgvector

Query Phase (Online)

User Query → Embed Query → Similarity Search → Top-K Chunks → LLM Generation
     │            │              │                   │              │
     ▼            ▼              ▼                   ▼              ▼
  "How do     vector[1536]    cosine dist      3-5 chunks     Answer with
   I deploy                   in pgvector      as context     context
   to ECS?"

Setting Up pgvector

First, add the pgvector extension to your PostgreSQL database:

-- Enable the extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create the documents table
CREATE TABLE documents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    title TEXT NOT NULL,
    source_url TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Create the chunks table with vector column
CREATE TABLE chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    content TEXT NOT NULL,
    chunk_index INTEGER NOT NULL,
    token_count INTEGER NOT NULL,
    embedding vector(1536),  -- OpenAI text-embedding-3-small dimension
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Create index for fast similarity search
CREATE INDEX ON chunks USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

-- Index for filtering by document
CREATE INDEX ON chunks (document_id);

The vector(1536) column stores OpenAI embeddings. The ivfflat index enables fast approximate nearest neighbor search. For collections under 100K vectors, exact search is fine too:

-- For smaller collections, exact search works well
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Document Ingestion Pipeline

Chunking Strategy

Chunking is the most important decision in RAG quality. Bad chunks mean bad retrieval means bad answers.

package rag

import (
    "strings"
    "unicode"
)

type ChunkConfig struct {
    MaxTokens    int // Target chunk size in tokens
    OverlapTokens int // Overlap between chunks
    MinTokens    int // Minimum chunk size (avoid tiny fragments)
}

func DefaultChunkConfig() ChunkConfig {
    return ChunkConfig{
        MaxTokens:    600,
        OverlapTokens: 100,
        MinTokens:    50,
    }
}

type Chunk struct {
    Content    string
    Index      int
    TokenCount int
}

func ChunkDocument(text string, cfg ChunkConfig) []Chunk {
    // Split by paragraphs first (natural boundaries)
    paragraphs := splitParagraphs(text)
    
    var chunks []Chunk
    var currentChunk strings.Builder
    currentTokens := 0
    chunkIndex := 0

    for _, para := range paragraphs {
        paraTokens := estimateTokens(para)

        // If single paragraph exceeds max, split it further
        if paraTokens > cfg.MaxTokens {
            // Flush current chunk if not empty
            if currentTokens > cfg.MinTokens {
                chunks = append(chunks, Chunk{
                    Content:    currentChunk.String(),
                    Index:      chunkIndex,
                    TokenCount: currentTokens,
                })
                chunkIndex++
                currentChunk.Reset()
                currentTokens = 0
            }

            // Split large paragraph by sentences
            sentenceChunks := splitBySentences(para, cfg)
            for _, sc := range sentenceChunks {
                chunks = append(chunks, Chunk{
                    Content:    sc,
                    Index:      chunkIndex,
                    TokenCount: estimateTokens(sc),
                })
                chunkIndex++
            }
            continue
        }

        // Check if adding this paragraph exceeds limit
        if currentTokens+paraTokens > cfg.MaxTokens {
            // Save current chunk
            if currentTokens > cfg.MinTokens {
                chunks = append(chunks, Chunk{
                    Content:    currentChunk.String(),
                    Index:      chunkIndex,
                    TokenCount: currentTokens,
                })
                chunkIndex++
            }

            // Start new chunk with overlap
            overlap := getOverlap(currentChunk.String(), cfg.OverlapTokens)
            currentChunk.Reset()
            currentChunk.WriteString(overlap)
            currentTokens = estimateTokens(overlap)
        }

        currentChunk.WriteString(para)
        currentChunk.WriteString("\n\n")
        currentTokens += paraTokens
    }

    // Flush final chunk
    if currentTokens > cfg.MinTokens {
        chunks = append(chunks, Chunk{
            Content:    currentChunk.String(),
            Index:      chunkIndex,
            TokenCount: currentTokens,
        })
    }

    return chunks
}

func splitParagraphs(text string) []string {
    return strings.Split(text, "\n\n")
}

func estimateTokens(text string) int {
    // Rough estimate: 1 token ≈ 4 characters for English
    return len(text) / 4
}

func getOverlap(text string, overlapTokens int) string {
    chars := overlapTokens * 4
    if chars >= len(text) {
        return text
    }
    // Find sentence boundary near overlap point
    start := len(text) - chars
    for start < len(text) && !unicode.IsPunct(rune(text[start])) {
        start++
    }
    if start < len(text) {
        start++ // Include the punctuation
    }
    return strings.TrimSpace(text[start:])
}

Embedding Generation with OpenAI

package rag

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "net/http"
)

type EmbeddingClient struct {
    apiKey string
    model  string
    client *http.Client
}

func NewEmbeddingClient(apiKey string) *EmbeddingClient {
    return &EmbeddingClient{
        apiKey: apiKey,
        model:  "text-embedding-3-small", // 1536 dimensions, cheap
        client: &http.Client{},
    }
}

type embeddingRequest struct {
    Input []string `json:"input"`
    Model string   `json:"model"`
}

type embeddingResponse struct {
    Data []struct {
        Embedding []float32 `json:"embedding"`
        Index     int       `json:"index"`
    } `json:"data"`
    Usage struct {
        TotalTokens int `json:"total_tokens"`
    } `json:"usage"`
}

func (c *EmbeddingClient) Embed(ctx context.Context, texts []string) ([][]float32, error) {
    // OpenAI supports batch embedding (up to 2048 inputs)
    reqBody := embeddingRequest{
        Input: texts,
        Model: c.model,
    }

    body, _ := json.Marshal(reqBody)
    req, _ := http.NewRequestWithContext(ctx, "POST",
        "https://api.openai.com/v1/embeddings", bytes.NewReader(body))
    req.Header.Set("Authorization", "Bearer "+c.apiKey)
    req.Header.Set("Content-Type", "application/json")

    resp, err := c.client.Do(req)
    if err != nil {
        return nil, fmt.Errorf("embedding request failed: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return nil, fmt.Errorf("embedding API returned status %d", resp.StatusCode)
    }

    var result embeddingResponse
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return nil, fmt.Errorf("decoding embedding response: %w", err)
    }

    embeddings := make([][]float32, len(result.Data))
    for _, d := range result.Data {
        embeddings[d.Index] = d.Embedding
    }

    return embeddings, nil
}

Storing Vectors in pgvector

package rag

import (
    "context"
    "fmt"

    "github.com/jackc/pgx/v5"
    "github.com/jackc/pgx/v5/pgxpool"
    "github.com/pgvector/pgvector-go"
)

type VectorStore struct {
    pool *pgxpool.Pool
}

func NewVectorStore(pool *pgxpool.Pool) *VectorStore {
    return &VectorStore{pool: pool}
}

func (s *VectorStore) StoreChunks(ctx context.Context, docID string, chunks []Chunk, embeddings [][]float32) error {
    batch := &pgx.Batch{}

    for i, chunk := range chunks {
        vec := pgvector.NewVector(embeddings[i])
        batch.Queue(
            `INSERT INTO chunks (document_id, content, chunk_index, token_count, embedding)
             VALUES ($1, $2, $3, $4, $5)`,
            docID, chunk.Content, chunk.Index, chunk.TokenCount, vec,
        )
    }

    br := s.pool.SendBatch(ctx, batch)
    defer br.Close()

    for i := 0; i < len(chunks); i++ {
        if _, err := br.Exec(); err != nil {
            return fmt.Errorf("storing chunk %d: %w", i, err)
        }
    }

    return nil
}

Full Ingestion Pipeline

func (s *RAGService) IngestDocument(ctx context.Context, title, content, sourceURL string) error {
    // 1. Create document record
    docID, err := s.store.CreateDocument(ctx, title, sourceURL)
    if err != nil {
        return fmt.Errorf("creating document: %w", err)
    }

    // 2. Chunk the document
    chunks := ChunkDocument(content, DefaultChunkConfig())
    if len(chunks) == 0 {
        return fmt.Errorf("document produced no chunks")
    }

    // 3. Generate embeddings (batch for efficiency)
    texts := make([]string, len(chunks))
    for i, c := range chunks {
        texts[i] = c.Content
    }

    embeddings, err := s.embedder.Embed(ctx, texts)
    if err != nil {
        return fmt.Errorf("generating embeddings: %w", err)
    }

    // 4. Store chunks with embeddings
    if err := s.vectorStore.StoreChunks(ctx, docID, chunks, embeddings); err != nil {
        return fmt.Errorf("storing chunks: %w", err)
    }

    slog.Info("document ingested",
        "doc_id", docID,
        "title", title,
        "chunks", len(chunks),
    )

    return nil
}

Retrieval: Similarity Search

When a user asks a question, we embed their query and find the most similar chunks:

func (s *VectorStore) Search(ctx context.Context, queryEmbedding []float32, topK int) ([]SearchResult, error) {
    vec := pgvector.NewVector(queryEmbedding)

    rows, err := s.pool.Query(ctx,
        `SELECT c.id, c.content, c.document_id, d.title,
                1 - (c.embedding <=> $1) AS similarity
         FROM chunks c
         JOIN documents d ON c.document_id = d.id
         WHERE 1 - (c.embedding <=> $1) > 0.7  -- Minimum similarity threshold
         ORDER BY c.embedding <=> $1
         LIMIT $2`,
        vec, topK,
    )
    if err != nil {
        return nil, fmt.Errorf("similarity search: %w", err)
    }
    defer rows.Close()

    var results []SearchResult
    for rows.Next() {
        var r SearchResult
        if err := rows.Scan(&r.ChunkID, &r.Content, &r.DocumentID, &r.DocumentTitle, &r.Similarity); err != nil {
            return nil, fmt.Errorf("scanning result: %w", err)
        }
        results = append(results, r)
    }

    return results, nil
}

type SearchResult struct {
    ChunkID       string
    Content       string
    DocumentID    string
    DocumentTitle string
    Similarity    float64
}

The <=> operator computes cosine distance. 1 - cosine_distance = cosine_similarity. We filter results below 0.7 similarity to avoid injecting irrelevant context.

Reranking for Better Precision

Simple similarity search returns the top-K closest vectors. But closeness in embedding space does not always mean relevance. A reranking step improves precision:

func (s *RAGService) SearchWithRerank(ctx context.Context, query string, topK int) ([]SearchResult, error) {
    // 1. Embed query
    queryEmb, err := s.embedder.Embed(ctx, []string{query})
    if err != nil {
        return nil, err
    }

    // 2. Retrieve more candidates than needed
    candidates, err := s.vectorStore.Search(ctx, queryEmb[0], topK*3)
    if err != nil {
        return nil, err
    }

    // 3. Rerank using keyword overlap and position
    scored := make([]scoredResult, len(candidates))
    queryTerms := extractKeyTerms(query)

    for i, c := range candidates {
        keywordScore := computeKeywordOverlap(queryTerms, c.Content)
        scored[i] = scoredResult{
            Result:       c,
            FinalScore:   c.Similarity*0.7 + keywordScore*0.3,
        }
    }

    // 4. Sort by final score and return top-K
    sort.Slice(scored, func(i, j int) bool {
        return scored[i].FinalScore > scored[j].FinalScore
    })

    results := make([]SearchResult, min(topK, len(scored)))
    for i := range results {
        results[i] = scored[i].Result
    }

    return results, nil
}

Generation: Context Injection

The final step: inject retrieved chunks into the LLM prompt as context:

func (s *RAGService) Answer(ctx context.Context, query string) (*RAGResponse, error) {
    // 1. Retrieve relevant chunks
    chunks, err := s.SearchWithRerank(ctx, query, 5)
    if err != nil {
        return nil, fmt.Errorf("retrieval failed: %w", err)
    }

    if len(chunks) == 0 {
        return &RAGResponse{
            Answer:  "I could not find relevant information to answer your question.",
            Sources: nil,
        }, nil
    }

    // 2. Build context from retrieved chunks
    var context strings.Builder
    for i, chunk := range chunks {
        context.WriteString(fmt.Sprintf("--- Source %d: %s ---\n", i+1, chunk.DocumentTitle))
        context.WriteString(chunk.Content)
        context.WriteString("\n\n")
    }

    // 3. Generate answer with context
    prompt := fmt.Sprintf(`You are a helpful assistant. Answer the user's question based ONLY on the provided context.
If the context does not contain enough information to answer, say so clearly.
Always cite which source number you used for each claim.

Context:
%s

Question: %s

Answer:`, context.String(), query)

    answer, err := s.llm.Complete(ctx, CompletionRequest{
        Model:       "gpt-4o",
        Messages:    []Message{{Role: "user", Content: prompt}},
        MaxTokens:   1000,
        Temperature: 0.1, // Low temperature for factual responses
    })
    if err != nil {
        return nil, fmt.Errorf("generation failed: %w", err)
    }

    // 4. Build response with source attribution
    sources := make([]Source, len(chunks))
    for i, c := range chunks {
        sources[i] = Source{
            Title:      c.DocumentTitle,
            DocumentID: c.DocumentID,
            Similarity: c.Similarity,
        }
    }

    return &RAGResponse{
        Answer:  answer.Content,
        Sources: sources,
    }, nil
}

Full HTTP Handler

Putting it all together as an API endpoint:

func (h *Handler) AskQuestion(w http.ResponseWriter, r *http.Request) {
    var req struct {
        Question string `json:"question"`
    }
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        writeError(w, http.StatusBadRequest, "invalid request body")
        return
    }

    if len(req.Question) < 10 {
        writeError(w, http.StatusBadRequest, "question too short")
        return
    }

    ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
    defer cancel()

    response, err := h.ragService.Answer(ctx, req.Question)
    if err != nil {
        slog.Error("RAG query failed", "error", err, "question", req.Question)
        writeError(w, http.StatusInternalServerError, "failed to generate answer")
        return
    }

    writeJSON(w, http.StatusOK, response)
}

Evaluating RAG Quality

You need metrics to know if your RAG system works:

Retrieval Quality

func evaluateRetrieval(testCases []TestCase, ragService *RAGService) EvalResult {
    var totalPrecision, totalRecall float64

    for _, tc := range testCases {
        results, _ := ragService.SearchWithRerank(context.Background(), tc.Query, 5)

        // Check if expected document appears in top-K
        found := false
        for _, r := range results {
            if r.DocumentID == tc.ExpectedDocID {
                found = true
                break
            }
        }

        if found {
            totalRecall++
        }
    }

    return EvalResult{
        Recall: totalRecall / float64(len(testCases)),
    }
}

End-to-End Quality

For the full pipeline, measure:

• Answer relevance: Does the answer address the question?
• Faithfulness: Is the answer supported by the retrieved context?
• Citation accuracy: Do cited sources actually contain the claimed information?

A simple evaluation approach:

type EvalCase struct {
    Question       string
    ExpectedAnswer string // Key phrases that should appear
    ExpectedSource string // Document that should be cited
}

func runEvaluation(cases []EvalCase, rag *RAGService) {
    for _, tc := range cases {
        resp, _ := rag.Answer(context.Background(), tc.Question)

        // Check key phrases
        for _, phrase := range strings.Split(tc.ExpectedAnswer, "|") {
            if !strings.Contains(strings.ToLower(resp.Answer), strings.ToLower(phrase)) {
                log.Printf("MISS: question=%q missing phrase=%q", tc.Question, phrase)
            }
        }

        // Check source attribution
        sourceFound := false
        for _, s := range resp.Sources {
            if strings.Contains(s.Title, tc.ExpectedSource) {
                sourceFound = true
            }
        }
        if !sourceFound {
            log.Printf("SOURCE MISS: question=%q expected source=%q", tc.Question, tc.ExpectedSource)
        }
    }
}

Performance Optimization

Batch Embedding on Ingest

Embed multiple chunks in a single API call. OpenAI supports up to 2048 inputs per request:

// Instead of N API calls for N chunks:
for _, chunk := range chunks {
    embedding, _ := embedder.Embed(ctx, []string{chunk.Content})
}

// Make 1 API call for all chunks:
texts := make([]string, len(chunks))
for i, c := range chunks {
    texts[i] = c.Content
}
embeddings, _ := embedder.Embed(ctx, texts) // Single call

Query Embedding Cache

Users often ask similar questions. Cache query embeddings:

func (s *RAGService) getQueryEmbedding(ctx context.Context, query string) ([]float32, error) {
    cacheKey := fmt.Sprintf("emb:%x", sha256.Sum256([]byte(query)))

    // Check cache
    cached, err := s.cache.Get(ctx, cacheKey).Bytes()
    if err == nil {
        var emb []float32
        json.Unmarshal(cached, &emb)
        return emb, nil
    }

    // Generate and cache
    embeddings, err := s.embedder.Embed(ctx, []string{query})
    if err != nil {
        return nil, err
    }

    data, _ := json.Marshal(embeddings[0])
    s.cache.Set(ctx, cacheKey, data, 24*time.Hour)

    return embeddings[0], nil
}

Index Tuning for pgvector

For large collections, tune the IVFFlat index:

-- Lists = sqrt(total_vectors) is a good starting point
-- For 1M vectors: lists = 1000
CREATE INDEX ON chunks USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 1000);

-- Set probes at query time (higher = more accurate, slower)
SET ivfflat.probes = 10;  -- Check 10 out of 1000 lists

Wrapping Up

RAG is the most cost-effective way to give LLMs domain knowledge. No fine-tuning, no GPU servers, no ML expertise required. Just good chunking, a vector database, and careful prompt engineering.

The patterns in this tutorial scale from a small knowledge base to millions of documents. Start with a few hundred documents, validate retrieval quality, then scale. The same code works at both scales because pgvector and PostgreSQL handle the heavy lifting.

If you are building on the patterns from my AI integration playbook and want a complete production system, these components fit together naturally with the Go API architecture I recommend.

Need help building a RAG system or other AI features for your product? I offer custom web application development with AI integration, from architecture design to production deployment.

See RAG and AI integration patterns applied in production in my MFunnel project and AI agent work.