How to Build a Local RAG Pipeline with Open Source Tools (Without Losing a Weekend)

I broke my first local RAG pipeline before I even got to the fun part — the retrieval was returning garbage chunks and I spent two hours blaming the embedding model when the real problem was my chunking script silently truncating documents at 512 characters. So this guide is the version of that weekend I wish I’d had: a working local RAG pipeline built entirely from open source pieces, plus the mistakes that actually cost time.

Quick Answer

If you just want the short version before the full walkthrough:

• Use Ollama or llama.cpp to run your LLM locally — no API keys, no cloud bill
• Pick an embedding model that matches your hardware: nomic-embed-text or bge-small for lighter machines, bge-large if you’ve got the VRAM
• Store vectors in Chroma for simplicity or Qdrant if you need filtering and scale
• Chunk documents by semantic boundaries, not fixed character counts — this fixes most bad-retrieval complaints
• Test retrieval quality separately from generation quality before you blame the LLM

Why Local RAG Pipelines Break

Most people assume RAG failures come from a bad LLM. That’s almost never actually the case. From what I’ve seen, the LLM is usually the last thing to fix, not the first.

Chunking strategy mismatches. Fixed-size chunking (say, every 500 tokens) chops sentences and context in half. Your retriever then pulls back fragments that technically match the query but don’t contain the actual answer.

Embedding and query mismatch. If you embed documents with one model and query with another — or even a different version of the “same” model — the vector spaces don’t line up. Retrieval quietly degrades and nothing throws an error to tell you why.

Context window overflow. You retrieve five chunks, stuff them into the prompt, and the model silently truncates the middle of your context. This one’s sneaky because the pipeline runs fine, it just gives worse answers.

Insufficient VRAM or RAM for the stack. Running an embedding model, a vector database, and an LLM at the same time adds up fast, especially on a laptop with 16GB of shared memory.

Stale or duplicated vector indexes. Re-running ingestion scripts without clearing the old collection first — I’ve done this more than once — leaves duplicate vectors in the store, which quietly skews retrieval toward whatever got indexed twice.

Common Scenarios Where This Bites People

On an M1/M2 Mac, the usual complaint is Ollama running fine but the embedding step crawling because it’s falling back to CPU for a model that isn’t Metal-optimized. On a Windows machine with an older NVIDIA card, people hit CUDA version mismatches between PyTorch and their installed drivers — the pipeline installs, then fails silently at inference. On headless Linux servers, it’s almost always a missing system dependency (usually something with libgomp or a missing BLAS library) that only shows up once you try to load a quantized model.

And if you’re running this inside Docker, add networking to the list — Chroma or Qdrant running in a container that your app script can’t actually reach because of a port binding issue.

Technical Comparison: Vector Stores

I didn’t include a fourth option here on purpose — Weaviate and Milvus are fine, but for a fully local single-machine setup they’re often overkill, and adding them just to fill a row would be dishonest.

Step-by-Step: Building the Pipeline

Step 1: Install a Local LLM Runner

Ollama is the least painful entry point right now. Install it, then pull a model:

bash

ollama pull llama3.1:8b

If you’re on a machine without a dedicated GPU, drop down to a smaller quantized model — a 7-8B parameter model at Q4 quantization will run on most modern laptops without melting your fan.

Step 2: Choose and Install an Embedding Model

You don’t need the biggest embedding model available. You need one that matches your document type and your hardware.

bash

ollama pull nomic-embed-text

Or, if you’re working in Python directly with sentence-transformers:

bash

pip install sentence-transformers

python

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('BAAI/bge-small-en-v1.5')

Step 3: Set Up Your Vector Store

For a first build, Chroma gets you running in minutes:

bash

pip install chromadb

python

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")

Note the PersistentClient — using the default in-memory client is the single most common reason people “lose” their index after a restart.

Step 4: Chunk Your Documents Properly

This is the step people rush, and it’s the one that determines whether retrieval actually works. Skip fixed character splitting if you can. Use a recursive or semantic splitter that respects sentence and paragraph boundaries instead.

python

from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_text(document_text)

The overlap matters more than people think — without it, answers that span a chunk boundary get cut cleanly in half.

Step 5: Embed and Store

python

embeddings = model.encode(chunks)
collection.add(
    documents=chunks,
    embeddings=embeddings.tolist(),
    ids=[f"chunk_{i}" for i in range(len(chunks))]
)

Step 6: Build the Retrieval Query

python

query_embedding = model.encode([user_question])
results = collection.query(query_embeddings=query_embedding.tolist(), n_results=4)

Start with 4-5 retrieved chunks. More isn’t automatically better — it just gives the LLM more chances to get lost in context it doesn’t need.

Step 7: Connect Retrieval to Generation

python

context = "\n\n".join(results['documents'][0])
prompt = f"Answer using only this context:\n{context}\n\nQuestion: {user_question}"

Send that to Ollama’s API and you’ve got a working local RAG loop.

What Actually Worked For Me

My first attempt used fixed 500-character chunking because a tutorial I skimmed used it, and the retrieval results were consistently half-answers — technically relevant, never complete. I assumed it was the embedding model, so I swapped from a small model to a much larger one. No real change. Then I tried increasing n_results from 4 to 10, thinking more context would paper over the gaps. That actually made things worse, because the LLM started mixing details from unrelated chunks into single answers.

The fix, honestly, came from a half-remembered comment on a GitHub issue thread about someone’s chunking overlap being set to zero. So I checked mine — also zero — and bumped it to 100 tokens along with switching to a recursive splitter that respected paragraph breaks. That fixed it in about ten minutes, after two evenings of chasing the wrong cause. Not exactly a proud debugging story, but that’s how it went.

Advanced Fixes and Edge Cases

Hybrid search and reranking. Pure vector similarity misses exact keyword matches sometimes — product codes, error strings, specific names. Combining vector search with a BM25 keyword pass, then reranking the merged results with something like a cross-encoder, fixes a class of retrieval failures that embedding tweaks never will.

Quantization and memory pressure. If your pipeline runs fine for a few queries then slows to a crawl, check whether you’re loading the embedding model fresh on every request instead of keeping it resident in memory. This is a common oversight in quick prototype scripts, and it’s an easy one to miss because it doesn’t throw an error — it just gets slower.

Diagnosing bad retrieval separately from bad generation. Before touching your LLM settings, print out the raw retrieved chunks for a failing query. If the chunks themselves don’t contain the answer, no prompt engineering will fix it — that’s a retrieval problem, not a generation problem. This single diagnostic step saves more debugging time than almost anything else on this list.

Prevention Tips

• Persist your vector store from day one, don’t rely on in-memory clients “just for testing”
• Clear or version your collections before re-running ingestion scripts
• Pin your embedding model version somewhere visible — a comment in code is fine, a config file is better
• Log retrieved chunks during development, not just final answers
• Test on a small, known document set first so you actually know what “correct” retrieval looks like

FAQ

Do I need a GPU to run a local RAG pipeline? No, but it helps a lot. CPU-only setups work fine for smaller models and lower query volume — just expect slower response times.

Can I use PDFs directly or do I need to convert them first? You can load PDFs directly with libraries like pypdf or unstructured, but scanned PDFs need OCR first or you’ll get empty or garbled text going into your chunker.

Why does my pipeline give different answers to the same question? Usually temperature settings on the LLM, not the retrieval step. Set temperature to 0 or close to it if you want consistent, repeatable answers for testing.

Is Chroma good enough for production use? For small to mid-size local apps, yes. For anything with heavy concurrent access or complex filtering needs, look at Qdrant instead.

My retrieval works but answers are still wrong — what’s left? Check your prompt template. A surprising number of “bad answer” complaints are actually the LLM ignoring the retrieved context because the prompt doesn’t make it clear the context should take priority over its own training knowledge.

Editor’s Opinion

honestly this stack is way more approachable than it looks from the outside, the hard part isnt the LLM, its the boring plumbing around it — chunking, overlap, persistence. most people give up right around the chunking step because it seems too simple to matter. it matters more than the model choice, in my experience anyway. your mileage may vary depending on your documents, but start there before you touch anything else.