If you’ve been looking for a way to turn Obsidian into a private, AI-assisted knowledge base — one that doesn’t send your notes to a third-party server — setting up a local LLM alongside it is the way to do it. I spent a few evenings getting this working, and there were more rough edges than I expected.
What You’re Actually Building
Before anything else, let’s be clear about what this setup is and isn’t.
You’re not building a magic “talk to your notes” product. You’re setting up a pipeline where a local language model (running via Ollama, LM Studio, or similar) can read your Obsidian vault — either through a plugin that sends context to the model, or through a retrieval-augmented generation (RAG) layer that chunks and indexes your markdown files and serves relevant ones as context before a query.
The end result: you can ask questions about your notes, get summaries, generate new entries based on existing ones, and build a searchable AI layer over your personal knowledge base — all offline, all private.
Why This Setup Fails (And What’s Actually Causing It)
This is where most tutorials skip the hard part.
The Model Can’t Actually Read Your Vault
The most common misconception is that running a local LLM “connected to Obsidian” means the model can see all your notes. It can’t, not by default. Language models don’t browse directories. What you need is a way to pull relevant note content into the model’s context window before it answers. Without that layer, you’re just running a chatbot inside Obsidian with zero awareness of what’s in your vault.
Context Window Limits Kill Naive Approaches
If you try to dump your entire vault into the prompt — and yes, people try this — you’ll hit token limits almost immediately on any model that runs locally without a monster GPU. Even models with 128k context windows will either truncate, slow to a crawl, or hallucinate from context overload. The fix is chunking and retrieval, not brute force.
Plugin + Model Version Mismatches
Some Obsidian plugins that bridge to local LLMs (Smart Connections, Copilot, Text Generator) have specific expectations about which API format the model server exposes. Ollama uses an OpenAI-compatible API, but the exact endpoint paths differ. If a plugin is pointed at localhost:11434 but expecting /v1/chat/completions and the model is serving at /api/chat, it just silently fails. No error. Nothing. That took me an embarrassing amount of time to debug.
Embedding Model vs. Chat Model Confusion
This one trips people up constantly. You need two different models for a proper RAG setup: one for generating embeddings (turning your note chunks into vectors for similarity search) and one for actually responding to your queries. Trying to use your chat model for embeddings doesn’t work — well, sort of — it’s actually more like trying to use a screwdriver as a ruler. Technically possible to fudge, but the results are garbage.
What You’ll Need
- Obsidian (any recent version)
- Ollama — easiest way to run local models on Mac, Windows, and Linux
- A capable embedding model —
nomic-embed-textis a solid choice, small and fast - A chat model —
llama3,mistral, orphi3depending on your hardware - Either Smart Connections plugin (for simpler setups) or a Python RAG script if you want more control
- Optional: Chroma or Qdrant for a local vector database if you’re going the full RAG route
Step-by-Step Setup
Step 1: Get Ollama Running
Download and install Ollama from their site. Once it’s installed, pull your models:
bash
ollama pull nomic-embed-text
ollama pull llama3Start the server:
bash
ollama serveBy default it listens on http://localhost:11434. Confirm it’s working with:
bash
curl http://localhost:11434/api/tagsIf you get a JSON list of your models back, you’re good.
Step 2: Install the Smart Connections Plugin
In Obsidian, go to Settings → Community Plugins → Browse, search for “Smart Connections,” and install it.
In the plugin settings:
- Set the API type to Ollama
- Enter
http://localhost:11434as the base URL - Set the embedding model to
nomic-embed-text - Set the chat model to whatever you pulled (e.g.
llama3)
One thing to double-check: Smart Connections has an option for “API Version” or endpoint format. If you’re on a newer Ollama version, you may need to point it at the /v1/ path. Check the plugin’s GitHub issues if it’s not connecting — there’s usually a thread about it for whatever Ollama version just released.
Step 3: Index Your Vault
In Smart Connections, there’s a “Rebuild Index” or “Update Embeddings” button. Click it. Depending on vault size, this could take anywhere from 30 seconds to 20 minutes. From what I’ve seen, anything under 1,000 notes is pretty fast even on modest hardware.
This is the step that creates the vector embeddings for your notes. The plugin splits each note into chunks, runs each chunk through nomic-embed-text, and stores the resulting vectors locally (usually in a .smart-env folder inside your vault).
Step 4: Start Querying
Open the Smart Connections panel (ribbon icon or command palette). Type a question. The plugin retrieves the most semantically similar note chunks and sends them to your chat model as context.
If the responses feel disconnected from your actual notes, the retrieval is probably not pulling the right chunks. More on that in the next section.
Going Further: A Manual RAG Pipeline
If you want more control — or if the plugin approach isn’t working reliably — you can build a lightweight Python pipeline yourself.
The basic flow:
- Walk your Obsidian vault and load all
.mdfiles - Split each file into chunks (roughly 512–1024 tokens each)
- Embed each chunk using the Ollama embedding API
- Store embeddings + metadata in a local vector DB (Chroma works well here)
- At query time: embed the query, find the top-k similar chunks, inject them into a prompt, send to the chat model
Here’s a stripped-down version of the embedding step:
python
import requests
def embed_chunk(text: str) -> list[float]:
response = requests.post(
"http://localhost:11434/api/embeddings",
json={"model": "nomic-embed-text", "prompt": text}
)
return response.json()["embedding"]And then a basic retrieval query using Chroma:
python
results = collection.query(
query_embeddings=[embed_chunk(user_query)],
n_results=5
)
context = "\n\n".join(results["documents"][0])From there you pass context and user_query to your Ollama chat endpoint. It’s not pretty but it works, and you have full control over chunking strategy, retrieval count, and prompt formatting.
Chunking Strategy Matters More Than Model Choice
This surprised me. I assumed using a bigger model would fix mediocre results. It didn’t. The real difference came from adjusting chunk sizes and overlap. Chunks that are too small lose context. Chunks that are too large dilute relevance. For Obsidian notes specifically, I’ve had good results chunking by heading sections rather than fixed token windows — each ## block becomes its own chunk. Your mileage may vary depending on how your notes are structured.
What Actually Worked for Me
I started with the Smart Connections plugin and hit the silent API mismatch issue almost immediately. The chat panel loaded but never returned any responses. I checked Ollama’s logs (ollama logs), and there were 404 errors on every request — the plugin was hitting /v1/chat/completions but Ollama at that version was expecting /api/chat.
I tried changing the API base URL, checking the plugin version, restarting everything. Nothing.
The fix came from a half-remembered GitHub comment I’d seen in a different thread: switching to the Ollama v1-compatible mode by updating the base URL to http://localhost:11434/v1 instead of just http://localhost:11434. One character change. Worked immediately. I wasn’t particularly methodical about finding it — I got lucky that I’d seen that comment before.
After that, everything clicked. Embeddings indexed, queries returned relevant chunks, the chat responses actually referenced my notes. Not perfectly — sometimes it would cite an adjacent note rather than the most relevant one — but good enough to be genuinely useful.
When Retrieval Goes Wrong
If your queries are pulling irrelevant notes or ignoring clearly relevant ones, there are a few things to check:
Re-index after major vault changes. The embeddings don’t update automatically in most setups. If you’ve added or reorganized notes, rebuild the index.
Check for corrupt or empty notes. A few empty files in your vault can throw off indexing without obvious errors. From what I’ve seen, Smart Connections handles this okay, but a manual Python pipeline will fail silently on empty files unless you add a guard.
Your note titles matter. If you’re using vague filenames like “Untitled 23” and putting all the meaning inside the body, retrieval suffers. Notes with descriptive titles and clear opening sentences index significantly better.
Try reducing chunk overlap if responses feel repetitive. If the same content keeps appearing across retrieved chunks, you’ve got too much overlap in your chunking config. But don’t cut it entirely — some overlap is necessary for context continuity across chunk boundaries.
Hardware Realities
Not 100% sure why this isn’t more prominently discussed in most guides, but the embedding step is generally fast even on CPU. The slow part is the chat model inference. If you’re on a machine without a dedicated GPU, you’re looking at 3–8 seconds per response on a 7B model, and longer on anything bigger.
For the AI wiki use case specifically, this is usually fine. You’re not expecting real-time autocomplete — you’re querying occasionally to surface connections or draft new entries. Latency of a few seconds per query is acceptable.
That said, if you’re on Apple Silicon, Ollama’s Metal support is solid and you’ll get noticeably better performance than CPU-only mode. On Windows with an NVIDIA card, make sure CUDA is configured — Ollama should detect it automatically, but check the output of ollama run llama3 to confirm it’s using GPU layers.
FAQ
Can I use GPT-4 or Claude instead of a local model with this setup?
Yes, and it’s honestly less setup work. Both the Smart Connections plugin and a custom Python pipeline support any OpenAI-compatible API. But if you’re asking this question, you might be missing the point of the local LLM approach — keeping your notes off external servers.
Does this work on mobile Obsidian?
For the plugin approach, not really. The plugin needs to reach your local Ollama server, which means your phone and computer need to be on the same network and you’d need to expose the server correctly. It’s doable but not worth the hassle for most people.
My vault has 10,000+ notes. Will this scale?
Indexing will take a while the first time. After that, retrieval is fast because you’re doing vector similarity search, not scanning every file. The bottleneck at that scale is usually storage for the vector database, not query speed.
What’s the difference between Smart Connections and Text Generator plugin?
Smart Connections is focused on semantic search and surfacing connections between notes. Text Generator is more about generating content inside your notes. They do different things and aren’t really in competition — some people use both.
Why do my responses ignore notes that are obviously relevant?
Usually a chunking problem or a stale index. Try reindexing. If the problem persists, look at how those notes are structured — dense, list-heavy notes without clear prose often embed poorly compared to notes with more natural language.
Editor’s Opinion
honestly this setup is more work than the tutorials make it look. the plugin route is fine if you hit the right combination of versions, but there’s a real chance you spend two hours on an API path issue that has nothing to do with your notes or your model. the manual python pipeline gives you way more control and is weirdly satisfying once it works. if you have ~200 notes and decent hardware, just do it. worth it.