How to Run Llama 3 70B on a Home Server

If you want to run Llama 3 70B on a home server and actually reach it from your phone or laptop when you’re not home, the short version is: it’s doable, but you need real hardware and a tunnel that doesn’t depend on port forwarding. I spent the better part of a weekend getting this working the “easy” way first, then the actually-easy way once I stopped fighting my router. Here’s what that looked like, step by step.

Quick Answer

• 70B at Q4_K_M quantization needs roughly 40-48GB of VRAM (or unified memory on a Mac) — there’s no getting around this with software tricks
• Ollama handles the model serving; Tailscale handles getting to it from outside your house without opening router ports
• A single consumer GPU won’t cut it solo — you’re looking at dual 24GB cards, a 48GB workstation card, or a Mac with 64GB+ unified memory
• Most home ISPs use CGNAT, which silently breaks traditional port forwarding — this is the part nobody warns you about
• Open WebUI on top gives you a normal chat interface instead of a bare terminal session

What You Actually Need (And Why People Get the Sizing Wrong)

The number that trips people up is the parameter count itself. People see “70B” and assume it’s roughly 10x the resource cost of an 8B model. So they buy a 24GB card thinking they’re being generous, and then can’t figure out why the model won’t load.

Here’s the actual math. At Q4_K_M quantization — the default quant most people end up using — Llama 3 70B needs somewhere around 40-43GB just for the weights. Add context window overhead on top of that, and you’re realistically planning for 48GB+ of VRAM or unified memory if you want headroom. That’s not a “buy a slightly better GPU” problem. That’s a different tier of hardware entirely.

A few specific reasons people hit walls:

They quantize too aggressively and don’t notice the quality drop. Q2_K shrinks the model enough to fit on less VRAM, but the output gets noticeably worse — repetition, weird logic gaps, that kind of thing. From what I’ve seen, Q4_K_M is the sweet spot. Going lower to save memory usually isn’t worth it for anything beyond casual chat.

They forget system RAM matters even with a GPU. If even a few layers spill over to CPU because VRAM is tight, your tokens-per-second drops off a cliff. Not gradually — it falls off hard. I’ve watched a generation that should take 4 seconds stretch into 40 because three layers didn’t fit.

They assume their home network will just… work for remote access. It won’t, and this is the part that actually derailed my first attempt, not the GPU.

Hardware Paths Compared

I didn’t fill every cell here on purpose — the “best for” column gets fuzzy once you factor in what you already own. If you’ve already got two 3090s sitting in a gaming rig, that’s your answer regardless of what a fresh-build comparison says.

Step-by-Step Setup

Step 1: Install Ollama on the server machine

Ollama wraps llama.cpp behind a clean CLI and REST API, and it’s genuinely the path of least resistance here. Install it via the official script on Linux, Homebrew on Mac, or winget on Windows. Confirm it’s running with ollama list before you do anything else — skipping this check is how you end up debugging the wrong layer of the stack an hour later.

Step 2: Pull the model

ollama pull llama3.3:70b

This downloads somewhere around 42GB depending on the exact quant Ollama selects by default. Make sure you’ve actually got the disk space before you start — and double-check it’s not your boot drive filling up, because that causes a whole separate category of weird errors that look nothing like a model-loading problem.

[Image: Terminal output showing the ollama pull command downloading the 70B model with progress percentage]

Step 3: Bind Ollama to listen beyond localhost

This is the step almost everyone misses, and it’s not your fault — the defaults are quietly restrictive. Ollama binds to 127.0.0.1 out of the box, meaning only the machine it’s running on can talk to it. So even once Tailscale is set up, you’ll get connection refused errors until you fix this.

On Linux, edit the systemd service file and add:

Environment="OLLAMA_HOST=0.0.0.0"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

On Windows, set OLLAMA_HOST=0.0.0.0 as a system environment variable and restart Ollama from the tray.

Step 4: Install Tailscale on the server and your devices

Tailscale creates a private mesh network between your devices using WireGuard underneath. No port forwarding, no static IP needed, no exposing anything to the open internet. Install it on the server, sign in, and it’ll show up in your tailnet with an address in the 100.x.x.x range.

Install it on your laptop and phone too, signed into the same account.

Step 5: Test the connection

From a device on your tailnet, hit the Ollama API directly:

curl http://100.x.x.x:11434/api/tags

If you get a JSON list of your installed models back, you’re through the hardest part. If you get connection refused, go back to Step 3 — that’s the OLLAMA_HOST binding again, and it’s almost always that.

Step 6: Add Open WebUI for an actual chat interface

Running everything through curl gets old fast. Open WebUI gives you a browser-based chat frontend, and it’s a single Docker command:

docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

Point it at your Ollama instance under Settings → Connections, using your Tailscale IP and port 11434 as the base URL.

What Actually Worked For Me

My first instinct, like everyone else’s apparently, was to just forward port 11434 on my router and call it done. That didn’t work, and not because I configured it wrong — turns out my ISP runs CGNAT, so my “public” IP isn’t actually public. I burned almost two hours convinced I’d fat-fingered something in the router config before I found a forum thread mentioning carrier-grade NAT as the actual culprit. That’s not something most setup guides warn you about, and it should be higher up in every one of them.

After that I tried a cheap commercial VPN as a workaround, mostly out of stubbornness. It technically worked but added enough latency that the chat felt laggy even though the model itself was fast. Not great. So I gave up on clever workarounds and just installed Tailscale, which took maybe fifteen minutes total including signing up for the account. The contrast was almost annoying — all that troubleshooting for a problem that had a fifteen-minute fix sitting right there the whole time.

Advanced Fixes and Edge Cases

Multi-GPU tensor splitting. If you’re running dual GPUs instead of one big card, set CUDA_VISIBLE_DEVICES to the comma-separated GPU IDs you want Ollama to use, and it’ll split layers across them automatically. Expect noticeably lower tokens/sec than a single 48GB card due to PCIe bandwidth between the cards — that’s normal, not a misconfiguration.

Model unloading mid-session. Ollama unloads idle models after a few minutes by default, which means your first remote request after a break eats an extra 30-60 seconds reloading 40+GB into memory. Set OLLAMA_KEEP_ALIVE to a longer duration, or -1 to keep it loaded indefinitely if you’ve got the memory to spare.

Diagnosing partial GPU offload. Run ollama ps to check whether the model actually landed fully on GPU or partially spilled to CPU. If you see CPU involved at all for a model that should fit, something’s off with available VRAM — close other GPU-using processes and try again.

HTTPS for browser features. Some browser APIs (microphone input in Open WebUI, for instance) require HTTPS, even over a private tailnet. tailscale serve --bg --https 443 localhost:11434 handles this without needing to fuss with certificates yourself.

Prevention Tips

• Don’t open ports on your router for this — Tailscale (or a similar mesh VPN) removes the need entirely, and an exposed Ollama API gets found by bots faster than you’d expect
• Check actual VRAM usage with nvidia-smi before assuming a model “should” fit based on parameter count alone
• Keep an eye on thermals if you’re running sustained 70B sessions — sustained load on consumer cards in a closed case adds up
• Back up your Modelfiles if you’ve customized system prompts or context settings; reinstalls wipe custom configs

FAQ

Can I run Llama 3 70B on a single RTX 4090?
No. 24GB isn’t close to enough at any quantization that keeps quality reasonable. You’d need a second card or different hardware entirely.

Does Tailscale slow down inference?
Not really — the model runs locally on your server’s GPU regardless of how you connect to it. Tailscale just handles getting your request there and the response back; the heavy lifting never leaves your home hardware.

Why does my remote connection say “connection refused” even after Tailscale is working?
Almost always the OLLAMA_HOST binding. Tailscale connecting successfully doesn’t mean Ollama is listening on anything but localhost — those are two separate fixes.

Is it actually cheaper than just using a cloud API?
Depends on your usage. If you’re running heavy sessions daily, the hardware pays for itself over months. For occasional use, a cloud API is probably still cheaper when you account for electricity and the upfront GPU cost.

Can I access this from my phone on cellular data?
Yes, as long as Tailscale is installed and signed in on the phone too. It works over any connection, not just WiFi.

Editor’s Opinion

honestly the hardware part is the boring half of this, everyone already knows 70B needs a lot of VRAM. the networking part is where people actually get stuck and nobody talks about CGNAT enough. if your port forward “should work” but doesn’t, that’s probably why. tailscale fixed in 15 min what i spent 2 hours fighting with a router for. just start there.