Lessons From Running Local LLMs on Consumer Hardware

I run language models on a lot of small machines — Raspberry Pis, Jetson Orin Nano, modest desktop GPUs. This is a practical note on what I’ve learned, focused on real use rather than benchmarks.

What works

• Small specialised models beat large general models for narrow tasks. A 3B model that’s been fine-tuned on the exact format you want will outperform a 70B general model that has to be prompted carefully. The small model is also fast enough to run interactively on a Pi.
• Quantisation is free in the right regime. 4-bit quantisation is the sweet spot for most small models — quality is barely affected, memory and speed gains are large. Going below 4-bit starts to hurt.
• The first second of latency matters more than tokens-per-second. A model that streams the first token in 200ms feels interactive. A model that takes 2 seconds to start streaming feels broken, even if it then types fast.

What doesn’t

• Long context is expensive. A model that runs fine at 2K context can choke at 16K. Most of my “make this work” moments come from trimming context, not from upgrading hardware.
• Tool calling is harder than it looks. A model that can chat well is not necessarily a model that can call functions reliably. The gap is real and shows up immediately in agentic loops.
• Reasoning models are slow on small hardware. The “think before you answer” models are great on a big GPU and painful on a Pi. For an edge deployment, stick to instruct-tuned small models.

The stack I’ve settled on

• For Pi-class machines: 1B–3B instruct models, 4-bit, served locally
• For Jetson Orin Nano: 7B–8B models with quantisation, served over local network
• For desktop dev: 70B+ on a real GPU, used for generation, not serving

The boundary is roughly: if the model has to answer in under 500ms, it has to be small. If it has a few seconds, the bigger model is worth the wait.

What’s still open

Multi-modal on the edge. A small vision model that can describe what a camera sees, on a Pi, is still hard. The models exist; the speed doesn’t. I’m watching this space closely.