On this page
Notes on Self-Hosted AI: Small Models, Big Wins
aiself-hosting
Running a useful local model on modest hardware.
You do not need a giant GPU to get value from local models. A mid-range desktop can run purpose-fit small models that handle summarisation, classification and code completion surprisingly well.
What to run
- Embeddings for search — a small model like
nomic-embed-text. - Classification and tagging — a 1–3B parameter instruct model.
- Code completion — a model tuned specifically for completion.
Practical tips
- Keep context small; small models degrade as context grows.
- Batch work at night when the machine is idle.
- Structure prompts as JSON you actually parse, not prose you eyeball.
The biggest win is not the model. It is deciding which questions deserve to stay on your own hardware and which you will never ask the internet.
Reading the numbers
Small models are measured in tokens per second, not in benchmark points. A model you can run locally and tune beats an abstract giant you can only call through an API.