There was a window — maybe 18 months — when running a language model locally felt like the principled choice. Privacy by default. No API costs. No vendor dependency. Your data never leaves your machine.
I believed it enough to build around it. Wolftrain, one of the early Wolflow tools, was designed for local LoRA fine-tuning. Wolfstitch Desktop processed documents offline. The whole early architecture assumed local-first.
Here’s what I actually found.
What works
Summarization and extraction on known document types. If you have a consistent document format — contracts, invoices, reports — and you’re pulling structured information from them, a well-quantized 7B model running locally can do this reliably. It’s slower than a cloud API call but the latency is acceptable and the privacy argument is real.
Classification tasks with narrow output contracts. Local models handle binary or small-set classification well when you constrain the output format carefully. “Is this expense report compliant: yes or no” is a tractable local task. “Analyze this expense report and explain any compliance concerns” is not.
Development and prototyping. Running locally during development means you’re not burning API credits on every iteration. This is genuinely useful and I still do it.
What doesn’t work
Anything requiring reasoning over long context. The models capable of this require hardware most organizations don’t have. We’re talking 24GB+ VRAM for models that can actually handle enterprise-length documents. Consumer hardware hits a wall fast.
Consistency at scale. Local models at the quantization levels that run on accessible hardware hallucinate more, drift from instructions more, and require more prompt engineering to stabilize. The delta between a quantized local model and GPT-4 is not marginal — it’s meaningful for production use cases.
Keeping up. The frontier moves every few months. Running locally means you’re always behind it.
The Ollama signal
When Ollama — one of the main local model platforms I was tracking — moved toward cloud and subscription offerings, it validated something I’d already concluded empirically: the “local first, always” position doesn’t survive contact with real organizational requirements.
That doesn’t mean local models have no place. It means the place is narrower than the hype suggested.
What I’d tell an organization today
Local models make sense for: specific, well-scoped tasks where data privacy is non-negotiable and the output contract is tight. They don’t make sense as a general AI strategy.
The question isn’t local vs. cloud. It’s: what does this task actually need? That’s a question Wolfprompt is designed to answer — classifying prompts by knowledge source, reasoning depth, context dependency, and output contract before routing them anywhere.
The routing decision should follow from the task requirements. Not from a philosophical position about where computation should happen.