Pushing Local Models With Focus And Polish

"Putting an API key into Pi and using a hosted model is a very boring operation. You select the provider, paste the key and then you are done thinking about how to get tokens. Doing the same thing locally, even when you have a high-end Mac with a lot of memory, is a completely different experience. You choose an inference engine, then a model, then a quantization, then a template, then a context size, then you've got to throw a bunch of JSON configs into different parts of the stack and then you discover that one of those choices quietly made the model worse or that something just does not work at all."

"We have an enormous amount of activity around local inference, which is great. We have good projects, fast kernels, and people are doing great quantization work. A lot of very smart people are making all of this better, and yet the experience for someone trying to make this work with a coding agent is worse than it has any right to be. Putting an API key into Pi and using a hosted model is a very boring operation."

"A lot of local model work optimizes for making models runnable. That is necessary, but it is not the same thing as making them feel finished. I give you a very basic example here to illustrate this gap: tool parameter streaming. For whatever reason, most of the stuff you run locally does not support tool parameter streaming. I cannot quit"

"I want them to work in the very practical sense that I can open my coding agent, pick a local model, and get something that feels competitive enough that I do not immediately switch back to a hosted API after five minutes. There are a lot of reasons why I want this, but the biggest quite frankly is that we're so early with this stuff, and the thought of locking all the experimentation away from the average developer really upsets me."

Local inference for coding agents is active and improving through fast kernels and quantization work, but the end-user experience remains worse than hosted APIs. Hosted usage is simple: select a provider, paste an API key, and start generating tokens. Local usage requires choosing an inference engine, selecting a model, deciding on quantization, applying templates, setting context size, and configuring multiple JSON settings across the stack. Small configuration choices can silently degrade model quality or cause failures. Many local projects focus on making models runnable rather than making them feel finished. Tool parameter streaming is a concrete example where most local setups do not support required functionality, limiting agent performance.

Read at Armin Ronacher's Thoughts and Writings