Local AI has been a major topic of discussion recently. Unless you have been living on a deserted island, you have almost certainly heard about the power and promises AI can offer, however exaggerated they may be. Big AI providers want you to think that they are the only ones with models worth using. Hand us $200 a month, so the story goes, and our systems can build you anything your heart desires (with 5-hour limits, of course).
And, in many cases, that’s true. The latest models from OpenAI, Anthropic, and xAI can build amazing things. They can spawn subagents and, with well-defined specs, agentic factories can take you from idea to finished product in minutes (even if those same models, trained on human data, will quote you days).
The problems show up later. Code written by a machine at lightning speed, barely reviewed by others, and put into production sounds great at first, until you have to explain it or feel reasonably certain that the AI-built solution has considered legal regulations, security risks, and infrastructure pitfalls. While lightning-fast code generation has value, it also has downsides.
Local AI presents an interesting alternative. People wrongly present egregiously expensive hardware as a potential cost saver; a way to bypass the cloud subscription fees and, if you spend $20,000 or so on a 128GB unified memory machine during the height of RAMageddon, you can avoid spending $200 a month.
The problem is that even 128GB of memory cannot get you close to frontier intelligence. These models run better and faster than anything your $20,000 or $80,000 setup can do.
And spending even $20,000 to save $200 a month is a tough pill to swallow, especially as hardware, models, and software evolve at breakneck speed.
But all of the software on this site, including this site itself, was built with local AI. The only time a frontier model was involved was for the initial AGENTS.md file; a frontier AI, Gemini 3.8 Thinking, wrote the architecture that the local AI implemented.
I truly believe local AI is the future. Frontier intelligence is powerful, but it is not the final destination. Here are three reasons local AI deserves serious consideration over a Claude subscription.
1. You Don’t Need That Powerful a Machine for Local AI
I built everything with my M4 Pro 14-inch MacBook Pro with 24 GB unified memory. I use Unsloth’s Qwen 3.8 27B IQ4_XS with my custom harness, Terrapin, in Xcode. I use Splash and using this setup, my laptop can achieve about 30-40 tok/s at the high end and about 20tok/s as the context window fills up. It achieves a 140,000-token context window, which is more than enough for agentic workflows.
To get the higher context windows, I have to set the iogpu.wired_limit_mb as follows:
sudo sysctl iogpu.wired_limit_mb=22528
Then, I just run the standard Splash command with the defaults. It does a good job of optimizing things and giving me lightning-fast prompts. Nothing fancy. It just works.
If I need more speed, I can use the Qwen 3.6 35B-A3B model with the same 140,000-token context window for long agentic workflows.
Is it slower than any model from Anthropic or GitHub? Absolutely - especially on prompt prefills.
But here’s the interesting thing: after using these models extensively, I have learned what does and doesn’t work. I’ve learned how to ask for clear results, and also what confuses the model. The 30-40 tok/s streaming has permitted me to read how Qwen3.8 thinks and learn what it can and cannot do effectively. I know which tasks it excels at best. I can also use Qwen3.8 for planning, but use Qwen3.6 for implementation at lightning speeds.
When I try the same exercises with GPT-6 Astra or GPT-6.1 Sol, much of that underlying thinking is hidden from me. I can see some of it, but much of it is obscured. When it works, great. When it fails, I often don’t quite know what’s going on.
For example, once I had Qwen3.8 write 3 planning documents. Then I had Qwen3.6 iterate through all three. The 3.6 version of Qwen hallucinated some Swift syntax and left the project with a few build errors, which Qwen3.8 then cleaned up overnight. In a few hours, I had three features written for me.
People often create YouTube videos about buying that 512GB Mac Studio and how it compares with the latest cloud models. But your Mac Studio won’t beat OpenAI, so the real question changes.
Instead, consider another question. What if you could augment the hardware you already know and use, or already want to buy and will use, with a helpful AI assistant? What if that new MacBook Pro PROLED touchscreen Mac, the one you already wanted to buy anyway, could crank out code for you? What if it could build apps, write documents, and help run your business locally, with no extra cloud subscriptions?
What if instead of buying some outlandish computer just for AI, you bought something that you can also use for yourself, that also happens to have agentic capabilities?
It’s a subtle distinction, but all too often we compare extremely costly local AI “servers” with frontier intelligence and conclude that local AI inference is too expensive to compete on arbitrary synthetic benchmarks. Instead, ask a simpler question: what can you build with a frontier model that you can’t build with a local one? That is the real test, and it reframes the comparison around practical output.
Put another way: sure, you can vibe code a clone of Photoshop, but is that truly worth spending thousands of dollars in AI credits? For workflows that genuinely matter to you: do you need frontier intelligence and speed, or is cranking something out that you can review overnight also an option? If the latter is also an option, isn’t the privacy and security of local compute worth it?
2. The Harness Matters - A Lot
As we are quickly discovering with AI, the harness matters as much as, or even more than, the model itself. All the instructions, tools, and directions, including retry capabilities and intelligent tweaks, matter a lot to a long-running agentic workflow.
In fact, Google released a project called Harness2 that improves models simply by recursively improving the harness. The idea is simple. Consider the following hypothetical example. Suppose you give an agent the following task:
Write a 3D game for Windows according to my long, detailed specification.
The framework will use AI to tweak the harness, run the same job, and compare the outputs. If harness A did better than harness B, it will use harness A for the next job and create a new candidate harness C to compare. The idea is that, with time, the model - which hasn’t changed during any of this - gets enough updated system instructions that it produces the best output for your specific tasks.
So, getting back to the hypothetical task above, suppose harness A was a generic harness, but harness B had a new system instruction in it:
While developing the game, pay specific attention to performance. Use tools to benchmark performance and never consider a project complete unless it achieves at least 60 frames per second.
If the AI model were good, presumably it would follow the above instruction with harness B and create a game that felt more fluid and responsive. Now, harness C might have another instruction that further optimizes the output.
That’s the power of a quality harness. Anthropic, GitHub Copilot, OpenAI, and other major providers spend a lot of time engineering these harnesses to provide optimal outputs, which is why the harness discussion matters here.
Raw model power matters, but the harness - especially one tailored to specific model types - can help overcome many obstacles.
3. A Local Model Never Changes
Raise your hand if you’ve seen this one before. A major AI company releases a brand new model, and everyone goes crazy. The amount that you can now vibe code is nothing short of astonishing: vibe-coded game clones, vibe-coded 3D models, and more. And then, about a week later, the questions start coming in: has such-and-such big AI company nerfed the model? What used to take 3 hours and 100 credits now takes 6 hours and 1,000 credits; what’s the deal?
You’ve probably seen that one before.
The reality is that Anthropic claims it ships 30,000 PRs per day. I find that a little dubious (how much code is in each PR?), but the more general point is that everything is changing constantly. Anthropic may add some code that degrades your model. Or, all of a sudden, OpenAI may add a safety rule that inadvertently catches your research project.
With local AI, your harness is your harness. It doesn't change unless you change it. Your model doesn’t get better or worse. It only changes if you download something else.
Consider two employees. One is wicked smart and brilliant, but their output is variable. Some days they seem incredibly gifted; other days, they seem less capable. Another employee isn’t the world’s best, but they’re dependable. You start to get a feel for what tasks they’re good at and which they’ll struggle with, and you know roughly how long each will take. Which employee is better?
For some people, who want to move fast and break things, the first employee is the best. But that doesn’t mean that there aren’t very valid use cases where the second employee might perform better in the long run. Now add cost. Suppose the first employee costs a lot while the second is free.
I’m not saying frontier models have no value. I’m saying local AI has immense business value because it never changes. It is dependable, local, requires no internet connection, and has no costs. You can give it a problem, and it will churn out a reliable, quality answer in time. That is the point: local AI gives you predictable value you can count on.
That predictability has real value in any business!
Local AI Has a Strong Future at a Developer’s Side
In short, I truly believe local AI has a strong future for any developer, no matter how much of a veteran or beginner you are. You don’t need to buy a $20,000 machine to get started. If you have a reasonably new Mac or PC, you can probably run a model on it. Even if your token generation speed doesn’t match the latest GPT-6, it’s worth tinkering with local AI. It helps you understand and learn more about AI concepts, and it can help you build quality apps, sites, and pages, all without giant cloud fees.
Now is a great time to try running local models. If you’re looking to run something in Xcode, I’d humbly suggest our project, Terrapin, the ACP server we use to build our Apple Swift projects. For more information, please visit this project’s GitHub page.