Ever since large language models (LLMs) became a thing in 2022-2023, there has been talk of a future in which we can all run models locally on our own desktops or laptops, thereby avoiding the costs and privacy risks that come with using flagship products such as ChatGPT or Claude. A whole community of so-called “localmaxxers” (not to be confused with that word’s original meaning) has emerged, one that experiments with open-source models small enough to fit on consumer hardware and tries to squeeze out as much performance as possible from them.
For a tinkering academic like myself, this was a tantalising prospect, because it promised to lower cost, provide safe processing of sensitive materials (like jihadi propaganda, in my case), and allow for deployment of fine-tuned models for specialised tasks. I have therefore been following developments in this domain quite closely and I have experimented more than I have dared tell my wife. In 2024 I even spent £1,500 on two used Nvidia RTX 3090 GPUs to get 48GB of VRAM to play around with.
Two years in, I have to admit that the whole effort has been a bit of a failure. Not because the models I deployed locally couldn’t do cool things, but because the larger models available from data centres were always much further ahead. Indeed, the gap only increased with the advent of agentic AI in late 2025, when a model’s ability to use tools became crucial. Many weak models struggled with this and became almost unusable in agentic settings.
Today, the strongest model I can run on my two GPUs at a decent speed (50+ tokens per second) is qwen-3.8-27b. It works in self-hosted chat interfaces like Open WebUI and agentic harnesses like OpenCode and is generally pretty good. Indeed, it is stronger than any ChatGPT model from before 2025, and it can probably perform 70 per cent of the tasks I use agents for. But it’s slow (takes around 30 seconds to cold start), and I don’t always know whether the task at hand is one that it can handle or not. That very uncertainty is enough to make me hesitant to use it for high-stakes tasks under time pressure. Of course, I could invest in more compute, but to get north of 48GB VRAM, I would be looking at several thousand pounds in extra hardware.
That investment might have been worth it if it hadn’t been for a new development in 2025-2026: the rise of independent data centre providers (so-called neoclouds) offering inference on very powerful open-source models. The models they serve – such as GLM-5.3 and Kimi K3 – are much too big to ever run on consumer hardware. You could create your own little data centre to run them, but it would cost you of the order of £30,000 to £40,000. These models are really powerful and only lag behind the flagship models by about 3-6 months. Given that the capabilities of the flagship models are now overkill for most people’s daily tasks, the large open-source models are really attractive, not least because they cost 1/10th to 1/20th of the commercial flagship models. A model like deepseek-4.1-flash-1017, for instance – the one I currently use for almost everything – now costs $0.26 per task whereas Claude Fable costs $7.63 (about 30 times as much).
Privacy-wise, these large open-source models are actually much better than ChatGPT and Claude, so long as you use the right inference provider. There are now European companies such as TensorX that offer inference with Zero Data Retention and GDPR compliance, which means that your chats are never stored and certainly not trained on. It is not as secure as a model on local infrastructure, which will work even with your internet connection turned off, but as digital services go, it is about as private as it gets. It is certainly a lot better than using LLMs through OpenAI and Anthropic (to say nothing of the Chinese labs), who openly say that they retain your data and train on it. Personally I trust TensorX-served models to process my most sensitive data, such as my taxes, my emails, and my diary.
I now almost never use my local qwen-3.8-27b, except for token-heavy, relatively simple tasks that I know it will not fail on. For example, I will regularly have it plough through my personal notes archive (3,000 markdown files) to weed out formatting errors and broken links. But for everything else, I turn to an open-source model from TensorX. And if that model struggles, I escalate to a flagship OpenAI model, but that is increasingly rare.
The whole experience has made me realise that, while there certainly is a place for local models, society as a whole is much better off pursuing collective solutions in the form of privacy-friendly inference providers that can serve powerful open-source models. The principle of economies of scale makes this fairly obvious. If private inference is only to be had on local infrastructure, it is going to be reserved for the rich, because it is very expensive to buy the hardware needed to run a decent LLM.
There is a clear analogy here to things like solar panels. Due to the fixed per-unit installation costs, it is much more efficient to build large solar farms than to put solar panels on individual houses. I haven’t seen data on the socioeconomic profiles of people with solar panels on their house roofs, but I’ll be damned if it isn’t overwhelmingly upper-middle-class families. (My neighbourhood in north Oxford has a lot of them.)
I’ve also come to realise that there are interested parties fishing in these waters, namely, the companies selling hardware for local inference. Apple’s Mac mini, Nvidia’s DGX Spark, AMD’s Strix Halo processor: these are all marketed to people pursuing the dream of the local model.
Maybe there will come a time when our AI needs can be met with models on local hardware. But that time is not now. In the meantime, we should promote collective solutions in the form of private inference from large, efficient data centres.