Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.
I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
> it seems really far off from the kind of experience even a basic $20/month subscription gets me.
The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.
The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.
But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.
I think 24gb is the bare minimum for a local qwen3.8 based setup. I've used qwen3.6 and it's not as straightforward as "can it replace <insert the most cost-effective cloud solution today>"
Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.
Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.
The principle of KISS keeps coming to mind when I consider local computing. I'm looking forward to the day we can just run Opus-level models at 100 tok/sec on consumer hardware.
But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)
I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).
The article implies the demand is for running locally. I’m not convinced, at least with a mac minis. Most people I know and myself buy the mini as it is always on, easy to setup, and isolated from my main computer which is a laptop. The mini is driving the use of the $20/month subscriptions.
IMO local models require a substantial amount of prompt+harness engineering to get in the neighborhood of what you'd get from a cloud model. Which isn't a bad thing, you'll learn a whole lot about how these things work.
What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.
When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.
Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.
I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference. It is nice to know that I have hardware under my desk that I control with open weight models that I can interact with on my terms.
Could you elaborate on your experience with local models on your card? I've been thinking of upgrading to 9070 XT, and was thinking the 16GB would be okay-ish to at least run something usable locally, no?
Usable certainly. But my impression is that useful models still need a bit more than 16GB. Something like Qwen 3.8 27B is useful but squeezing it into 16GB requires fairly aggressive quantisation which will make it unreliable (e.g it'll get stuck in loops) and won't leave enough space for a long context (which qwen 3.8 really likes)
I was getting semi-useful results from a 128GB M4 Max. That was a few months ago, and the models have improved (quite a bit) since then, but now I'm happy to send my $20/month to get Claude code.
It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
Bloody training data full of humans demanding sleep. I tells ya...
I have multiple 48GB friends that successfully run smaller quantized models for general assistance + light editing (coding, spreadsheets, etc), that don't require very heavy models.
So yes, they are genuinely very useful, but they are not yet a full replacement unless you have more powerful hardware and or don't need more intelligent ai.
It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
Maybe a bit of hindsight bias / the outside view here, but I feel like they're completely asleep if they didn't anticipate strong demand for this specific use case.
I don’t understand how that’s possible. They should have had a better idea of what was happening in the memory markets than pretty much any other entity.
My vibes were that Apple wound down the “actual work” side of their operations (including machines like Xserve), because Ives couldn’t handle the unsexiness and unpredictability of business requirements in hardware.
He was self-indulgent and only wanted to work on things that “vibed” with him, rather than what the customers needed. It’s easy to be creative when you get to do what you want to do, it’s hard when you have hard constraints.
If I had to pick a product, I'd say an affordable 32GB mac would be the sweet spot for running local models that function well like Qwen 3.8.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.
I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
I am missing something in the article. From what I am reading, AI companies are so desperate for memory they are buying Apple Systems and other hardware and striping them for RAM and maybe other components.
It definitely puts it out of the range of every day users but a non-insignificant proportion of people who use it for ai have become multi-millionaires because of ai. So there's definitely no shortage of people who have no problem paying those high prices.
What makes you think that? There's a lot of data centers that sell you access to colocated Mac Mini's, they have added FileVault unlock via SSH in the boot process which also makes things easier. There's not that many reasons to run a Mac in the cloud unless you have some very specific Mac related workload.
At this point it’s a well known secret that Apple has real rack mount servers for their internal processes. They actually have officially released video of their servers in the WSJ report on their chip supply chain.
They did. Now think they feel a stack on Minis or Studios fills the reduce needs better. The multiple machines one gets software redundancy in addition to everything else.
Apple has a huge opportunity here to lead the market for machines to run local models if they step into it. Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.
It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.
If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.
I realize I’m somewhat limited (16GB RTX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.
The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.
But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.
IDK if that might be a concern for Apple or their AI partners.
Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.
Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.
But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)
I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).
What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.
When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.
It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.
Bloody training data full of humans demanding sleep. I tells ya...
So yes, they are genuinely very useful, but they are not yet a full replacement unless you have more powerful hardware and or don't need more intelligent ai.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
My vibes were that Apple wound down the “actual work” side of their operations (including machines like Xserve), because Ives couldn’t handle the unsexiness and unpredictability of business requirements in hardware.
He was self-indulgent and only wanted to work on things that “vibed” with him, rather than what the customers needed. It’s easy to be creative when you get to do what you want to do, it’s hard when you have hard constraints.
It's true, most people don't run models, but being the default platform for running open weights seems like it has plenty of advantages right now. Just like sales benefited from developers defaulting to MacOS for most open source languages like Ruby, Go, Rust, and TypeScript.
I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.
Is that what others read ?
[1] https://tech-insider.org/ca/steam-deck-price-increase-2026/
Your letter will help feed the boiler that heats the building and is much appreciated.
What makes you think that? There's a lot of data centers that sell you access to colocated Mac Mini's, they have added FileVault unlock via SSH in the boot process which also makes things easier. There's not that many reasons to run a Mac in the cloud unless you have some very specific Mac related workload.
https://www.macrumors.com/2026/08/26/leaked-images-of-apple-...
https://www.reuters.com/business/apple-begins-shipping-ai-se...
https://forums.macrumors.com/threads/photos-of-apples-own-ne...
https://wccftech.com/apples-private-cloud-compute-server-m5-...
https://security.apple.com/blog/private-cloud-compute/
Here's a leaked / rumor image of Apple servers themselves.
https://www.macrumors.com/2026/08/26/leaked-images-of-apple-...
There’s articles about them, Apple uses them internally for AI services. https://forums.macrumors.com/threads/photos-of-apples-own-ne...
> [still claims there is no reason]
It's a niche market but it's a market that overlaps heavily with professionals in the AI space and lead developers, so it's a market that gets them customers in those roles.
If I were running Apple I'd call the RAM price bubble for what it is and temporarily eat some margin to offer machines with more RAM than competitors, especially these models that are great for edge AI, and capture market share.