We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I think the discrete GPUs as a concept is coming to an end. Computer graphics is shifting more and more to using local AI-like things like DLSS that allows to hard-render simpler geometry and shadows and then apply AI-realism filters in real-time.
It's harder and harder to justify gaming PCs having separate pools of GPU-only memory and "everything else" RAM.
I think unified memory is the obvious next step for personal computing and vendors will be coming up with solutions that enable more modular approach to building enthusiast grade upgradable personal computers.
How would DLSS, which runs on GPU hardware — and specifically, as I understand it, GPU hardware that accelerates AI inference — eliminate the GPU?
And if a system is intended to be modular and upgradable, what would combining the GPU and CPU into a single module contribute to this goal?
Now an architecture that separates the memory controller from the CPU and gives pluggable CPUs and GPUs (and storage and network controllers, for that matter) equal access to a single pool of RAM in a secure and high-performance way, that would certainly be interesting, and not only or even primarily for gaming.
DLSS does not require any specific packaging configuration. There's no reason DLSS can't run on DGX Spark or GPU cores of the Apple's M-chips aside from specific Nvidia software optimizations that can surely be worked out if needed.
I imagine the PCs for the future would maybe have motherboard with two sockets: a normal CPU one and a GPU one, both having access to some high bandwidth unified memory that.
Right now the unified memory needs to be pretty much soldered super close to the chip to ensure the bandwidth and energy efficiency, but hopefully there will be some solution to circumvent this limitation in the future.
Putting CPU and GPU cores into the same physical die works for your average consumer machines, but hardcore professionals or enthusiasts would always strive for more custom configurations than existing packages can provide, like more GPU cores.
DLSS still needs discrete hardware to do that (and keeps requiring new hardware on each generation). Not to mention the types of RAM are not the same and have entirely different tradeoffs - using GDDR for a CPU would tank performance due to an order of magnitude worse latency, using DDR for a GPU would make games a slideshow due to an order of magnitude worse bandwidth. And you'd have to redesign the entire concept of how CPUs and motherboards work to get there, while making AMD and NVIDIA agree to have their chips be behest to the bridges of shitty motherboard and consumer RAM manufacturers even more than they already are.
Integrated systems have come a long way for highly optimized mobile and laptops designs with specific hardware in mind. But even if everything went perfectly - DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
Then again, maybe AGI will come and hardware will fundamentally revolutionize overnight.
> DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
The technology is very new and currently in its infancy. I do believe it's the future of computer graphics though, where instead of doing "real" photorealistic geometry and lighting, software can render a very simplified scene and have a good enough photo-realistic filter applied to it so that most people wouldn't be able to tell the difference.
Obviously, I believe it'll be flexible enough to do things other than photo-realism in the future.
> In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
I think we have different definitions of "discrete" here. In my mind AIPUs are the opposite of discrete. I might be wrong here, but I always interpreted this term as a standalone GPU unit with its own onboard memory — a thing that currently fits into a PCIE slot.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
You should bite the bullet now and buy a decent card on the secondhand market, before more people figure out what they have and double, even triple, the price.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
I see them at a little less than 400€ in the second hand market. So expensive for their age but not hard to replace if you were willing to pay 600-700 for it in 2020.
That's an 8GB card. For gaming (DLSS, etc.), you can still get newer 8GB cards like the 3070 which represent solid upgrades over the 1070 Ti on the used market for under $300. Still not great since similar cards were available at similar prices new a couple years ago, but at least not trading at multiples of MSRP like higher-end cards.
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.
I just did a cost breakdown of owning a GPU vs paying for tokens.
An RTX 5060 Ti 16GB card costs about $800 on Amazon right now. Running 18.4 hours a day ( off peak electric rate schedule ) it can process approximately 109.3 million tokens per month. The card has a 90 day return window. Credit card interest + electricity puts the cost at $80 total out of pocket, to use this card for 3 months and return it.
Paying an AI provider came out to $1,639.50 for the same 3 months of 18.4 hour a day usage.
You know its a bubble when ... $80 vs $1,639.50
I understand that if you're a major corporation, or a researcher that needs 100+ gpus for a large project, unless you have access to your own supercomputer, the economics will be different.
This was for "individual coder at home" scenarios.
tl;dr ... im exploiting jeff bezos, and math is racist. too funny.
I guess people have never heard of "options" before.
If you’re returning a product you used because you wanted to pretend it was a rental at the seller’s expense, bluntly, you’re as much the problem as the seller.
Everyone in the heirachy above you is an exploitative business psychopath. Might as well exploit their calculated greed as they attempt to exploit your empathy.
In fact I'm running qwen3.8-flash-next with a Playwright-mcp docker and it is far better for scraping than my ChatGPT or Claude subscription, for the simple reason that sites aren't lying to Playwright.
I don't understand, your subscription models can also use Playwright, you don't even need to setup an mcp, they'll automatically download playwright-cli and get going.
Yes, but even the smaller models, when used within a harness and orchestrator, can be accurate enough for real work, especially when combined or validated by other smaller models.
The 5060 wasn't the fastest/$, it just provided the most VRAM / $ as of yesterday.
Im installing it in a box with a 4070 Ti to run 2 different models simultaneously. Coder + MoE via paperclip or multica.
That probably is the cheapest per 1 GB vram right now. However, the R9700 a few months ago was only $1350/32GB or $42/1GB VRAM. And everyone knew it, because the card is now $1800 or so with supply issues.
I think the market discovered the reasonable min is 32GB. That runs Qwen3.8 Q6 with 100k+ context all in vram. 24GB does not.
You're counting on most people being willing to abuse a return policy like that. Also, the 3 month return window must've activated 3 days ago. These prices didn't start just then.
Yeah I did something similar but more extensive, because not every model is same plus depend on your options how much you want to degrade model from fp8. Some models is just slow on local hardware and not practical really.
- rate at which tokens are generated
- quality of generated tokens
- ethics of abusing return policies
- long-term viability of abusing return policies
I remember when the top GPU was 550€ and the overclocked, custom-cased, water-cooling ready version cost 660€ tops. Any GPU price out there even $800 for 5060Ti from january 2025 is not only out of any sane band, but look as if we're in a ultra-recession, yet other markets aren't affected nearly as extreme.
I bought two 2TB SSDs from TeamGroup last year, about $100/ssd. I put them side by side in a raid setup for my home server. (I wanted raid, and they were less than half the price of Samsung, etc.). I'm on a budget, so I went with the cheap ones and raid reduduncy vs one expensive one with no reduduncy.
The exact same ones are now pushing $300/ssd and Samsung's (which was $300 18 months ago) is now $500.
You're comparing running a heavily quantized 10-30B model to running a multi trillion one. Being half the price (unless you expect to only need AI 3 months in your life) is absolutely not a good deal, here.
What it does buy, though, is privacy and freedom (nobody can decide to "change the terms" to milk you more).
Yeah, and Qwen 3B that you run on 16GB of VRAM and 10K context length surely is comparable to whatever subscription you're comparing to, that also gives access to 3B parameter models? You might want to take actual use cases into account before you try this scam in real life.
I too did a cost breakdown of buying my GPU for ~10K EUR more than a year ago. It came out "Invaluable" as I use it for analyzing my diary, financials and other private things I cannot send to random 3rd parties.
But even the best of models you can run with 96GB VRAM really doesn't come close to the hosted SOTA models, sadly.
Thankfully, the environment we are all forced to live in, doesn't care what we think about it, and adapting to it accurately at minimal expense is the difference between surviving and thriving.
What people don't like is the environment, and they just decided to stone my post to death for adapting to it, instead of doing the same.
People are just being people.
I'm a much easier target than that Sam guy, who locked up 40% of the planets GPU's using purchase orders, with no obligation to follow through.
He's an untouchable elite. I'm much easier to mug.
We are all opportunists.
The entire thinking behind the post, was spawned by the article the other day, about the "video game construction kit" guy, bragging about spending $3.1 million a month on tokens in an article, with $0 revenue.
> What scam are you speaking of ? The return window ?
Your calculation seems to have depended on "Buy this hardware with the explicit intention of returning it" process, which is a scam. If you buy hardware you do so with the intent of keeping it, otherwise you find renters, if you don't like scams.
So if you don't plan to return it, wouldn't the cost at minimum be the cost of the GPU??? Why count on returning it if you don't plan to do so?
Firstly you are assuming $0.048$/hr for GPU, which is highly misleading and you can easily sell it for double(vast.ai is $0.12) as you are forgetting lot of cost like CPU, cooling etc.
Also even if you can somehow fit a few 100 billion parameter model(like gpt 6 luna or deepseek flash) in 16GB, and magically assume you can run at 200 tokens/sec according to your calculation, the blended cost of it would be around $100, not $1639.
You're comparing apples to oranges. 109 million tokens is ~nothing, easily a day's worth of casual use. Compare price per token and cloud providers are probably much closer. But even that isn't a fair comparison between cloud AI providers and abusing retailer return policies just yet, because usable local models even remotely comparable to the frontier don't fit in 16gb of vram.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
What's the problem?
Frontend has been solved for years.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
Who is "we"?
I am geninuely interested what stack you are using here. Could you please elaborate?
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
It's harder and harder to justify gaming PCs having separate pools of GPU-only memory and "everything else" RAM.
I think unified memory is the obvious next step for personal computing and vendors will be coming up with solutions that enable more modular approach to building enthusiast grade upgradable personal computers.
And if a system is intended to be modular and upgradable, what would combining the GPU and CPU into a single module contribute to this goal?
Now an architecture that separates the memory controller from the CPU and gives pluggable CPUs and GPUs (and storage and network controllers, for that matter) equal access to a single pool of RAM in a secure and high-performance way, that would certainly be interesting, and not only or even primarily for gaming.
I imagine the PCs for the future would maybe have motherboard with two sockets: a normal CPU one and a GPU one, both having access to some high bandwidth unified memory that.
Right now the unified memory needs to be pretty much soldered super close to the chip to ensure the bandwidth and energy efficiency, but hopefully there will be some solution to circumvent this limitation in the future.
Putting CPU and GPU cores into the same physical die works for your average consumer machines, but hardcore professionals or enthusiasts would always strive for more custom configurations than existing packages can provide, like more GPU cores.
Integrated systems have come a long way for highly optimized mobile and laptops designs with specific hardware in mind. But even if everything went perfectly - DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
Then again, maybe AGI will come and hardware will fundamentally revolutionize overnight.
The technology is very new and currently in its infancy. I do believe it's the future of computer graphics though, where instead of doing "real" photorealistic geometry and lighting, software can render a very simplified scene and have a good enough photo-realistic filter applied to it so that most people wouldn't be able to tell the difference.
Obviously, I believe it'll be flexible enough to do things other than photo-realism in the future.
> In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
I think we have different definitions of "discrete" here. In my mind AIPUs are the opposite of discrete. I might be wrong here, but I always interpreted this term as a standalone GPU unit with its own onboard memory — a thing that currently fits into a PCIE slot.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
An RTX 5060 Ti 16GB card costs about $800 on Amazon right now. Running 18.4 hours a day ( off peak electric rate schedule ) it can process approximately 109.3 million tokens per month. The card has a 90 day return window. Credit card interest + electricity puts the cost at $80 total out of pocket, to use this card for 3 months and return it.
Paying an AI provider came out to $1,639.50 for the same 3 months of 18.4 hour a day usage.
You know its a bubble when ... $80 vs $1,639.50
I understand that if you're a major corporation, or a researcher that needs 100+ gpus for a large project, unless you have access to your own supercomputer, the economics will be different.
This was for "individual coder at home" scenarios.
tl;dr ... im exploiting jeff bezos, and math is racist. too funny.
I guess people have never heard of "options" before.
I optioned a video card for 90 days. My bad.
On the other hand, if you do include the cost of the hardware it doesn't seem change the overall conclusion.
They gave me a 90 day evaluation window.
They did this because it benefits them.
I paid 35% above retail for the card.
And I'm the greedy one ? really ?
Someone needs redpilled on how the monkey planet really works, which was the entire point of this post.
We can choose to spend months crying about card prices and waiting for christmas, or ...
We can choose to analyse the market, calculate risk for our projects, and move forward confidently after doing so.
Or, we can stand on the sidelines and throw rocks at the marathon contestants because they pooped on our lawn.
Choose wisely.
Edit: I do like that you came back to edit a whole bunch more into your reasoning. So I'll do the same.
I like that you describe yourself as someone who shits on other people's lawns.
I fully believe that you randomly shit on other people's lawns but this marathon you've imagined yourself in is pure delusion.
I wish you much success in life.
Thank you for contributing to current year.
MBA school calls it responsible corporate stewardship.
As for MBA school, don’t really give a shit how it is colorfully worded to attempt to excuse it.
Seems like a great way to help the species get a bit lower still.
In fact I'm running qwen3.8-flash-next with a Playwright-mcp docker and it is far better for scraping than my ChatGPT or Claude subscription, for the simple reason that sites aren't lying to Playwright.
Not to invalidate your point, just saying it's a bit of a straw man.
The 5060 wasn't the fastest/$, it just provided the most VRAM / $ as of yesterday.
Im installing it in a box with a 4070 Ti to run 2 different models simultaneously. Coder + MoE via paperclip or multica.
That probably is the cheapest per 1 GB vram right now. However, the R9700 a few months ago was only $1350/32GB or $42/1GB VRAM. And everyone knew it, because the card is now $1800 or so with supply issues.
I think the market discovered the reasonable min is 32GB. That runs Qwen3.8 Q6 with 100k+ context all in vram. 24GB does not.
You're counting on most people being willing to abuse a return policy like that. Also, the 3 month return window must've activated 3 days ago. These prices didn't start just then.
localaicost.com
quality determined by model(s) and model size(s), not card size.
no return policy has been abused, you imagined this.
the plan was for 1 card today.
RIF.
You can also keep it and return a bag of flour weighing the same.
Or even go to people houses and rob them at gun point to steal their hardware.
The exact same ones are now pushing $300/ssd and Samsung's (which was $300 18 months ago) is now $500.
The prices are insane.
What it does buy, though, is privacy and freedom (nobody can decide to "change the terms" to milk you more).
It’s akin to saying “the new iPhone costs $2.5k, it weighs 250 g, I can get a piece of scrap aluminium for $30, checkmate liberals.”
I too did a cost breakdown of buying my GPU for ~10K EUR more than a year ago. It came out "Invaluable" as I use it for analyzing my diary, financials and other private things I cannot send to random 3rd parties.
But even the best of models you can run with 96GB VRAM really doesn't come close to the hosted SOTA models, sadly.
What scam are you speaking of ? The return window ?
I don't plan on returning it, unless the economics of upgrading in 89 days favor doing so.
The point was to calculate "my risk" on the transaction.
As for the results of the calculation, I didn't invent math or risk.
For $80 total "risk", I can get 10+ programmers worth of work for 3 months. A much better gamble than paying per token.
What people don't like is the environment, and they just decided to stone my post to death for adapting to it, instead of doing the same.
People are just being people.
I'm a much easier target than that Sam guy, who locked up 40% of the planets GPU's using purchase orders, with no obligation to follow through.
He's an untouchable elite. I'm much easier to mug.
We are all opportunists.
The entire thinking behind the post, was spawned by the article the other day, about the "video game construction kit" guy, bragging about spending $3.1 million a month on tokens in an article, with $0 revenue.
Your calculation seems to have depended on "Buy this hardware with the explicit intention of returning it" process, which is a scam. If you buy hardware you do so with the intent of keeping it, otherwise you find renters, if you don't like scams.
So if you don't plan to return it, wouldn't the cost at minimum be the cost of the GPU??? Why count on returning it if you don't plan to do so?
Also even if you can somehow fit a few 100 billion parameter model(like gpt 6 luna or deepseek flash) in 16GB, and magically assume you can run at 200 tokens/sec according to your calculation, the blended cost of it would be around $100, not $1639.
This seems more like a sociopathic "hack" than an actual comment on the economy of selfhost vs inference-as-a-service.