9 comments

  • paxy 22 minutes ago
    I don't see any difference between the conversations around self-hosting local AI and self-hosting regular software, which have been going on for decades.

    Yes, there are a hundred valid reasons to self-host. But 99%+ of consumers aren't going to care.

    People don't even care when the cost of hosting is ~free, like a simple home server. How many are ready to buy thousands of dollars worth of AI hardware, which could very well be obsolete a couple of years from now?

    • wry_xy 10 minutes ago
      The self-hosting crowd, Linux fans, OSS contributors, etc, are our own worst enemies.

      If we care about self-hosting and open source and decentralization because we believe it leads to a better world, and not simply because we want to be control freaks and snobs, then we need to learn how to build shit that is actually convenient and usable and understandable to normal people.

      I think it's possible. I think many people would want this, and care, if we could package it up in a digestible way.

      • goda90 7 minutes ago
        The problem with open source is that it is often written by people who are fixing issues they have or making features they miss. To get to the point of contributing, you have to go through the initial setup process. Once you've done that, it's not longer a problem you care about, so you're less motivated to contribute code to make that process easier.
      • nemomarx 9 minutes ago
        There's a decent opportunity where AI coding agents seem good at creating a very "average" and understandable UI. Maybe not perfect, but probably better than Gimp?
    • jbjbjbjb 3 minutes ago
      It’s more about self-hosting the hardware which is more desirable than thin clients.
    • barbazoo 10 minutes ago
      It's so sad that we are not empowered to act as more than just consumers as if it was some kind of shared religion. How do we get out of this cycle?
  • giancarlostoro 56 minutes ago
    If Nvidia didnt abandon consumer GPUs we all could be running better models locally by now. The moat is companies like Nvidia not releasing better consumer hardware so they can partner with bigger companies so they can sell us overpriced hardware.

    They release a new RTX series every two years, last time was 2022.

    Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.

    Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.

    • codehorses 50 minutes ago
      Not defending Nvidia here, but the RTX 50 series was released in 2025, was it not?

      Regardless, they aren't planning on sticking to the two year release cadence any more which does suck.

      • paxy 34 minutes ago
        RTX 5090 - release price $2000, average price today $6500

        RTX 5080 - release price $1000, average price today $2300

        Nvidia also stopped selling its own version of these cards directly to consumers and is happy with letting partners and resellers/scalpers control the price.

        So it is pretty obvious that the company doesn't particularly care about this market segment.

      • Buttons840 49 minutes ago
        Ironically, if consumer GPU progress stalls, then slowly PC becomes the best console. Everyone has the same top-end GPU that can run any game.
        • ghostbrainalpha 21 minutes ago
          Or game streaming actually becomes a thing. Because the best new games only play in the cloud on computers we could never afford.

          Almost like going back to the Arcade era except we all pay monthly subscriptions instead of 1 quarter at a time.

          • unrented7977 18 minutes ago
            Game streaming isn't going to happen. It's been tried several times now and failed every time.
    • Forgeties79 49 minutes ago
      No gut feeling necessary. They announced weeks ago that they are going to be $1200-$1800 (Xbox specifically, PS6 somewhere in that range too).
      • lubujackson 38 minutes ago
        Not to mention the required monthly subscription... Income inequality like 1929, might as well bring sharecropping back.
    • scronkfinkle 55 minutes ago
      > Hoping AMD steps up to the plate

      AMD's aggressive marketing as an AI-first company leaves me pessimistic about that.

    • jjtheblunt 49 minutes ago
      > we all could be running better models locally by now

      Apple silicon says "hold my beer", which I know because i've been doing this on my 3 year old quite beefy mac studio, for example. it's good enough for many tasks.

      • vadansky 39 minutes ago
        Can't wait for M7! I hope the rumours are true.
  • herf 18 minutes ago
    I love local AI, but even so, I admit the economics point the other way. Unlike gaming, AI is a uniquely good match for data centers because the applications today aren't latency-sensitive at all (except perhaps DLSS). Most importantly, utilization is massively better with always-on servers doing parallelism (in-flight batching). Also in some areas (like California) our local electricity is quite expensive, so moving the work 100ms away can help. The privacy implications are horrifying, though.
    • younglunaman 4 minutes ago
      I tend to disagree here. Think about how useful it would be to have a nice llm in real-time debugging your applications loaded entirely on your computer. That sounds pretty sweet to me. I don't see how that's not really cool / immediately useful to you?

      Maybe your point is the majority of the time that's not needed, fair, then maybe the approach is both?

    • drdexebtjl 9 minutes ago
      [delayed]
    • musha68k 12 minutes ago
      The economics point the other way because capital is aligned against you. Should this be a bubble that pops we will hopefully see prices come down again.

      MooresLawIsDead and his spidey web / senses thinks it already popped:

      https://youtu.be/YJeK2gumpEI

  • maxdo 50 minutes ago
    i'll share a lonely voice on hn, i care very little about local ai.

    I feel like people who consider it as an option thinks of narrow scenarios vs ai is part of our life like internet is. How do you things on mobile for example with your local models?

    AI is very competitive, I use various models, I like to be on cutting edge and to be productive, its the opposite from doing something permanent, and not changing, where local models can be good. I like to do things on the go, and the last thing any normal person would want is to have DC at their basement.

    • segmondy 3 minutes ago
      I run local AI, there are tons of local models with no cloud alternatives, the only way to unlock and have these functionality is by being able to run local. So you are not on the cutting edge by using cloud only.
    • cableshaft 20 minutes ago
      I'm not willing to feed my personal data (emails, files, etc) to a non-local AI model (well, beyond what I can't avoid, like Google and my Gmail account).

      But I would like to have an AI that does that, like synthesize my journals and notes and provide a local personal knowledgebase I can query, things like that.

      I don't think I have a good enough GPU to manage a local model powerful enough to be worth the effort just yet, though, so I'm waiting a bit longer.

      For coding and bouncing ideas off of though? Yeah sure, online is fine.

    • ale42 38 minutes ago
      I think that many people want similar things. But (at least personally), not giving my data to BigTech is more important that this.
      • maxdo 27 minutes ago
        out of curiosity, is there anything special your data has?
        • montroser 8 minutes ago
          It's not about anything special -- it's about not wanting to have big tech sharing my personal details that will allow them to influence my world view and how I think, in exchange for political/corporate money.

          Like, you know it's already happening now on Grok that they will sell influence. And it can be so effective if they already know what's inside the head of the person whose mind they're trying to change. All the sudden we have masses of people believing vaping isn't that bad for middle school kids, or that we really should just go ahead and trust Gemini with the most intimate details of our personal lives...

        • cableshaft 19 minutes ago
          This sounds dangerously close to saying "If you have nothing to hide, you've got nothing to fear."

          https://en.wikipedia.org/wiki/Nothing_to_hide_argument

    • betterunix2 5 minutes ago
      It would be a lot easier to do local AI if we were not using natural language as the interface. In fact there is already plenty of local AI, including on mobile, that most users do not even realize is there -- because nobody has "conversations" with those models.

      Niche? Sure, but let's be honest here, a lot of AI use right now is forced, which is usually a sign that the product is not actually in demand or has questionable value for users. The big, natural-language-interpreting models are also niche -- a minority of people find them extremely useful and a majority find them to be "meh" at best.

    • jakswa 20 minutes ago

          How do you things on mobile for example with your local models?
      
      You think you will be doing those things? And if you do keep doing those things forever, won't AI be smart enough to figure it out for you, whatever needs to happy locally?
    • neutronicus 5 minutes ago
      No, I'm the same way. I started trying to spec out a local AI box, looked at the numbers, and it felt like trying to run my house off a generator.

      Obviously worse than the centralized option, and not the hobby I want. I'd rather just use Cloud tokens help me do what I actually want to do.

    • codybontecou 38 minutes ago
      Whisper and other stt models work fine on mobile.

      Mobile is far behind frontier coding agents but they are already solving valuable problems with competitive quality and privacy, one-time payments as viable options.

    • NoidFonsense 30 minutes ago
      I just set up a wireguard tunnel to my home network
    • dofm 22 minutes ago
      > How do you things on mobile for example with your local models?

      The same way you can screenshare or forward from your local Mac? Paseo, LM Studio and LM Studio Bionic all support this kind of tailscale-ish access to local models from phone apps. (Haven't tried any of these myself)

      I do, personally, have "narrow scenarios" for AI. I have to use it for work, and want to be informed and engaged on that level, but I want to keep it out of my personal life as much as is practical.

      If you don't, that's fine, but I reframed my mass social media use down to an extremely minimal pattern in 2014 and I am healthier for it; I intend to keep the same distance from AI in my hobbies, for several of the same reasons.

      Paradoxically, for me that has involved taking the control over it that local models offer. I use the smallest, simplest things that get the job done and I try to retain my engagement with my work.

      It's all pretty low-effort and low-tinkering; no more than setting up local VMs for dev work. Old M1 Max macbook, Unsloth, Pi, a model in the Muse Glimmer/Qwen 27B/Nemotron Bolt sort of bracket.

      Good enough to be a rubber duck that talks back, write the minimal kind of AI-generated support code I want, and teach me how it all works.

      I might one day buy an M5 Pro Mac Mini, or an M5 Max Mac Studio, or stick an R9700 in a homebuild PC, or I might just use openrouter for the few things the Macbook won't do.

      I am a cynic: I believe AI has some value, but I also believe it should be a commodified technology rather than a branded SaaS subscription product, and it should demonstrate that value without me letting a couple of large corporations FUD me into pushing it into every corner of my life at once, or simply handing over control of my job to it.

      It is a matter of personal discipline, for me: I am going to resist the endless FOMO of new model benchmarks, learn how it works, keep it in a box, and take it out when I need it.

    • nater5000 36 minutes ago
      First, I'll say that I, too, care little about local AI. I do like toying around with local AI, and I do see how it can be a very important option in some contexts. I also loosely agree that it's good for keeping frontier models on their toes, prevent this technology from being locked away behind a paywall, etc. But for me, as of now, they're not very useful, and I use AI quite a bit.

      However, I do think your perspective on local models is a bit narrow. For a lot of people, they want AI tools but do not need cutting edge AI. They'd prefer to just have their own AI which is capable enough and under their control. They'd run it on a small server at home and connect to it as needed for their work (including on their phone). It's worth noting that much of the consumer-grade infrastructure definitely doesn't exist for this, but one could blame the lack of consumer-grade hardware as the source of this rather than just a general lack of demand. You can definitely see people picking up on this by running their various assistants, like OpenClaw, on Mac Minis from their home. It's definitely not accessible or cheap enough for most consumers at this point, but it's also not impossible to imagine us getting to the point where it is more ubiquitous.

      I'll say, personally, that I run Qwen 3.8 27B on an RTX 3090 from my home server, and it operates pretty well for what it is. I'm certainly not giving up my Claude subscription anytime soon, but it is nice to have the option and I can chat with it directly on my phone securely (using Tailscale) and using a nice UI.

      Regardless, I think the whole point of these kinds of discussions is to point out that, right now, local models aren't viable, but not because they can't be viable, but because companies are actively preventing them from being more viable. I can see for myself that they work well enough that proper investment could easily push them into the "good enough" territory for many users, and I suppose it is inevitable that more and more people will end up running local models in one way or another. But I definitely agree with the sentiment that most consumers simply do not care about it. They don't want to deal with hardware, they don't want to think about software, and they're happy to pay $20 per month for a subscription that "just works." I definitely don't blame them.

    • lubujackson 24 minutes ago
      I agree. It reminds me a lot of building your own desktop computer from parts - it started as a requirement, then a cost-effective choice, then became for power users only, and now it has little place beyond tinkerers/purpose-driven/privacy tinhats.

      I refused to use Macs for a decade because of how offensive I found it that they locked down their computers from even adding RAM yourself. But this is the direction of the tide and swimming against it will only continue to become more and more arduous. At best, we will have some Tor-like hosted AI, but I am not holding my breath or wasting my money/energy.

    • yieldcrv 18 minutes ago
      I disagree, I think local is the supercycle here once it gets out of enthusiast territory

      We have LLMs baked directly into silicon getting 20,000 tokens/sec with low power consumption, Taalas was bought by AMD weeks after that demo. In that world, people get a new LLM only on hardware refreshes, and only after LLMs are good enough to be in stasis for years

      Apple’s M7 Pro/Max if it meets promised will be very expensive, I see some enterprises rolling them out to some staff

      Some language models in the Apple Ecosystem are already baked into iOS and MacOS, and its ubiquity is invisible

      Thats the direction I see things going, but like not anytime soon to alleviate supply chain crunch. 2029, 2032

  • jkingsman 36 minutes ago
    Even dumber local models that are heavily quantized to fit on consumer GPUs can be lovely as a dimwitted household assistant. A lot of people look to offload things they already do to LLMs, which can be great with frontier models, but you've got the highest standards and preferences for tasks you already do, so it's the most likely to fall into a "should have just done that myself" outcome. I've found great success in instead lowering my threshold of things I'd take time to do -- I wouldn't spend time on it, but I'd spend 60s prompting an LLM to take a crack. I'm fortunate to have the hardware to run Qwen 3.8 Flash Next at q8, but most of the tasks I have it do would be totally doable for a mid-quant Qwen 3.8 27b.

    Some recent tasks I've thrown it:

    * Go through my blood test data (from EHR XML dumps from prior years) and give me some over-time charts for consistently-sampled measurements

    * Go through my Good Articles bookmarks, epubify them, and drop one down onto my ereader each morning for my commute, then archive the offline version into cold storage

    * Go through my music library, identify my most-listened low-bitrate tracks, and source or assemble a shopping cart to get high res versions

    * Analyze local tax/foreclosure auctions for vacant land; grab aerial screenshots, tax data, ownership data, and compile a weekly report of things maybe worth bidding on based on some specific selection criteria (fun facts: if you're willing to accept some headaches, 5+ acres can be yours for under $5k, if you're in the right area)

    * Dump my grocery store loyalty data, analyze my purchasing habits, and prepare a weekly instacart for my approval that aims to sync with usage habits. Tweak and learn volume/order adjustments over time.

    * Given a KVM that can mount ISOs onto the host, reimage two old laptops of unknown status with Debian and install Minecraft so my nieces and I can have a LAN party

    * Organize my music (high confidence MusicBrainz Picard matches), movies/TV shows, ebooks, and video game ROM collections with metadata and good folder organization

    * Archive bookmarked fanfiction for my offline library (slowly, and respectfully, and with an actual browser. The browser is anti-bot-countermeasured to the gills, but QFN regularly solves hCaptcha and Cloudflare turnstile challenges. Once I gave it the ability to drag, it even reasonably reliably solves drag-shape captchas. A little while back it made a purchase on eBay which is NOT a bot friendly site.)

    Would I personally spend time on doing most of these? Generally not. Could an LLM screw them up so badly I regret it? Also generally not. I get the luxury of a psuedo-assistant that doesn't judge how menial the tasks I send it are, and I save time. With a local LLM, my privacy concerns are far less than with frontier models, and local abliterated models are less (or just not at all) twitchy around things that frontier classifiers balk at.

    I'm hopeful that big projects like Muse can help to normalize menial task offloading, and that local LLMs provide the bridge for more technically minded folks to use them even more aggressively and limitlessly (as in compute limits/subscription costs) and cheaply.

  • philipodonnell 37 minutes ago
    I think one of the more compelling reasons for personal-local AI is that when two services you use add their own AI that only works in that service it creates the need for a third AI to bridge those two. But any service the proposes to sell you that third AI will be blocked by the other two, thus, only people who can create their own local service will be able to bridge them, and that will be the new version of DIY.
  • Betelbuddy 2 minutes ago
    [dead]
  • dahunt33 20 minutes ago
    [flagged]
  • routebraid 15 minutes ago
    [flagged]