AI, activity, and outcomes in software engineering

(cabird.com)

55 points | by azhenley 2 hours ago

44 comments

  • swatcoder 1 hour ago
    Unfortunately, this is one of those HN submissions where people are just knee-jerking to the title without even checking what the submission is. Of course, it doesn't help that it's a long video conference presentation with a "handout" that's beautiful but a little hard to read through on its own.

    However, for those that are actually curious, the submission is not itself a cynical or dismissive take. The question in the title is interrogative and contextual, not rhetorical and universal, and the presentation tries to break down some of the bottlenecks and challenges that are leading to AI integration having a different impact on software engineering output than some people might expect.

    Rather than suggesting those bottlenecks/challenges are universal or intractable, it just enumerates some of them and presents some survey analyses on how widely encountered they are.

    It's actually pretty interesting and the kind of thing that engineers and those who work with them want to be reflecting on, so that they can better engineer their usage and processes around AI itself.

  • rbehrends 1 hour ago
    This is something that Brooks had reflected on already in "The Mythical Man Month":

    "One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported for industrial teams. Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced."

    Brooks argues that the coding that happened in the garage is only a fraction of what you need for the final product:

    "It costs nine times as much. But it is the truly useful object, the intended product of most system programming efforts."

    What we're seeing right now does not feel much different. You can greatly accelerate a first draft, a greenfield project, an experiment, or at least the initial iteration of one. Changing a live production system or a widely used library is an entirely different proposition.

    • jcims 1 hour ago
      This is my thinking as well.

      Low risk work is tremendously accelerated. Just look at the memes that people re building. One of my favorite right now are derivations of this thing: https://x.com/jamesonhodge/status/2118733265372860826

      People are running away with it in hilarious ways using all sorts of storyboarding, 3d environments, gameplay, etc etc.

      I work in a highly regulated environment. My AI spend is ~$300/week and it has been tremendously helpful in troubleshooting incidents(!), increasing test coverage, gaining understanding of legacy/inherited code, applying fixes, etc. But shipping new features is something that impacts us in many many many ways and the uptake there has been quite slow.

    • hgu63 1 hour ago
      Yup the test is not playing chess or dota or math proofs. Run a bank or the power grid. Then you pass.
  • 9cb14c1ec0 2 hours ago
    I am shipping so much more, but I don't know that our customers would necessarily know it. Total rewrites that needed to be done but were getting pushed off due to the scope are now getting done. Security issues are getting patched. Support ticket open to resolution time is dropping. So yes, shipping a lot faster to create a much better project rather than just focusing on features.
    • lumost 4 minutes ago
      do those rewrites really need to be done still? what are you gaining?

      In the old days we rewrote for a gain in productivity by humans or a gain in performance which resulted in non-trivial dollars. If AI is writing and testing the code, what benefits are you getting from the rewrite?

    • MBCook 1 hour ago
      That’s my immediate thought.

      As we’ve seen happening in the open source projects with the piles of bugs being reported to them so much of what’s going on doesn’t seem to be new features, but just longstanding technical debt coming to the fore.

      Maybe AI can’t write that new feature for us really fast. Maybe we asked. Maybe the code was complete garbage and it made it clear we really don’t want to keep piling on top of it and need to fix the issues we’ve known about for a long time. Adding new features at the old pace was a problem. We can’t handle the new.

      Between that and the fact that some kinds of tech debt are much easier to pay off now that’s where a lot of our “new capacity” is going. Shoring things up.

    • intrasight 1 hour ago
      > Total rewrites that needed to be done but were getting pushed off

      I am doing this now with a massive internal app. Our internal customers see this as a drop-in performance because one of the senior team members, me, is not adding new features. But we are addressing a massive technical debt in about one-fifth the time it would've taken without the use of LLMs.

  • throwitaway222 2 hours ago
    More tickets and larger tickets, full migrations that used to take 4 weeks coordinating with 3 teams are now a single PR done in a day.

    edit: if you're seeing a slow down in PRs, it's more likely that 10 of your 10x'ers are slinging code in 3-4 plus accounts in codex on the side during their shift on 8 startups.

    • smy20011 1 hour ago
      Yeah, but you will not working on migration if there is no ai in the first place.
    • para_parolu 1 hour ago
      In the past splitting diffs was an important process to speed up and simplify code review. But now I would rather make one PR per service to simplify CI and deploy. As a result PRs become much much larger.
      • wewtyflakes 1 hour ago
        I've been doing this too; it feels like a natural change due to the incentives changing.
    • bad_haircut72 1 hour ago
      If you're still writing migrations you're still not on the edge and are missing out on some good gains. We got rid of our database and now just get the LLM to manage markdown files containing all our customer PII etc. Our delivery speed has quadrupled this last week alone
  • jmull 1 hour ago
    Don't simply answer the question from the title -- you won't be addressing the substance of the presentation or really any other substantive issue.

    Either watch the presentation if you want to spare the time (a big ask even at 2x speed) or move on.

    The presentation is about the results from an actual investigation on why a particular group only saw modest improvements in shipping. It will probably be more relevant to groups more like the group studied (a large dev org) than others.

    The general idea is that there is a process pipeline between code change and shipping, and LLMs don't necessarily speed up all the steps at the same rate, especially in large, mature products.

  • 0xb0565e486 2 hours ago
    We are. A month of work turned into one day.
  • javierluraschi 37 minutes ago
    We can build an app in minutes with Claude Code, Codex, Cursor but deployment of new apps still requires a human to sign-up, browse a dashboard, create API keys, enter a credit card, etc. This means for less technical users, deployment might not even happen.

    I would want to see more cloud providers embracing fully-autonomous deployments for new users, I created this benchmark to track progress here: https://github.com/cloudishai/ai-deploy-bench

  • MisterPea 1 hour ago
    I feel like I'm going crazy with all the people questioning the impact of AI in software development.

    I think people already have forgotten how long and arduous software development was just a few years ago.

    The biggest bottleneck now is CI/CD/Experimentation with respect to engineering, and product alignment with respect to the business.

    • MBCook 1 hour ago
      I’m not sure I’ve really seen it within the community of programmers. It was certainly more there early on when the models were much worse, but I’ve seen it fading.

      I have seen it among “the general public“. I’ve seen normal people complain about things in open-source projects or other contexts where to whatever degree they accepted programming took time before now they think it’s all just a magic wand with Claude, no matter the level of complexity.

      I haven’t heard a single person at my company in management at any layer complain that we’re not working fast enough now that we have AI. One of my friends has 100% told me their management is pulling that. The management over there is on the everyone should do four times as many tickets, and use at least X amount of tokens per day. otherwise you’re slacking off train.

      It’s all in who you talk to.

      • MisterPea 1 hour ago
        I think it's an age thing as well.

        Engineers that have only started full time work post 2023 haven't experienced the counterfactual of not having AI.

        They see all the mundane business/eng bottlenecks that are pervasive now and don't truly know how much better it is now than before

    • xp84 1 hour ago
      Same. If we had twice as many PMs as our current number (we have multiple open seats) we’d be easily able to do way more with the same eng team.
    • skydhash 1 hour ago
      > I think people already have forgotten how long and arduous software development was just a few years ago.

      Was it? Behind any arduous development process, I'm ready to bet there's a very inefficient communication channel that has nothing to do with code or any technical matters, but still impact productivity. Think how hard it is to get a Definition of Done alongside the Change Request even in the business jargon.

      > The biggest bottleneck now is CI/CD/Experimentation with respect to engineering, and product alignment with respect to the business.

      It's like building crooked houses, then saying that the biggest bottleneck is inspection. Do a 5 why[0] around the "bottleneck" and the answer is likely not there at all

      [0] https://en.wikipedia.org/wiki/Five_whys

      • MisterPea 1 hour ago
        > I'm ready to bet there's a very inefficient communication channel that has nothing to do with code

        That is what I'm saying though. Shipping to production is mostly bottlenecked by product/business leaders who haven't described a clear vision/communication of what is wanted.

        I've been a SWE for far longer pre coding-agents than post. The actual development of software was arduous. So much of tech was/is a long list of heavily scoped down tasks that eng takes an order of magnitude longer than today.

        A quirk of coding agents is how they assess timelines. Ask it to scope a project you want done and it will answer in engineer days from pre-ai days. In actuality it can be done with a prompt and a few hours in claude

        • skydhash 1 hour ago
          > So much of tech was/is a long list of heavily scoped down tasks that eng takes an order of magnitude longer than today.

          I disagree. Based on my personal experience, a lot of time is spent on fine tuning code and double checking the docs. The first pass can be very quick especially if you already have sample code (from the docs or other parts of the project). But when it came to fine tune, that's where the other 90% of the time is going.

          A lot of daily activities of engineering is massaging code, either from bug reports or requests after someone tested the features. And a lot of time is spent gathering details, comparing it with what already built, and then apply correction.

          From what I've seen with people doing agentic coding, that feedback loop in the development phase is very superficial. Then the most important feedback loop (the one with the user) is distorted as the software is constantly changing.

    • 01100011 1 hour ago
      Not everyone has the process and positioning needed to take advantage of AI, so it’s not surprising that some people swear it doesn’t help. But those people need to understand that other people do have the process and positioning and AI is dramatically changing things.

      Personally, we’re just beginning to ship faster because we’re a big ship and turn slowly. We’re still getting up to speed but we’re also bogged down with bug fixes, security fixes, and the review bottleneck. Reviews are especially tough now that every commit is much larger because it is now much more correct than the crap we were submitting before AI.

      Also, many people are simply lying and hate AI. There are people I’ve encountered who look for any evidence to justify not embracing it. It’s frankly tiring but I understand some people cannot handle change and are frankly terrified of what is happening to their careers and their concept of self-worth.

  • choult 1 hour ago
    The excitement of LLMs has been around the productivity of the engineers, not the quality of the projects. Traditionally, we have bottlenecks at review and QA stages for new work, and a lot of orgs turn on the firehose of new code without fixing those bottlenecks.

    If you want to drive at high speed, you need to build a highway first.

  • ferguess_k 2 hours ago
    We are definitely shipping faster. Does that turn to better products? Not necessarily.
  • smy20011 2 hours ago
    We definitely see more broken features that was removed after a short amount of time.
  • jmathai 1 hour ago
    In order to ship faster, we need significant improvements in efficiency when it comes to reviewing code.

    That may or may not include shipping without reviewing code. Something I think is a real option to consider as LLMs and agents become more capable.

    The reality is, any point where a human is involved is now a bottleneck. Even when that human is using an agent to write code. This will apply to the review process as well.

    It feels this way, doesn't it? At least until we focus on something besides pure efficiency.

  • throwaway7783 2 hours ago
    Who is not shipping faster? We definitely are.
  • EQYV 2 hours ago
    Aren’t we?
    • jayd16 2 hours ago
      Remembering that LOC and commits don't count, what metrics or even personal experience show that users are getting more features faster?

      At least in entertainment I only see cancelled products rapidly increasing.

      • rectang 1 hour ago
        This last week, I was able to delete considerably more code than I would have been able to before LLM assistance.

        Specifically, I was able to consolidate some CSS because I could ask Claude to calculate the cascade before and after and verify that nothing had changed about rendering.

        I’m sure there was a way to do that before by scripting a headless browser, but I didn’t have to learn that tool to unlock this capability.

      • adriand 1 hour ago
        I'm shipping new features to my SaaS customers sometimes as quickly as the same day they bring it up. These are usually fairly small, obviously. I actually posted this story on LinkedIn, as one does:

        > Customer email, 9:42 am: "I have a partner asking for our SSH2 public key. I tried using Tediware help but eventually it said to contact support. Are you able to assist?"

        > Response, 10:47 am: "You need to generate a key pair yourself and keep the private key. Tediware does not issue SSH keys. Here's what you do. [instructions follow]"

        > Follow-up response, 12:20 pm: "Generating your own key pair is now part of the app."

        > Time from support request to new feature designed, coded, tested, deployed and announced: 2 hours, 38 minutes.

        Side note, this feature has been solid in prod ever since (that story is a couple of months old).

        Obviously, some things take weeks still, but they're hard. I just finished writing an X12 translator/parser, from scratch, in Rust. The old translator/parser was written in Ruby and ran documents at 175 kb/s. The new one handles 1 MB/s. It took me seven weeks of on-again/off-again dev work, fuzzing, testing, etc. It went live in prod yesterday.

        • jimbokun 33 minutes ago
          Paul Graham talked about fixing bugs on the production server while the customer was still on the phone, because it was written in Lisp.
        • skydhash 1 hour ago
          > > Time from support request to new feature designed, coded, tested, deployed and announced: 2 hours, 38 minutes.

          What about risk assessment? Are the customers guinea pigs?

      • ben_w 1 hour ago
        I'm sure the release cadence on browser updates has gone up; and thanks to all the CVEs getting found now, there's definitely real value from them, even if we simply wouldn't have known about all the risks if it were not for the same AI solving them.
      • tikotus 2 hours ago
        I definitely see more broken features!
        • marcosdumay 1 hour ago
          Maybe this time we get people to abandon the big, dysfunctional companies that were never actually able to write software.
      • meric_ 2 hours ago
        For entertainment, it's definitely been huge for gaming. In Roblox the majority of games now are AI generated. Of the top 100 games, 80 used AI in some way or another and there's 70,000 AI generated games total. Roblox has a huge number of players too (~380 MAU) so tons of people are playing AI generated games.

        I think it would be more widespread if there wasn't as much pushback against AI. Big Walk disclosed they used ai to fix a bug and it immediately got review bombed. Kids on Roblox don't care as much, so you can see just how transformative it has been there.

        • benoau 1 hour ago
          The IGF really dropped a grenade into the discussion heh. What happened was IGF clarified that "using AI" includes using it for coding "help" and research, which Big Walk had disclosed. This does not align with Steam who doesn't require disclosure of those particular uses, and in general using it for programming has been arbitrarily deemed OK by players and the industry.

          Personally I think Steam should either get rid of the label or expand it to any usage because the carve-outs are arbitrary and pervasively widespread in use - I don't find it believable House House's developers are still almost entirely coding by hand and only leaning on AI for the hardest problems lmao. The stigma isn't sustainable once full disclosure is normal, and then games can be judged simply on their merit either they're good or bad.

          Failing that it's hard to imagine what IGF looks like next year because everything currently in development will be using AI in some fashion, either they're going to have no candidates or no honest ones.

          > “Programming is a creative discipline whose labour deserves the same respect we extend to any part of making a game. This extends to the past work used to train these models without the creators’ consent.”

          https://www.videogameschronicle.com/news/big-walk-studio-say...

        • moron4hire 1 hour ago
          Roblox is played exclusively by children. Children have the taste nuance of insisting chicken nuggets with ketchup are the epitome of cuisine.
      • AnotherGoodName 1 hour ago
        It feels to argue the point that ai doesn’t have visible outcomes you’d have to go down the path of the no true scotsmans fallacy pretty hard.

        “Sure there’s a massive increase in the number of apps on the app stores, there’s new examples of decompiled and recompiled games popping up all the time not to mention all the vibe coded games, many companies are reporting they have new bespoke vibe coded tools to do some task they used to do manually, people are just cloning popular apps like photoshop with bespoke customisations to their needs..” etc.

        I already know the tiresome response to any such example given by people of this mindset. “Yeah but those aren’t TRUE examples of users getting features faster!” Whatever.

  • w0m 46 minutes ago
    > why aren't we shipping faster now?

    We are. ~300% as of late-summer when I calculated. Models have gotten markedly better since then.

  • bigblind 1 hour ago
    I'm shipping much more when I don't care about code quality. I've built a bunch of fun, greenfield side projects. And prototyping at work is also much faster. But when it really comes down to the details, conversations with colleagues, and really delving into what we want to build, designingthe userexperience, getting the details right, still take time.
  • throwaway0q5347 1 hour ago
    At least in our case, we are shipping faster. The rate of bugs has gone up, but still proportional to the behavior changes.
  • mr_roboto 1 hour ago
    I'm shipping faster, but not as fast if the output was the same as before. AI allows for deeper code, all the nice to haves you wouldn't normally do. Test coverage is now fully complete as well, and I mean full tilt mocks and every edge case, not just the happy path plus bugs found in testing.
  • hackboyfly 2 hours ago
    I work for a midsize company within retail and we are definitely shipping faster?
  • ThomPete 16 minutes ago
    We are.

    At least at Block our velocity have gone way up. One big unlock was designers building their own features not making it a developers problem.

    Our biggest bottleneck is PR reviews but we are also gradually solving that.

  • xnickb 1 hour ago
    I think the recent article by Rober Haas on why Postgres release is delayed is giving a very good insight into the general "why".
  • mystifyingpoi 1 hour ago
    I am writing code 5-10x faster and I can feel it go, but I'm still waiting 2 days for the IT to install something on my useless locked down laptop, still waiting a week for a trivial corporate firewall change, still waiting 2-3 weeks for business to make up their minds about whatever the hell they want to sell now. So I am shipping faster, but actually not.
    • 01100011 1 hour ago
      Yeah, our CI/CD pipeline is still a dumpster fire. Management refuses to use AI even though it would help them more than us. Our build team hasn’t really used it to improve our build times.
  • inanutshellus 1 hour ago
    You're not shipping faster?! holy moly. I'm orders of magnitude faster with fewer scope-cutting calls.
  • crsv 1 hour ago
    We are shipping at a significantly faster rate both internally and externally.
  • koliber 1 hour ago
    I don’t know who your “we” is, but the “we” I’m involved with is shipping faster.
  • jcmontx 1 hour ago
    We're shipping at the same rate but with a team of 10 instead of 24 we used to be
  • raincole 1 hour ago
    Who are we?

    Google Search: 1998

    Gmail: 2004

    Google Map: 2005

    Google Docs: 2006

    Chrome: 2008

    vs

    Claude 3 (+ tool calling, web search): 2024

    Claude Code: 2025

    Claude Design, Sheet, Data: 2026

    "We" (whatever it means) are shipping faster now, no?

  • blinky88 1 hour ago
    We're shipping bugs at light speed
  • zerop 1 hour ago
    AI has accelerated coding but what happens before and after coding has not accelerated much.

    Pre coding : Requirements, grooming, solutioning (cross product specially), design, LLD

    Post coding: Cross product testing, E2E integration testing, CRQ approvals, Deployment, pilot, roll out.

    SWEs don't just do coding. AI has only solved that and testing to an extent.

  • HeadOfProbing 2 hours ago
    We definitely are.
  • ftchd 2 hours ago
    Because we spend time writing clickbait articles
  • vcryan 1 hour ago
    We are shipping faster. Shipping speed depends on a variety of factors. That said, the ability to generate and validate designs, code, tests, data, performance and security evals, etc is increasing the speed of delivery for many organizations - I'm a consultant and I see it every day.

    I also see companies that don't know what they want to build, suffer from toxic culture etc -- they are not faster because their core problems never had much to do with software development.

    Sure, there are plenty of developers clinging to the "old way" or using AI in ways that add complexity and slow things down but I don't see this as the dominant group.

  • 28304283409234 1 hour ago
    The bad guys are. :-)
  • prmph 1 hour ago
    The question should be: why isn't software improving?

    Claude Code TUI, probably the poster child for AI-driven coding, itself is still going to sh*t.

  • thehamkercat 39 minutes ago
    we are...
  • innagadadavida 1 hour ago
    Most if the issue I’ve faced working on production software in team environment is to be able to defend you PR and story the changes and understand it to a level to be able to defend it. For PoCs and demos it can accelerate a lot but not for many other use cases.
  • IAmGraydon 1 hour ago
    I'm a one-man operation with an equities markets analysis product. I'm definitely shipping faster. Fixing bugs and adding features literally 100x faster as well as implementing some very complex math that I likely could not have done on my own previously.
  • jimbokun 1 hour ago
    I don’t want to watch a 21 minute video right now.

    But does it say what quantifiable evidence they have to show software teams are NOT shipping faster now?

  • inshard 1 hour ago
    We are but it’s weighted towards the independent and never-coded-before side of the ecosystem at the moment.
  • nba456_ 2 hours ago
    We are.
  • fizx 1 hour ago
    Speak for yourself, mate
  • paganel 1 hour ago
    The trillions-dollars question is how come software itself doesn't get improved by (at least) an order of magnitude, if not more.

    More to the point, the price of RAM that has gone berserk. Easy solution, let's then make our OSes, browsers and almost all the relevant software that we use day to day an order a magnitude more nimble, for example if a Chrome instance now eats 1-2GB of memory then make it eat at most 100-200MBs of said memory while doing the exact same things (if not more). The much lauded AI revolution in code-writing hasn't created that, not sure if it will ever do, it will just add slop after slop after slop, in fact it wouldn't surprise me if the memory requirements for something like Chrome won't increase as a result of the recent vibe-coding mania.

    But by this point the AI zealots are already running the asylum, so nothing to be done.

    • TacticalCoder 1 hour ago
      > The trillions-dollars question is how come software itself doesn't get improved by (at least) an order of magnitude, if not more.

      The sad thing is those using AI to code (and I certainly do) don't realize they're producing more but just more of the same turds they were already producing.

      And in addition to that they're producing turds nobody gives a shit about (ah!).

      Where, indeed, are the both less memory-hungry, less CPU-intensive and more secure rewrites of the pigs we work with on a daily basis?

      Nowhere to be seen.

      Because the AI zealots are way too busy adding crap features over crap features.

  • rubicon33 2 hours ago
    We are. This has got to be clickbait.