Astra 6.2 and Sol 6.2 flop at Simple Bench

(simple-bench.com)

52 points | by algoth1 2 hours ago

16 comments

  • ricardobeat 8 minutes ago
    Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?

    Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.

    • raincole 1 minute ago
      Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.
  • 0xbadc0de5 18 minutes ago
    Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.

    I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.

    • adverbly 4 minutes ago
      No offense, but aren't you wasting your time commenting about that?

      It should be obvious that random hacker news comments are worth less than an influencer benchmark, so why waste the time commenting?

      What is your goal other than to be an even worse influencer than the very one you're complaining about?

      Are you a bot?

  • davidmurdoch 6 minutes ago
    I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about it's plans/reasoning in the slightest.
  • martinjc 21 minutes ago
    For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.
  • algoth1 2 hours ago
    Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
    • schleck8 32 minutes ago
      This is so strange. Perhaps it's recurrent depth. I don't think the new Anthropic models use it but GPT 6 does
      • dist-epoch 24 minutes ago
        Astra 6.2 is below Astra 6.
  • TrackerFF 7 minutes ago
    Makes me wonder if people are getting different models?

    So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.

  • hoppp 22 minutes ago
    All the new Gemini models perform better on this than Opus 5.5

    Is that something that can be noticed during real life use?

    I really like opus 5.5, It's perfect and I haven't tried the new gemini models.

    • zf00002 1 minute ago
      My experience is Neon is good. The others were tiers below.
    • j_maffe 18 minutes ago
      Neon has Opus 5.5 vibes in my experience
    • PunchTornado 10 minutes ago
      Neon is really really good.
    • varispeed 12 minutes ago
      For me Opus performs worse than Gemini. Even Fable is worse. Likely I am routed to nerfed models still.
  • delichon 0 minutes ago
    > SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions).

    We could create a utopia if we could screen voters for linguistic adversarial robustness.

  • Culonavirus 23 minutes ago
    6.2 is for sure the worst received update OpenAI has ever released.

    What happened lol.

    I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.

    • esperent 18 minutes ago
      I'm not finding it lazy -on the contrary my WTF per session has gone way up because it's doing loads of stuff I'm not asking for. About a third of my time with it over the past week has been making it strip back unwanted work it did, even when I have a clear plan and spec to work from, and when challenged every time it'll admin the extra work either wasn't specified or frequently that it directly contradicts something in the spec.
    • AmazingTurtle 20 minutes ago
      Absolutely underwhelming, I agree. I thought 6.2 would place them on-par with Opus 5.5/Fable 5.2 at least but nope. I'm disappointed.

      Generally speaking the whole 6 series is a flop to me. Only bright side is that it's more efficient/lasts longer/is cheaper than 5.6.

      But for now, I'm using opus 5.5.

    • j_maffe 17 minutes ago
      And you're basing this statement on the report of an AI model of the negative online comments?
  • arthur-st 12 minutes ago
    This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.
  • goodcake3963 20 minutes ago
    Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware
    • fallingbananna 10 minutes ago
      If you check out the timeline leaderboard chart, you will see that it's no comeback.

      They were always the leaders at "common sense".

  • bieganski 41 minutes ago
    I tried taking the test, but ultimately stopped at Question 4:

    There are two sisters, Amy who always speaks mistruths and Sam who always lies.

    :)

    • sixhobbits 22 minutes ago
      This is intentional I think, the models assume it's the classic riddle and choose A even though it's actually much simpler if both will always lie
    • InsideOutSanta 28 minutes ago
      I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense.

      Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.

      • pvitz 27 minutes ago
        The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one.
      • dist-epoch 21 minutes ago
        It's a simple one.

        There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path.

      • unglaublich 27 minutes ago
        And then you click one and it'll confidently say that another one is correct. The one later with the car and the glove is also bullshit.
    • xqcgrek2 17 minutes ago
      Every one of the reply comments so far has missed the part where Sam is the one who always lies. Whoosh.
  • varispeed 13 minutes ago
    I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.
  • xqcgrek2 14 minutes ago
    As the kids say, OpenAI is cooked.

    Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.

  • floornext 37 minutes ago
    [dead]