Neon is a contender. Neon "high" appears faster and cheaper than Opus 5.5 "medium" with similar results on long-running language porting work in a large monorepo.
The speed is the biggest win IMO. When Carbon came out, I tried to give it a shot, but it was so slow that I went back to Opus just for productivity, nevermind quality. Today I started Neon on one worktree and Opus on another, and Neon did so well that I have now switched for both.
Code quality looks great. Comment-wise it has its own Claudese(exclamations like "this function has a concurrency bug and it's a sharp one!"), but overall I like reading it more than Opus. Might just be because I am not yet tired of its quirks though.
Oh, after a productive day on this particular work I had only spent $30 of Neon vs. $70 of Opus yesterday. YMMV
Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.
But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.
looking at the graphic below the table, it does seem like data is missing for Neon which is present for Fable, and isn't penalized for the missing benchmarks.
I've been impressed with my usage of Neon so far. Early impression at High and Extra is near enough to Opus 5.5 Extra for me. Usage has been pretty good for that effort level too.
How do I use it? Do I have to create a Google Console account and navigate through 15 menus in order to generate an API key? Or are there another 7 different ways to achieve similar outcomes?
The subscription is the most cost effective way to use. But if you want to avoid the monthly cost you can create an API key and pay per use.
There are a few ways to create/use an API key. Through Google cloud, through vertex, but by far the easiest way is to use AI studio. https://aistudio.google.com/ pretty good platform to try out any Google model or just create an API key and use in whatever you want without having to deal with Google Cloud UI.
Alternatively you can get from other sources as well such as openrouter.
I wouldn't trust any particular one, but obscure / indie indexes are going to be less affected by benchmaxxing than the well known ones, so it's a positive sign for Neon that it is performing so well on a lot of them including this one.
Is it just me or all Google models are terribly bad?
They all seems benchmaxed to hell and producing total crap compared to Claude or openai models from my own experience (mostly coding)
After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.
It's just you. Since Argon, the Gemini models have been producing good results for me side-by-side with Opus/Fable models on the same prompts. Not always winning, but increasingly so with Carbon/Neon.
I think it's just you. Carbon is excellent but extremely slow. If Neon is better than or equal to Carbon while also being a lot faster, that'd be a great model. I haven't tried Neon, yet, but they seem to have understood the assignment with the latest round of models.
this week I ran out of codex astra quota and switched to antigravity, boy is Gemini Carbon amazing model! It actually helped achieve a breakthrough I was stuck on for weeks with Astra.
And tokens/task is very high. They don't seem to have figured out making the models usable in a typical interactive workflow, ie., they're in a Opus-4.7 era.
Yes. Every Google model so far has been disappointment. And their biggest moat, the integration into other google products has been laughable. Today I've asked gemini in gmail to download all attachments from one e-mail address and it turned out it's unable to do it. It's unbelievable how incompetent they are, Google has such a massive edge over OAI and Anthropic and it's still losing.
The speed is the biggest win IMO. When Carbon came out, I tried to give it a shot, but it was so slow that I went back to Opus just for productivity, nevermind quality. Today I started Neon on one worktree and Opus on another, and Neon did so well that I have now switched for both.
Code quality looks great. Comment-wise it has its own Claudese(exclamations like "this function has a concurrency bug and it's a sharp one!"), but overall I like reading it more than Opus. Might just be because I am not yet tired of its quirks though.
Oh, after a productive day on this particular work I had only spent $30 of Neon vs. $70 of Opus yesterday. YMMV
But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.
looking at the graphic below the table, it does seem like data is missing for Neon which is present for Fable, and isn't penalized for the missing benchmarks.
There are a few ways to create/use an API key. Through Google cloud, through vertex, but by far the easiest way is to use AI studio. https://aistudio.google.com/ pretty good platform to try out any Google model or just create an API key and use in whatever you want without having to deal with Google Cloud UI.
Alternatively you can get from other sources as well such as openrouter.
https://one.google.com/about/google-ai-plans/ had a 75% off on the Pro plan ($5/mo for 3 months) and limits are decent.
Very high hallucination rates too.