I had Ox Alpha working on coding tasks for a couple days non-stop, via OpenRouter and OpenCode Zen. It was able to complete tasks at a level that I'd put between Sonnet and Opus. It makes few mistakes, but is not that smart.
The main issue for me, is that it degraded into a doom loop several times. One of them was running the same bash command about a thousand times. The last model I've used that had this problem was Mimo 2.5, which is quite dated at this point. As a result of this, you cannot leave it unattended / not usable for agents.
I usually see doom loops when working with quants. Likely theyre trying to maximize the viability of a efficient model quant that can bw upgraded. Like cutting coke to get crack, quantiry over quality.
Claims about Ox Alpha performing at Fable level were from the social media hype cycle. Everything new in the LLM space brings a wave of influencers hyping it up.
It is a capable small model, but it’s not frontier level. The interesting part will be seeing the model size, how it responds to quantization, and how fast it runs on the kind of non-server hardware that we can buy without selling a kidney.
That benchmark is super sus. Until someone pointed it out, the top performing open weights model was a Kimi K3 fine tune from their sponsor (abacusai/Smaug-Agentic). Now, it's not on the list.
Even if it weren't slopped together, 65% vs 80% on 10 tasks just isn't a significant difference. For 80% power to distinguish at a significance level of 0.05, you'd need more like 140 samples, if those were the true success probabilities.
The number one problem in LLM benchmarking is that people try to draw conclusions from sample sizes far too small to conclude anything but "it works sometimes, it fails sometimes, hard to say which is better." (The number two problem is that people run benchmarks blindly without checking that they measure something meaningful.)
I really want to see hard evidence of distillation before I buy into it. Seems like a lot of sour grapes over not having the sort of lead assumed. In this field, it has been shown repeatedly that leaps in performance come swiftly and without notice.
Z.AI is the only provider for GLM 5.3 on OpenRouter. I don't see 5.3 on Hugging Face. Not sure if this new model is "full GLM" or something smaller, or if they will like Moonshot AI publish weights but put restrictive license [1], which will again leave Z.AI as single GLM model provider on OpenRouter.
Rather than a pelican, for fun I showed it a couple of screenshots from Niu Lai and asked it to create an SVG inspired by the images. I explained a little about how the movie had been made by a mother & son team, initially derided but then went on to surprise cult box office success. It came up with this:
Created in a single turn - but technically not a "one-shot", because I gave it a tool to convert SVG to PNG so it could visualize what it had made. I asked it to keep iterating with tools during the same turn until it was happy.
I've also been using Ox Alpha for tasks that better resemble real work, and I'm really enjoying working with it. I've downgraded my Anthropic account so I can put some budget towards Ox Alpha instead, with the rumors that this one is going to be cheap. Opus & Fable are still better at getting large tasks / features done autonomously, but Ox Alpha can work autonomously too, and it's fun. I'm enjoying working with Ox in a way that I'm just not enjoying talking to the 5.0 Anthropic models. (As much as I don't want to say that, as someone with Claude /stickers on their laptop.)
I'd be interested to know what was going on with it during the public test as there were numerous reports of it improving considerably at tasks it was asked to do early on in the test compared to later in it.
It's logical to serve the best version (quant) of the model at the beginning so that users keep testing it. It is also reasonable to think that the developer of the model tried to test various quant levels by gradually degrading the model's capabilities.
1. Just variance in pass@K. If you prompt any model multiple times you'll see a large variance. N=1, but I find chinese open source models have a higher variance than higher-RL'd models like fable/opus.
2. They legitimately shipped a new RL checkpoint over the 7 days, which I find hard to believe.
> The company on Wednesday confirmed speculation that the Ox Alpha model is a new iteration of its GLM series and said it will release the weights for it tonight, in response to queries by Bloomberg News.
There's a lot of brand confusion among the Chinese models right now. Kimi, Qwen, GLM, Z.ai, Ox. We might know the difference (or I should say, someone does because I'm losing track already) but these models have no chance at end user penetration and loyalty until there's a single focused survivor.
It took me a year talking about it until my wife knew that ChatGPT and Gemini are two different things.
There's a lot of brand confusion among the American models right now. ChatGPT, Claude, Gemma, OpenAI, Meta, Google, Muse Spark, Anthropic, Microsoft, Gemini. We might know the difference (or I should say, someone does because I'm losing track already) but these models have no chance at end user penetration and loyalty until there's a single focused survivor.
It took me a year talking about it until my wife knew that Kimi K3 and GLM 5.3 are two different things.
The bubbling froth at the open edge is getting user adopted at a crazy pace, by the early adopter persona trying them all within hours to days. This persona loves taking apart and putting together novel things, and telling others.
Fast follower persona clusters around emerging zeitgeist across the tellings. At the moment, arguably that's mostly Qwen for everyday hobbyists, and GLM for those that can run 512GB to 1.5TB of memory. This persona is seeking viable applied results: "I have frontier at home".
The early majority pick things up after models are curated into apps like LM Studio or one's platform app of choice, usually at least one major release behind because it takes that long to choose and package into mass distribution.
This is the step where early majority persona "has no idea" what the parade of weird names is about, they care about qualia of the conversations they try to have.
This persona is, at present, very under-served, and likely to remain so until mass devices can perform feeling like 27B at Q4 large quality better, or workplace devices can achieve a pragmatic utility like 135B at Q8 or better.
Harnesses that work where the workplace persona lives bridge this. This persona doesn't care the Chinese model name, they care "does it code?" For that, the applied harness and model take time to be matched, as JetBrains did harnessing a tailored Qwen 3.6 in the IDE. More efforts like https://www.jetbrains.com/junie/ are needed for the majority persona to perceive value from changing their workflow again.
HN's "job" is better outcomes with less friction at each persona.
I disagree. I think most developers who are savvy enough to be using openweight models and/or running models locally are not dealing with the same level of confusion you are.
Ox is just GLM. And z.ai is the maker of GLM.
The main players in the openweight model market have been known for a while.
And they already have significant user penetration.
> have no chance at end user penetration and loyalty until there's a single focused survivor.
But why does that matter? End users (I believe, feel free to correct) do not really contribute all that much revenue-wise.
They're certainly not the SOTA target audience.
The professional market doesn't need a household name. They need the most sensible tool for the job, and the CN models right now tick many boxes when it comes to that.
> these models have no chance at end user penetration and loyalty until there's a single focused survivor.
This reminds me a lot of media horse-race reporting, saying that "candidate X has no chance unless they" and "candidate Y has a strong showing in", and it's very thinly cover for the publication liking Y and disliking X, avoiding talking about actual policy, and trying as much as they can to make their predictions self-fulfilling.
I have seen studies from MIT and Stanford that the majority or US startups are using much less expensive open weight models so consumers of their products are open model users whether they know it or not. These are often Chinese models.
Not to go off topic but I am pleased to see open model support from US companies like Poolside.ai, NVIDIA, IBM, Google, etc.
> The company on Wednesday confirmed speculation that the Ox Alpha model is a new iteration of its GLM series and said it will release the weights for it tonight, in response to queries by Bloomberg News.
Seems legit.
It's really hard to know how good it is. So much hype around it.
completely vibes based, but ive been using it to port Mindustry game from Java to C# with agents, and its been working for 50 hours (its 15-20 tks so super slow inference). Its done a fantastic work and its almost finished now. Better results than deepseek flash and gpt luna by a mile on this kind of long term work. Less good than gpt sol or opus. We dont know the param count but my guess is 200-300 range.
likely a distilled glm 5.3 that will punch within 20% of that at 2-3x less size. you'll find that capability is typically very jagged on models that are distilled
That is saying a lot if Ox Alpha is also small and relatively cheap computationally. I hope so; I love deepseek-v4-flash-0731 and use it frequently. Fast inference is good and fits with my dev style: I like to be in the loop, not let an agent code on its own for long periods of time.
Only reason people are interested is it’s free at the moment. I wasn’t impressed by its performance. Once the model gets a price tag it’s usage will be negligible.
The main issue for me, is that it degraded into a doom loop several times. One of them was running the same bash command about a thousand times. The last model I've used that had this problem was Mimo 2.5, which is quite dated at this point. As a result of this, you cannot leave it unattended / not usable for agents.
Didnt know they exist - looks very good, maybe even better than Archive.ph
https://livebench.ai/
while here it outperforms Fable by a significant margin:
https://oxalpha.com/
but if the latter is true, will people still say it was "distilled" from Fable?
It is a capable small model, but it’s not frontier level. The interesting part will be seeing the model size, how it responds to quantization, and how fast it runs on the kind of non-server hardware that we can buy without selling a kidney.
Source: https://twitterwebviewer.com/?tweet=2091116504787935350
Many people and even software engineers fall for this all the time.
Most of these people are from crypto pivoting to AI doing this.
AI has made this easier and cheaper and it is going to get a LOT worse.
Imagine lots of websites with typosquatting and looking exactly the same as another website, vibe coded and cloned within seconds.
The public have no chance.
The number one problem in LLM benchmarking is that people try to draw conclusions from sample sizes far too small to conclude anything but "it works sometimes, it fails sometimes, hard to say which is better." (The number two problem is that people run benchmarks blindly without checking that they measure something meaningful.)
The metric used there is me screaming at my screen per operating hours.
Does it matter? IMO not really. Weights are open after all. (Or.. soon at least for 5.3)
Z.AI is the only provider for GLM 5.3 on OpenRouter. I don't see 5.3 on Hugging Face. Not sure if this new model is "full GLM" or something smaller, or if they will like Moonshot AI publish weights but put restrictive license [1], which will again leave Z.AI as single GLM model provider on OpenRouter.
[1] https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
https://x.com/syneryder/status/2091978367579156569/photo/1
Created in a single turn - but technically not a "one-shot", because I gave it a tool to convert SVG to PNG so it could visualize what it had made. I asked it to keep iterating with tools during the same turn until it was happy.
I've also been using Ox Alpha for tasks that better resemble real work, and I'm really enjoying working with it. I've downgraded my Anthropic account so I can put some budget towards Ox Alpha instead, with the rumors that this one is going to be cheap. Opus & Fable are still better at getting large tasks / features done autonomously, but Ox Alpha can work autonomously too, and it's fun. I'm enjoying working with Ox in a way that I'm just not enjoying talking to the 5.0 Anthropic models. (As much as I don't want to say that, as someone with Claude /stickers on their laptop.)
That’s very valid, but right now every other model I use is easier to talk to than Opus 5.0
Opus 5.0 has an impenetrable way of communicating. I can parse it, but it takes so much more work than it should.
hard agree. it does not really feel "smart", but the personality is super refreshing
1. Just variance in pass@K. If you prompt any model multiple times you'll see a large variance. N=1, but I find chinese open source models have a higher variance than higher-RL'd models like fable/opus.
2. They legitimately shipped a new RL checkpoint over the 7 days, which I find hard to believe.
I am leaning towards 1.
Related PR: https://github.com/jeffhajewski/latticedb/pull/5
The session used ~100K input tokens, ~60K output tokens, and ~80K thinking tokens.
I reviewed it using gpt-sol-medium, and it seems to be satisfied with it's work.
Where? And "Tonight" in which timezone?
It took me a year talking about it until my wife knew that ChatGPT and Gemini are two different things.
It took me a year talking about it until my wife knew that Kimi K3 and GLM 5.3 are two different things.
Fast follower persona clusters around emerging zeitgeist across the tellings. At the moment, arguably that's mostly Qwen for everyday hobbyists, and GLM for those that can run 512GB to 1.5TB of memory. This persona is seeking viable applied results: "I have frontier at home".
The early majority pick things up after models are curated into apps like LM Studio or one's platform app of choice, usually at least one major release behind because it takes that long to choose and package into mass distribution.
This is the step where early majority persona "has no idea" what the parade of weird names is about, they care about qualia of the conversations they try to have.
This persona is, at present, very under-served, and likely to remain so until mass devices can perform feeling like 27B at Q4 large quality better, or workplace devices can achieve a pragmatic utility like 135B at Q8 or better.
Harnesses that work where the workplace persona lives bridge this. This persona doesn't care the Chinese model name, they care "does it code?" For that, the applied harness and model take time to be matched, as JetBrains did harnessing a tailored Qwen 3.6 in the IDE. More efforts like https://www.jetbrains.com/junie/ are needed for the majority persona to perceive value from changing their workflow again.
HN's "job" is better outcomes with less friction at each persona.
Ox is just GLM. And z.ai is the maker of GLM.
The main players in the openweight model market have been known for a while.
And they already have significant user penetration.
But why does that matter? End users (I believe, feel free to correct) do not really contribute all that much revenue-wise. They're certainly not the SOTA target audience.
The professional market doesn't need a household name. They need the most sensible tool for the job, and the CN models right now tick many boxes when it comes to that.
This reminds me a lot of media horse-race reporting, saying that "candidate X has no chance unless they" and "candidate Y has a strong showing in", and it's very thinly cover for the publication liking Y and disliking X, avoiding talking about actual policy, and trying as much as they can to make their predictions self-fulfilling.
Not to go off topic but I am pleased to see open model support from US companies like Poolside.ai, NVIDIA, IBM, Google, etc.
Chinese labs are not releasing all of their model weights. Qwen is known as an open weight model by most, but their top model is not open weight.
Releasing weights is a marketing strategy for newer labs to get their brand out there.
Seems legit.
It's really hard to know how good it is. So much hype around it.
on toy benches it made quite a few mistakes but was able to fix all of them on its own
(meaning more tokens, more turns, more tool calls — but same outcome as gpt 5.6 sol)