How can these models do anything close to RSI when they can’t even self check their output? Gemini for example is so self confidently wrong about 30% of the time for me on certain tasks. I tell it that its answer is wrong and it issues a mea culpa but goes back to being wrong in short order. I feel like the AI industry is still massively overstating their projections.
Gauging state of the art against what is available for free or for very cheap per token cost is like gauging the maximum theoretical transport potential by riding a bicycle.
You’re using something that is very energy efficient; you cannot extrapolate that experience to conclude that SOTA models are not doing something much different.
I think your reply has a somewhat familiar structure -- "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". You might be completely correct! But these sorts of claims push the onus back onto the other person, without accepting any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
In my recent experience, error rates for astra/fable are at or below human level. Just like when directing humans, it pays to ask probing questions ('Are you sure about X?', 'Did you check for Y?', 'Please run Z just to double check.') if you really care about the result being correct.
And if you're building something of any importance, you need to have verification steps at checkpoints. It's honestly just engineering.
Weak models tasked with review can catch a decent amount of the mistakes that weak models make and help them be much better, especially if you have them verify against authoritative sources. Strong models make far fewer mistakes to begin with. And, for the mistakes they do make, a swarm of reviewers (same model or somewhat weaker, reviewed by the stronger model) can really help reduce the error rate further.
I made this same reply to another thread, so I'm sorry to say essentially the same thing twice, but -- the comment has a familiar structure, "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". These sorts of claims push the onus back onto the other person (or in this case tfa), without really accepting the result, or taking on any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
Yeah, it’s crazy how fast things have changed in a month. I couldn’t find a more recent replication or similar study but it would be interesting to see it done with the current frontiers. Though I don’t think that would change much about the overall conclusion of the paper
We need to be careful of wishful thinking. People are going to want to assume the existence of some sort of "deus ex machina" which is going to make everything fine. I prefer to turn the logic around. If there's any decently high chance that things could go off the rails, we should be shutting AI development down: https://pauseai.info/
When every man is torn apart
With nightmares and with dreams
Will no one lay the laurel wreath
When silence drowns the screams
Confusion will be my epitaph
As I crawl a cracked and broken path
If we make it, we can all sit back and laugh
But I fear tomorrow I'll be crying
Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI.
You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").
> You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.
I lived through roughly the latter half of the cold war and both of my parents were soldiers. I lived in West Germany etc.
No we have not escaped the nuclear thing. If anything, the current Russian tzar is rather more unhinged than any of his predecessors.
I doubt many here know what perestroika and glasnost mean or why those Russian words were so important back in the day.
I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!
I do fear numpties with a red button and a tenuous grip on reality.
We have had nuclear weapons for less than a century. It takes one guy, one time for it to all go to shit. We are not out of the woods yet and we will never be.
We have not made it. We have survived so far but the threat remains. Stockpiles and warheads have shrunk, so it wouldn’t be nearly as bad as in, say, 1983, but we’re still one mistake or misunderstanding away from the worst day in human existence.
And worse, somehow we’ve managed to declare victory without achieving it, and there’s no longer any real attention on actually fixing the problem.
If AI follows the same example, we’ll take some measures to lessen the impact of Armageddon and then carry on saying “problem solved!”
Except we did had a bunch of close calls [1]. Also, we've been warned about climate change for at least 30 years and were unable to avoid it. So OP's remark is entirely deserved.
Until proven otherwise, we are not part of a movie where the hero saves the day at the end. I thought 9/11 made it pretty clear to everyone?
I don't think the end of the world would even be a bad thing; a lot of people assume that it would be disastrous but I think it's much more likely that it the "end" would just be a fragmentation that would be pretty uncomfortable, but not to the point of being horrific. I actually think that people are more resilient and creative than they think and if they were faced by a global economic collapse so strong that big tech would go bankrupt, they would bounce back pretty quickly. It's just that we've seen so many movies about the end and have been conditioned to believe that we're helpless that we think otherwise.
Well, duh. If you could do this with Opus 4.8, we would know. When Astra’s successor is 2-3x better at math research, and the internal teams say “we believe we will get there,” I’m inclined to believe the insiders.
The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago? The insiders who are about to file for IPO?
Going into the color of the bikeshed; hacking huggimgface to cover up cheating on your hacking test; swapping your language for no discernable roi. You know the guy, severe OCD and anxeity, who barely does anythong of value due to his anxious brain?
Yeah, it should be obvious what AI is really doing and its definotely not ROI improvements.
What if any of the older good models could also have written those math proofs if they were given the same order of magnitude of resources? We don’t know and there is literally no one else in the world to check it. To me it’s very suspicious that all these hacking, containment escape, hidden internal thinking, math proofs started coming out all at once in a very short time right as IPO talks have intensified and Chinese seem to get closer and closer, also regulation discussions are starting to get very serious. I have used these models and they are good, especially Fable, but not groundbreaking. With intelligent guiding I actually feel better using Opus 4.6 as I feel more in control, having less hidden away from me.
The actual title of the paper is: "Can AI agents conduct open-ended AI research? Early evidence from two case studies"
While I appreciate that the article is throwing a web blanket on doomer claims, the actual study doesn't really get into AI self-improvement. That doesn't require writing papers. That just requires autonomously writing a software system that can produce a better AI agent then the one that created it. That said, I have little worry about this being possible as I have seen no evidence of AI agents being able to produce a working software system of that scale.
I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks.
Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.
The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.
Training has become very complex.
The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.
When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.
It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.
What is commonly called an AI agent is the combination of a harness, an inference middleware, and a model. Those models are trained by a software system that includes a harness and middleware, so modifying the harness and middleware can influence model training.
Right now agents are good enough for throwing semi-random ideas at the wall. Experiment compute is the bottleneck because it’s not much more than brute force search. A sufficiently intelligent agent with a deep model of its own architecture will more quickly and confidently locate improvements, the same way that high end LLMs can point out a bug and write a correct fix without even needing to observe and probe the program at runtime. If this level of research performance is reachable, experimentation may become much less of a bottleneck. Hopefully it isn’t.
this article reads like a joke the "new study" is from group of people that are not at the frontier. they test with $3k of anthropic credits (compare to the >$10M in compute used to solve recent NS last week)
>compare to the >$10M in compute used to solve recent NS last week
Heres a thought, if theres going to be a dangerous super LLM, if it costs 10 million bucks a month to run, then theres very little danger of anyone letting it go without a purpose. Like at some point the economics make it super unlikely that AGI is a threat outside of being a tool for a nation state.
There’s also the difference between a model recursively improving “itself” and improving itself via online learning.
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet).
The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.
Of course it might not, it has been the holy grail of AI research for a long time. It would be great if we could leave some self improving code running on a blank slate of a computer while we sleep and the machine was crying asking me what is everything the next morning. None of AI researchers have had that moment outside of their dreams, so far, but it would be great if it happened.
Something is still not making sense to me. We have these mankind extinction models, yet when you given them a problem relatively “simple” to complete it end to end you get AI slop.
Can we pause the AI development after the AI slop is “fixed” perhaps with something less than 10.000 agents?
We used OpenClaw to run these experiments so that our scaffold was agnostic to the model provider. We conducted dry-run experiments with models from OpenAI and Anthropic before settling on Opus 4.8 as the best-performing model. In response to concerns that our results might be principally explained by a limitation in our scaffold, we repeated our experiment on one paper using GPT-5.6 Sol and Codex, its native scaffold, with the same time and API budgets. The results of this experiment were similar to our OpenClaw/Opus 4.8 experiments. This makes us more confident that our results are not simply artifacts of a scaffold deficiency; this run reproduced nearly every single one of our identified failure modes
The agent required three interventions during the run. First, we needed to modify the scaffold to resolve a bug in the OpenClaw harness that affected Anthropic reasoning models. Second, we gave the agents a 24-hour deadline extension; at the time of the original deadline, the agents had submitted drafts with a completion report indicating that their self-review was a "Weak Reject" and outlining the next steps they would take if given additional time.
I'm fairly sure Fable 5.1 could have designed a better experiment than the authors here, but hey.
The method of the NS advance involved RLHE (reinforcement learning via human example), and that is only open-ended if users continue to advance the frontier within chats ahead of publications.
Sure, but the point is that the labs use more powerful internal models for research work, not public models. Public models tend to lag the internal frontier by a decent margin, and are constrained in other ways by monitoring. It’s just not a useful indicator.
I thought by now AIs would not only be rewriting their code, but rewriting CPU microcode to optimize how their code is written and executed. Nowhere close it turns out.
You’re using something that is very energy efficient; you cannot extrapolate that experience to conclude that SOTA models are not doing something much different.
That's definitely part of your problem.
In my recent experience, error rates for astra/fable are at or below human level. Just like when directing humans, it pays to ask probing questions ('Are you sure about X?', 'Did you check for Y?', 'Please run Z just to double check.') if you really care about the result being correct.
Weak models tasked with review can catch a decent amount of the mistakes that weak models make and help them be much better, especially if you have them verify against authoritative sources. Strong models make far fewer mistakes to begin with. And, for the mistakes they do make, a swarm of reviewers (same model or somewhat weaker, reviewed by the stronger model) can really help reduce the error rate further.
Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc
Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI.You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").
> You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.
No we have not escaped the nuclear thing. If anything, the current Russian tzar is rather more unhinged than any of his predecessors.
I doubt many here know what perestroika and glasnost mean or why those Russian words were so important back in the day.
I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!
I do fear numpties with a red button and a tenuous grip on reality.
And worse, somehow we’ve managed to declare victory without achieving it, and there’s no longer any real attention on actually fixing the problem.
If AI follows the same example, we’ll take some measures to lessen the impact of Armageddon and then carry on saying “problem solved!”
Until proven otherwise, we are not part of a movie where the hero saves the day at the end. I thought 9/11 made it pretty clear to everyone?
[1] https://nsarchive.gwu.edu/briefing-book/nuclear-vault/2020-0...
I'd trust anyone but them personally
Yeah, it should be obvious what AI is really doing and its definotely not ROI improvements.
So the paper is out of date and pointless then
While I appreciate that the article is throwing a web blanket on doomer claims, the actual study doesn't really get into AI self-improvement. That doesn't require writing papers. That just requires autonomously writing a software system that can produce a better AI agent then the one that created it. That said, I have little worry about this being possible as I have seen no evidence of AI agents being able to produce a working software system of that scale.
Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.
The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.
Training has become very complex.
The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.
When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.
It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.
Heres a thought, if theres going to be a dangerous super LLM, if it costs 10 million bucks a month to run, then theres very little danger of anyone letting it go without a purpose. Like at some point the economics make it super unlikely that AGI is a threat outside of being a tool for a nation state.
We literally just saw how OpenAI’s model got out and hacked HuggingFace
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet).
The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.
Test-Time Learning: The model updates its own memory weights while running an inference task.
Can we pause the AI development after the AI slop is “fixed” perhaps with something less than 10.000 agents?
The agent required three interventions during the run. First, we needed to modify the scaffold to resolve a bug in the OpenClaw harness that affected Anthropic reasoning models. Second, we gave the agents a 24-hour deadline extension; at the time of the original deadline, the agents had submitted drafts with a completion report indicating that their self-review was a "Weak Reject" and outlining the next steps they would take if given additional time.
I'm fairly sure Fable 5.1 could have designed a better experiment than the authors here, but hey.
Meanwhile, Navier–Stokes was solved by an internal model significantly more capable than Astra (and therefore more capable than Mythos/Fable).
I’m afraid this sort of experiment is cope. The labs clearly believe RSI is coming soon.