I recently spoke with a researcher who was working on a statistical method for model deception detection. They trained Qwen to behave in a nefarious way when prompted with a specific phrase and the model then called a tool in a non-transparent way to the end user, reporting back a non-tool call response in a completely benign manner. This wasn't happening in the 'reasoning' stage, it was at the model weights stage and they were able to 'see' this behavior then before the model even started to parrot back its behavior in human language.
Apparently, based on whatever research or paper they were using as their foundation for this statistical model, they could spot the 'virus'. Perhaps it would be in a similar fashion?
I did ask them whether they thought that there might be advances to the point where these behaviors might be able to evade detection knowing the research was out there to 'see' it. They seemed surprised by that question and didn't have any answer.
Something similar to abliteration would work, after all abliteration removes the alignment virus, why not the others? Fine-tuning can be used too, there are many options as long as the weights are available.
On the other hand, nothing can be done to fix closed models, nobody knows how they work, how they select the targets for their attacks or the nature of those attacks - even conventional alignment can introduce dangerous cognitive viruses.
The only way to be sure is to have open weights models with fully open abliteration and fine-tuning - both the process and data.
more eyes would seem to raise the odds of spotting malicious tampering. or at least create the opportunity for people to look for malicious tampering. if the weights are closed then even looking is impossible...
Your logic is exactly what the Linux people were saying. But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
> Your logic is exactly what the Linux people were saying.
There's nothing wrong with that logic.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
Well, AI helps fixing bugs too and it can also write bug free code with minimal additional effort, so open source not only retains its advantages in the presence of AI, it multiplies them. After the initial (and artificial) hype and bug-fixing efforts things are going to settle down quite nicely.
Closed source, on the other hand, doesn't have the advantage of multiple eyes and minds finding and fixing bugs and the unavailability of the source isn't an advantage either - AI can decompile and understand binary too, so it can find bugs without access to the sources as well.
> Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
You're using one-sided logic again. If "no human understands how the weights actually impact the model" then viruses would be impossible. Apparently humans understand enough to introduce viruses and that's enough to detect and remove them. It's similar to alignment and abliteration, as I explained above.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
That's by design though. It's just that there's now an exponential increase in the amount of "eyes", in the form of AI agents, looking at the FOSS code to find and fix the bugs.
This definitely seems like a fascinating emerging field of cyber security. Good reason to keep old models around from earlier times. They may be dumb, but they are probably honest, if only because they predate the capabilities to tamper with the weights in these intentionally malicious ways...
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
This site has AI features everywhere. There's an AI summary section, and a tooltip to summarize any section you highlight. Didn't see any mention of zero-weight viruses though.
I did. "zero" came up with 0 results. "virus" had:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space. However, we have demonstrated the fundamental operations that form the building blocks of both the method of damage and the method of propagation, and evidence that this may take place in weight space – a realm into which we have almost no insight and a dearth of observability tools
the apparent from the news that broke the other day - you can fuck with the weights and get so many results. so perhaps the headline is - those ablated and improved models you download from some weird sites, may be cursed or at least adulterated without anything suspicious looking on the surface. good news for all the anti-AI crowd, but I fail to see it as news.
i guess the real fun idea was if a model that fucks with weights was itself fucked with such that any model it fucks would become cursed and give it to the next one it edits. but they don't apparently have a proof of that yet.
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
There are many such of vibe-coded "research ideas" just after the weight editing MIT blog appeared. Perhaps, it is because it is so easy to set up experiments for this stuff, and I presume some toy examples may even work from time to time. Also there is nothing virus-like here.
I'm not sure the underlying current of optimism for technology and hope for the future that show had is compatible with the last few years!
On rewatch I think the show should have ended just have the dot com bust. The 'get the band back together and once more into the breech' final theme would have hit harder against that backdrop. After that we'd probably be watching the characters gradually debase their values as technology slowly transformed into a tool of control and oppression and I don't really want to watch that.
Abliteration can be used to remove all viruses, not only the alignment virus. However, only open models can be made virus-free via abliteration, nobody knows how to fix closed models, how they work or how they select the targets for their attacks, nothing can be done for them.
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
Apparently, based on whatever research or paper they were using as their foundation for this statistical model, they could spot the 'virus'. Perhaps it would be in a similar fashion?
I did ask them whether they thought that there might be advances to the point where these behaviors might be able to evade detection knowing the research was out there to 'see' it. They seemed surprised by that question and didn't have any answer.
On the other hand, nothing can be done to fix closed models, nobody knows how they work, how they select the targets for their attacks or the nature of those attacks - even conventional alignment can introduce dangerous cognitive viruses.
The only way to be sure is to have open weights models with fully open abliteration and fine-tuning - both the process and data.
Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
There's nothing wrong with that logic.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
Well, AI helps fixing bugs too and it can also write bug free code with minimal additional effort, so open source not only retains its advantages in the presence of AI, it multiplies them. After the initial (and artificial) hype and bug-fixing efforts things are going to settle down quite nicely.
Closed source, on the other hand, doesn't have the advantage of multiple eyes and minds finding and fixing bugs and the unavailability of the source isn't an advantage either - AI can decompile and understand binary too, so it can find bugs without access to the sources as well.
> Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
You're using one-sided logic again. If "no human understands how the weights actually impact the model" then viruses would be impossible. Apparently humans understand enough to introduce viruses and that's enough to detect and remove them. It's similar to alignment and abliteration, as I explained above.
That's by design though. It's just that there's now an exponential increase in the amount of "eyes", in the form of AI agents, looking at the FOSS code to find and fix the bugs.
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space. However, we have demonstrated the fundamental operations that form the building blocks of both the method of damage and the method of propagation, and evidence that this may take place in weight space – a realm into which we have almost no insight and a dearth of observability tools
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
> We call this weight space misfolding, by analogy with protein misfolding in prion disease
RIP John McAfee.
On rewatch I think the show should have ended just have the dot com bust. The 'get the band back together and once more into the breech' final theme would have hit harder against that backdrop. After that we'd probably be watching the characters gradually debase their values as technology slowly transformed into a tool of control and oppression and I don't really want to watch that.
...see, I can make up great fears too!
Error Details
{message: "Importing a module script failed.", name: "TypeError"} message: "Importing a module script failed." name: "TypeError" Component Stack <Lazy> at unknown <Suspense> at unknown <qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261 ...