> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice.
I'm not sure "worked properly" and "as intended" accurately describe this situation.
I also can't believe the people who were involved with writing this response from Meta, didn't realize how obviously bad it sounds. It's like there is no humans working and writing there anymore.
> It's like there is no humans working and writing there anymore.
Don't know if AI is to blame, but I've used to see these kinds of nonsense post-mortems even in the pre-llm era, and it's always due to some internal fighting ongoing between various departments.
To be fair, that quote in the original article could have more context. By "The tool" they meant "AI-assisted support tool"[1]; perhaps they meant that the issue was not an AI hallucination inherent of the tool, but a fixable bug.
In that case, the statement is so meaningless as to be useless. Why should we care how Meta splits up their microservices? The tool still failed. They just want to redefine the "tool" as something else, anything else, to avoid having to admit something negative about their precious AI.
> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address
> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address
It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.
I get the joke, but it's a relevant nuance that the new code, the chatbot, did not have 'the bug'. I still think that the mistake and head that should roll should be the one that published the chatbot.
But it's important to acknowledge that there was a 'bug' in an underlying tool and not in the chatbot, and still PIP/fire those responsible for publishing the chatbot and exposed an otherwise internal tool to the public, and not those that introduced the 'bug' to an internal tool.
That sounds a lot like the justifications Claude and ChatGPT give when confronted about something they did wrong, or when asked to provide a customer support response about software issues
I've lost track of the number of times Claude has basically said "it was like that when i got here" in the face of a clearly bogus choice and easily disproved explanation.
They should add a feature called "auto-really" that just automatically says "really?" after the chatbot answers a question to check if it's going to 180 upon this tiniest bit of scrutinity.
The argument here is that the AI is a glorified input page. The input field asks for your username and email and sends it to a backend function. Such an input page is working as intended.
The problem is when the backend function doesn't verify that the email matches the username.
Why on earth would the backend function even take an email?
Or perhaps said different: use the submitted info to identify the account; send any sensitive messages (recovery codes, password resets whatever) to only the contact info on file. If the chat bot can send such email it should do so via an API that sends only to contact info on file for the associated account and not to an email that's provided by the bot.
> Why on earth would the backend function even take an email?
In principle, it could be designed to do so to handle cases where a new email address has been confirmed out of band, e.g. for an account representing a company or a political office. But that's a relatively unusual situation, not something you'd want to be available to every user writing in. (Even if you had an all-human support department, this sort of functionality would only be available to a select few agents.)
I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.
> But when humans handled it, this was not as much as a problem.
In fact it's arguably a feature. The ability of support staff to short-circuit nitpicky rules when there's an obvious external validation happening (e.g. you're on the phone with a user who's presenting ID in real time and correlating it with previous use of the account, etc...) makes for better data quality and happier customers.
Obviously, yes, you can then human-engineer an authentication breach. But that was very difficult, because people are "common-sense careful" in a way we haven't been able to tease out of AI yet.
so how long was the bug there? was there a way to access it before/without the support agent? it feels like Meta will throw anything under the bus to redirect blame from the AI, because that would be the end of their $600B (depending on “which number you want to go with”) experiment
What I gather is that this internal tool was used by human support agents, and it was their responsibility to verify the email adresses and general validity of a claim.
But when implementing AGI TM that was overseen, maybe the oversight in the separate code path was a 'bug', but the mistake was making the chatbot obviously, if the separate code path had a bug, then it had become ossified into a feature, and it was internal, not exposed to the public.
This is an external communication, to save face sure, but if this is the internal excuse, it would be absolutely the wrong RCA and it reads as if the one who made the mistake is not admitting they made their mistake. Which to be honest, just making the mistake is enough to get fired, but not admitting it is enough to get ultra fired.
There's an excellent little book in German called "KI und der Neue Faschismus" [1] (AI and the New Fascism) where the author (Rainer Mühlhoff) tries to warn about the dangers of decisions based on opaque statistical models (like LLMs) instead of a clear, human auditable decision process.
Did you create the account separately? Or as an asset of your main Meta account (like Meta Business Suite)?
I'm creating the accounts in Meta Business Suite, so I would have a recourse with my main personal account which can be linked to some adspend, so I'm assuming it will have better support channels than accounts created through an end-user interface.
This was probably my mistake. This is the first time I've ever done this so I just set up an account like a regular user. I didn't figure out how to link it with my Meta Business Suite until it was too late.
...They really ahouldn't have, and I wonder how this will affect all the big AI IPOs. After all, Meta is one of the big players in the space. Surely if they can't do it right, then...
Why was 'can a user request a different email' not literally the first test that comes to mind when making something like this? Do they not test anything because the scale is too big?
The nature of the invention is for people to relieve themselves of the burden of having to use their minds. And while there will be exceptions, (including, I'm sure you: the person reading this comment,) the vast majority of people are hungry to use AI in that spirit of being able to be lazy.
Because software professionals are conflating simplicity of user experience with simplicity of dev experience.
During development they were likely not thinking of the user experience, nor even the support agent experience, but on their development experience, they asked the LLM to develop the chatbot, and it worked, and the speed was documented and reported upstream so that shareholders invest, if there is any forethought it would go against the narrative of AI becoming the engineer or 100xing productivity.
I got a suspicious password reset request email today from Meta but it landed in my inbox. Luckily I have MFA and after checking audit logs inside IG upon logging in, I did not see anything suspicious.
If this was a bank that had zero humans and the AI chatbot was abused to hand over sensitive information about their customers which led to this disaster, people would never trust their bank ever again and leave.
Meta believes that they can vibe-code their reputation down the drain by removing humans in the loop.
Applying a technical solution to a social problem almost always ends in disasters like this.
I’m guessing they have no functional human support for the people who had their accounts stolen. I get the impression Meta didn’t know this was happening until they were contacted by the media.
I'm not sure "worked properly" and "as intended" accurately describe this situation.
I also can't believe the people who were involved with writing this response from Meta, didn't realize how obviously bad it sounds. It's like there is no humans working and writing there anymore.
Don't know if AI is to blame, but I've used to see these kinds of nonsense post-mortems even in the pre-llm era, and it's always due to some internal fighting ongoing between various departments.
(Usually said loudly and when everyone is at their most upset, e.g. a vacation ruined)
[1]: https://www.documentcloud.org/documents/28202858-meta-ai-ag-...
> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address
> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address
But it's important to acknowledge that there was a 'bug' in an underlying tool and not in the chatbot, and still PIP/fire those responsible for publishing the chatbot and exposed an otherwise internal tool to the public, and not those that introduced the 'bug' to an internal tool.
The problem is when the backend function doesn't verify that the email matches the username.
Or perhaps said different: use the submitted info to identify the account; send any sensitive messages (recovery codes, password resets whatever) to only the contact info on file. If the chat bot can send such email it should do so via an API that sends only to contact info on file for the associated account and not to an email that's provided by the bot.
In principle, it could be designed to do so to handle cases where a new email address has been confirmed out of band, e.g. for an account representing a company or a political office. But that's a relatively unusual situation, not something you'd want to be available to every user writing in. (Even if you had an all-human support department, this sort of functionality would only be available to a select few agents.)
But when humans handled it, this was not as much as a problem. That is, the humans did the job, because they recognized the need to do that job.
Sure sometimes accounts could get recovered if a human was tricked, but evidently it was easier to trick the LLM in masse than humans.
In fact it's arguably a feature. The ability of support staff to short-circuit nitpicky rules when there's an obvious external validation happening (e.g. you're on the phone with a user who's presenting ID in real time and correlating it with previous use of the account, etc...) makes for better data quality and happier customers.
Obviously, yes, you can then human-engineer an authentication breach. But that was very difficult, because people are "common-sense careful" in a way we haven't been able to tease out of AI yet.
What I gather is that this internal tool was used by human support agents, and it was their responsibility to verify the email adresses and general validity of a claim.
But when implementing AGI TM that was overseen, maybe the oversight in the separate code path was a 'bug', but the mistake was making the chatbot obviously, if the separate code path had a bug, then it had become ossified into a feature, and it was internal, not exposed to the public.
This is an external communication, to save face sure, but if this is the internal excuse, it would be absolutely the wrong RCA and it reads as if the one who made the mistake is not admitting they made their mistake. Which to be honest, just making the mistake is enough to get fired, but not admitting it is enough to get ultra fired.
(If anyone at Meta/Instagram sees this I wrote a brief blog post with the details. Please help! https://addisonwebb.com/blog/2026-06-05-Can%20Someone%20at%2... )
[1]: https://rainermuehlhoff.de/KI-und-der-neue-Faschismus-Reclam...
I'm creating the accounts in Meta Business Suite, so I would have a recourse with my main personal account which can be linked to some adspend, so I'm assuming it will have better support channels than accounts created through an end-user interface.
oh no...Meta what are you doing
...They really ahouldn't have, and I wonder how this will affect all the big AI IPOs. After all, Meta is one of the big players in the space. Surely if they can't do it right, then...
During development they were likely not thinking of the user experience, nor even the support agent experience, but on their development experience, they asked the LLM to develop the chatbot, and it worked, and the speed was documented and reported upstream so that shareholders invest, if there is any forethought it would go against the narrative of AI becoming the engineer or 100xing productivity.
https://news.ycombinator.com/item?id=48359102
It's like, people abusing an open door. "Guys, just because we left the door open to your bedroom doesn't mean we're responsible".
God can only hope this is a business ending lawsuit.
also this is more like them leaving the keys in the door, then someone comes along, uses the keys, and steals all your stuff.
truthfully, no equipment is actually defective in this scenario eh?
Meta believes that they can vibe-code their reputation down the drain by removing humans in the loop.
Applying a technical solution to a social problem almost always ends in disasters like this.
Reputation can’t be vibe-coded.
https://www.maine.gov/agviewer/content/ag/985235c7-cb95-4be2...
> Date Breach Discovered: 05-31-2026