The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"
There's also a need to consider the rights that this new intelligence should have.
> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
Without trying to sound crass, food banks _are_ the reason we don't see people dying of starvation in first world countries. If people need food, a food bank will give it to them no questions asked.
Unfortunately this is not just precarious, it's extremely vulnerable to changing political conditions. Many of the food bank services in my state have lost significant funding owing to the use of words like 'women' or 'black' in their grants, which were duly grepped for and shut down.
Is it not sufficient? Americans give more to charity than citizens of any other developed country. The arguments that all food aid should be routed through government bureaucracies are entirely unconvincing. I'd rather donate to organizations like Second Harvest than pay higher taxes.
This is what happens when you don't think through a problem completely.
>I'd rather donate to organizations like Second Harvest than pay higher taxes.
I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.
Hence the argument of 'just donate' are just as unconvincing to me.
Does this account for the portion of taxes that go to the type of efforts that donations do? If not, is it really giving more? OR is it giving less and feeling better about it?
It is my understanding that this claim about Americans' charitable giving does not disaggregate giving to food banks, Girl Scout cookie drives, political causes, the environment, cultural events, religious institutions, etc. Much of this charitable giving does not feed the hungry or house the homeless.
Plus the fact that there’s bias in the first place - citizens in a country where they know social programs will care for their needy would not need to donate.
We used to live in a world where individual virtue was what everyone fell back on. That is the default state. It has many problems -- free riding by the selfish, for one, but also a lack of capacity to prepare for large-scale disasters like the Great Depression. We have a government precisely to solve this sort of problem. It works (when it is competent and not corrupt) for defense, international trade, and the enforcement of law. Solving resource distribution and aid coordination is right in its wheelhouse.
The fact food banks exist suggest over all food scarcity is solved in that specific culture. - if it wasn't there wouldn't be any spare food for a food bank
Food is allocated using a variety of mechanisms in peaceful first world countries, primarily money but also via government assistance, kinship, friendship, community, etc.
At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.
The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.
Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".
It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
It's circular to say "it's an allocation problem". Yes, that's the entire point: we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.
So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.
So I agree with you that AI is likely to create or exacerbate allocation problems because currently lots of stuff is allocated by money.
> as a species, we can't guarantee the allocation of a livable baseline to every person.
Like we have been discussing, this is pretty much solved in the West.
For food I feel like "as a species" seems impractical? Like humans are not organised "as a species", we are organised into groups, which are supposed to have autonomy and self governance at some level of administration. Specific allocation mechanisms are arrived at and operated by groups of people.
But whose obligation is it exactly to ensure EVERY HUMAN has food? Is that my job in particular? Yours? Not just for like, "your circle" or your neighborhood or country but EVERY HUMAN? How do we even know who EVERY MEMBER OF OUR SPECIES is, does that mean there needs to be a global registry of EVERY HUMAN? Who enforces and has access to that?
The worst failures are mostly happening in other countries. What specifically should we do about that? Food aid exists but it often doesn't reach the people who it's meant to help for various reasons. Who or what group exactly is responsible for making sure everyone gets their livable baseline, and what powers should they have? Should we invade countries that fail to allocate food efficiently, forcibly census their citizens and establish a militarised distribution network to ensure EVERY member of our species is fed?
> It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
Is that a fact? Do you have references to back it?
The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication. This is pretty normal in some poor communities and that cash is commonly diverted to various vices. Some malnutrition is a consequence of this. Adding friction to the conversion of welfare benefits to cash is a feature.
You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.
I mean it's pretty straight forward that this won't fix all hungry people as some portion of said hungry people are addicted to drugs in the extent they'll starve. That in itself may be solvable by things like free drug clinics (drugs are cheap, it's prohibition that makes them expensive).
And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.
A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
It's a limited area of desirable land zoned for housing. You could build all the tiny homes in the world in bum frick nowhere, but the unhoused would rather be homeless in a big city for various reasons. I probably would too.
I recommend doing some reading about the inputs modern agriculture depends on (e.g. chemical fertilizer) then thinking about the issues they're associated with beyond an anthropocentic frame (e.g. ocean dead zones).
Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.
Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.
Uh, I think threatening the work of a large swath of humanity would make a lot of people poor? The rich historically have never shared the wealth until they're forced to.
Just as an example (ironically from google!): "Google co-founder Sergey Brin has spent $82 million to oppose California's proposed wealth tax."
This is a man that has more money that he could ever hope to possibly spend, and he's spending an absurd amount just because the mere thought that he might benefit from the society he lives in and should contribute back to it apparently infuriates him.
I support progressive income tax and very high capital gains tax. I do not support wealth tax. There are very sound economic and technical arguments against wealth taxes. You can oppose the wealth tax proposal while supporting higher taxes for the rich.
Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.
> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
> There's also a need to consider the rights that this new intelligence should have
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
> They don't have free will, and I don't mean in any especially philosophical / theological sense
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.
My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open.
If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.
Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...
You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.
We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.
> There's also a need to consider the rights that this new intelligence should have.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
> Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.
As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]
> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?
> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their
tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between
dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax
liability by 14.5 percentage points."
> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.
Spoiler: The plan is .. add massive regulation, but only to the US, don't affect other countries developing it in any way other than "setting a good standard that'll hopefully influence them". Seems like an airtight plan.
If the USA takes up his suggestions, it will have an incentive to work on international frameworks and treaties with competing nations to make the regulation global.
Slowing down helps China catch up though. I'm also not sure that the CCP is really the most enthusiastic supporter of unlimited access to arbitrarily powerful LLMs
China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.
There won't be any kind of international framework, because by now countries have learned that the US' word is pretty much worthless, similar to Russia's. They now know from experience, US-ratified treaties will only be honored if they feel like it on a particular day. If it's inconvenient, it will simply be ignored.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
Those sticks look less effective today. The hegemony is in recession as the rest of the world smiles, pays omage to dear leader as needed, and works to decouple in the background.
Demis has only 1 plan, how to dodge releasing a new model at this point, jokes aside I value thinking about AI safety, but are we really so close to AGI? It doesn't feel like it, LLMs still diagnose my headache as a chronic illness or a brain tumor from time to time... honestly stressful.
regarding your headache diagnosis. If you just ask: could my headache be a brain tumor, the answer is probably: very unlikely but it might be.
thats because its actually true. there is actually a non 0 chance that headache is a symptom of a brain tumor. From a bayesian standpoint though its really super super unlikely, especially if there are no other symptoms that might shift the prior.
a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.
maybe you are doing this already, idk. just wanted to share what works for me.
>This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
On the STEM side, yes no doubt he is a lot smarter than me.
But as per the documentary of his life, he is wholly focused on AGI and will remain unfulfilled if he, or indeed anyone else, doesn't achieve it within his lifetime.
But even geniuses make mistakes. For example, Demis bet hard against language having any sort of real impact in AI research. So hard in fact that DeepMind nearly had its language research team walk out on them as they were about to be labelled as "applied". Now, to be fair, once the evidence was mounting that large language models rather than pure reinforcement learning was the way forward Demis and DeepMind changed course and has been great at catching up. However, it is important to remember that no person is infallible.
No, but it would have been a move from the research division to the applications division at the time. This is around ten years ago, between Atari (pre-Google acquisition) and AlphaGo (post-Google acquisition), so I am unsure exactly what the effect would have been inside the organisation. But, if it was bad enough for people to want to leave I suspect it can not just have been about the label.
I should emphasise that DeepMind (London) and Google DeepMind (remember now that Google's research labs across the globe got merged into this entity, including the one in London that was previously split from DeepMind "proper") has always produced good research. But what led to the LLM explosion primarily came from Google Research Mountain View (Transformer), Google Research Seattle (BERT), OpenAI (GPT-3), and various academic labs. During this time DeepMind had a broad research agenda, but my impression is that most resources went towards more traditional deep learning agents in 3D environments, foundational algorithmic research, and games (StarCraft 2). I would provide more detail, but as as an NLP researcher, I did frankly not pay much attention to DeepMind in that era as what was interesting to me just came from elsewhere.
When Google made its first big move into LLMs, its first interesting work did not come out of DeepMind, but rather again Mountain View with PaLM in 2022. Someone with closer connections to Demis at the time can probably explain why the research agenda was what it was, but I lack those connections and DeepMind (even now) has been notoriously tight-lipped since the founding of OpenAI (Demis allegedly sees OpenAI as founded on DeepMind technology by interns and staff walking out on them) and was rather insular compared to Google Research, FAIR, etc. I say "was", but not because DeepMind is now more open, but rather that all industry research outlets are now incredibly tight-lipped after LLMs becoming a product and point of pride among them. At this point, research is held back for maybe even a year before publication or even withheld publication to deny any perceived advantage to the competition.
Blah-blah-singularity, so let's cripple the models so much they refuse to talk about React, because who knows if you are not cooking chemical weapons or meth in your browser's DOM, right?
By the way, if you try to use it to write prose, it will drive you mad by adding "adult" everywhere, e.g. "adult chessboard". Applies to GPT, Claude/Fable and Grok.
Not surprised, seems these labs start calling for regulation once they are losing or have competition. OpenAI started calling it for it once Anthropic got better, Anthropic started calling for it once the Chinese models got good, Google is now calling it for it because they are falling far behind.
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
"It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials" — but these are not the real problems plaguing modern developed societies, are they? What developed societies really need to figure out right now is how to distribute the already available resources without making people miserable, and so far AI hasn't been helpful.
How will this be enforced, at least with financial markets money is discrete, can largely be counted. This seems like a slippery slope to full blown surveillance of the internet and in general computing.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
Agreed. There needs to be full transparency on the capabilities of the models. Being given access is one thing, but you can't have labs building powerful models that could be manipulating the entire planet without the public knowing. The public needs to have a good pulse on what the leading models are capable of so that we can remain in touch with reality.
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
This genre of AI researchers begging third parties to stop them from destroying the world is getting really tiresome. Setting up regulatory bodies with ill-defined goals to counter hypothetical threats from science fiction is a recipe for disaster.
The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.
For better or worse, humans (or any animal) are a lot better at reacting than planning. I'm sure this technology will play out differently than any one of us, or any collection of us, can imagine. The possibility space is enormous.
I do wonder what type of AI some of these leaders expect to be able to harness. If you create something that is true AI, won’t it be smarter than you to a level you cannot fathom. I was thinking of this idea/though-experiment (which I know is ridiculous) of what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
Well dogs specifically have supplanted children in many households so one could argue the dogs won. In a selfish gene sense, the dogs created a machine that carefully ensures a much larger reproduction scale than could be managed by the dogs themselves. This is an unbridled success.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.
The real risk of AI is: risk of global recession if they turned out unprofitable, risk of them propping up fascist movements Thiel and Musk love so much, risk to environment, risk to mental health of people who have to listen to constant doom trolling.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
All the frontier labs are lobbying hard to lock down the AI market, because they see that their position at the top is temporary and that there's no secret sauce.
I've reas the tweet and...Am I mistaken or is the plan "Do what Anthropic has been doing and advocating for this past year" ?
My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now.
The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.
Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.
In Sparta, according to Plutarch, in his The Life of Lycurgus:
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".
It's not clear to me whether this is being done with genuinely good intentions, or if it's just a way to put barriers in front of open-source models. We'll see.
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
Not exactly a "plan" - he's just saying we should have a standards body that assesses models for safety.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
Has anything really important been solved by AI yet, or where is this radical (imo) belief around AGI coming from?
Genuinely curious, I know there are some math problems solved and ML has been used for far longer than AI to improve things, but where is clean (efficient) energy, the cure for cancer (or any of the horrible neurological disorders, take your pick), new hardware designs, quantum computing solutions, etc etc, you get the gist.
Where are the things that will actually send humanity into the next era of civilization, I don't care about more React apps (but I do enjoy my coding companions for other things).
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
> Has anything really important been solved by AI yet
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
It's also very very far from anything general intelligence. This is actually where I think we see the best ROI, we have a specific task or problem and we target it. I remember reading about a model trained on railroad tracks. By putting cameras on trains to take images while running like normal, then using the model to detect cracks before they cause issues.
We don't need general intelligence to create/destroy many things with Ai. Its frustrating that so much of the conversation centers around the variety that concept fissioned language from humans.
The point is that we need to have a safety plan in place before an AI is smart enough to radically reshape the world. If you've got an AI that's ready to start sending humanity into the next era of civilization, it may be too late to control when and how it does that.
> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.
Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
Experts as in AI-2027, Jenson “I think we’ve achieved AGI in March 2026” Huang, or that AI Researcher(tm) calling to pre-emptively bomb Microsoft data centers in summer 2022?
A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"
>I hope you'll keep this in mind when those milestones are reached.
The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.
What is an example you have in mind of a self-fulfilling prophecy? I genuinely don't know what you could be referring to. It seems to me that they keep making surprising prophecies, and the popular reaction to them seamlessly transitions from "that's crazy, no way it will happen" to "that's silly, it's just a cloud pattern". Did you find it obvious or self-fulfilling in 2025 that LLMs would soon be able to resolve open questions in mathematical research?
Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.
None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.
Why would we be able to design a safety plan that would control such a powerful force? It would be like putting umbrellas on an asteroid hoping to slow its fall. That sounds like delusions of grandeur.
It'll certainly require a bit more than umbrellas! NASA has spent hundreds of millions of dollars developing asteroid defense systems, including a proof of concept for redirection in 2022. So, you know, let's try at least that much before declaring there's probably no way to do it.
These people who read too many scifi books and confused them with reality are royally annoying.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
If in 2020 I had sent you a book about the LLM achievements of 2026, you would probably have thought it was a science fiction book with no relation to reality, wouldn't you?
That it generates plausible texts? I would find it unlikely technological progress. That does not make this whole "AI safety" rhetoric any less bullshit distanced from reality. Especially from companies that are doom trolling all the time while also trying to maximize the negative impacts they are doom trolling about.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)
Well, if he had proposed more concrete plans for what that body might do... but yeah, it seems too abstract that indeed sounds like a bureaucrat wanting to wrap everything under a big pile of paperwork.
I am looking forward to a bunch of investigative documentaries exploring “How the cult of Omnissiah infected every single Doomsayer CEO of an AI company in Delusional 20s” (or whatever monicker our times will have got in the future).
Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.
How did he reach that conclusion, that it's only a few years away? Guy seems like a complete shill, when he talks to a non expert audience.
Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.
All that would happen from what he's proposing is such a watchdog would just be an explicit formal declaration of the US's national interests as being somehow the most legitimate, which in the context of current international relations is basically putting up a sign saying: "reject this!"
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
No snark. I personally feel a lot more productive, but my pain points with my work remain.
The 4-6 hours day which I actually spent typing out code, was actually my downtime which has been taken away from me.
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"
There's also a need to consider the rights that this new intelligence should have.
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
Obesity is much more related to the type of food than the quantity.
Many developing countries have obesity issues due to scarcity of fresh and healthy food.
In some places coca cola is cheaper and / or more available than drinking water.
How do you explain the existence of food banks in peaceful first world countries?
https://www.whitehouse.gov/wp-content/uploads/2025/05/Cuts-t...
Civilization can't rely effectively on systems that are this fragile.
That's American politics in a nutshell. We've spent 250 years assuming scruples and common decency would be sufficient.
https://www.shfb.org/
>I'd rather donate to organizations like Second Harvest than pay higher taxes.
I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.
Hence the argument of 'just donate' are just as unconvincing to me.
At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.
The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.
Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".
It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.
> as a species, we can't guarantee the allocation of a livable baseline to every person.
Like we have been discussing, this is pretty much solved in the West.
For food I feel like "as a species" seems impractical? Like humans are not organised "as a species", we are organised into groups, which are supposed to have autonomy and self governance at some level of administration. Specific allocation mechanisms are arrived at and operated by groups of people.
But whose obligation is it exactly to ensure EVERY HUMAN has food? Is that my job in particular? Yours? Not just for like, "your circle" or your neighborhood or country but EVERY HUMAN? How do we even know who EVERY MEMBER OF OUR SPECIES is, does that mean there needs to be a global registry of EVERY HUMAN? Who enforces and has access to that?
The worst failures are mostly happening in other countries. What specifically should we do about that? Food aid exists but it often doesn't reach the people who it's meant to help for various reasons. Who or what group exactly is responsible for making sure everyone gets their livable baseline, and what powers should they have? Should we invade countries that fail to allocate food efficiently, forcibly census their citizens and establish a militarised distribution network to ensure EVERY member of our species is fed?
Is that a fact? Do you have references to back it?
You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.
And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
https://www.fao.org/interactive/hunger-map/en/
Just as an example (ironically from google!): "Google co-founder Sergey Brin has spent $82 million to oppose California's proposed wealth tax."
This is a man that has more money that he could ever hope to possibly spend, and he's spending an absurd amount just because the mere thought that he might benefit from the society he lives in and should contribute back to it apparently infuriates him.
Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.
We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
But the motivating justificatory structure for any inequality in allocation will have completely evaporated.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.
As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]
> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?
[1] - https://taxpolicycenter.org/sites/default/files/publication/...
> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax liability by 14.5 percentage points."
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
Although I imagine you have a point: American police do perhaps act this way.
The beauty of the United States' global hegemony is that it also has lots of sticks
Funny thing about beating people with sticks (or even threatening to) is they tend to want to get out of way and stay away from you.
a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.
maybe you are doing this already, idk. just wanted to share what works for me.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
But as per the documentary of his life, he is wholly focused on AGI and will remain unfulfilled if he, or indeed anyone else, doesn't achieve it within his lifetime.
https://en.wikipedia.org/wiki/The_Thinking_Game
You’re selling yourself short.
Just how smart? "A few short years" sounds like someone smart enough to know how to make a safe prediction.
I should emphasise that DeepMind (London) and Google DeepMind (remember now that Google's research labs across the globe got merged into this entity, including the one in London that was previously split from DeepMind "proper") has always produced good research. But what led to the LLM explosion primarily came from Google Research Mountain View (Transformer), Google Research Seattle (BERT), OpenAI (GPT-3), and various academic labs. During this time DeepMind had a broad research agenda, but my impression is that most resources went towards more traditional deep learning agents in 3D environments, foundational algorithmic research, and games (StarCraft 2). I would provide more detail, but as as an NLP researcher, I did frankly not pay much attention to DeepMind in that era as what was interesting to me just came from elsewhere.
When Google made its first big move into LLMs, its first interesting work did not come out of DeepMind, but rather again Mountain View with PaLM in 2022. Someone with closer connections to Demis at the time can probably explain why the research agenda was what it was, but I lack those connections and DeepMind (even now) has been notoriously tight-lipped since the founding of OpenAI (Demis allegedly sees OpenAI as founded on DeepMind technology by interns and staff walking out on them) and was rather insular compared to Google Research, FAIR, etc. I say "was", but not because DeepMind is now more open, but rather that all industry research outlets are now incredibly tight-lipped after LLMs becoming a product and point of pride among them. At this point, research is held back for maybe even a year before publication or even withheld publication to deny any perceived advantage to the competition.
>"...probably only a few short years away."
Would he not create something directly instead?
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
That's because most execs proposing solutions are "Technocrats" or think like one: https://en.wikipedia.org/wiki/Technocracy
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
The self importance of these AGI prophets turned bureaucrats is funny.
My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now. The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.
Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
Who is watching the watchers?
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
Being good at developing AI and being good at AI safety are diametrically different skillsets with obvious conflicts of interest.
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
>I hope you'll keep this in mind when those milestones are reached.
The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.
None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)
Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.
Then maybe start listening...
Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.