The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election, we put the future of humanity at risk.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; are infiltrated by bad actors; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
> If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
doesn't seem that far from what you are doing with this comment
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.
Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; are infiltrated by bad actors; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
Having humility and delegating a topic to people that understand the topic better than you is not a bad thing, and certainly preferred to this state.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
doesn't seem that far from what you are doing with this comment
More or less the same content but parent is much more palpable, IMHO.
[1] https://news.ycombinator.com/item?id=49737985