I think the "is the world in equilibrium?" criterion really goes out the window with people like Trump. See: Tariffs, incoherent export control policies, and other moves seemingly driven by ego or whim instead of the interest of any nation or even large interest group.
For a policy perspective, I think it is EXTREMELY IMPORTANT to not apply rational actor theory to nations unless the nation has stably followed the predictions of that kind of theory.
To be specific, a rational theory would say "it doesn't matter if Xi Jinping calls Trump a 'pussy on AGI development' on national TV. The nations' incentives remain basically the same." But I guarantee that such a statement would MASSIVELY shift US policy and politics. Granted, such brazen public provocation is unlikely, but I'd imagine that some button pushing happens regardless.
I don't want to claim that the world is actually in equilibrium! However, I think it's still important to use this as a sanity check for plans. If a plan *only works if at least one major actor behaves significantly contrary to its incentives* then that's a mark against the plan. It's not a dealbreaker necessarily. Like, "This plan requires POTUS to behave irrationally, but here's an argument that he probably will behave irrationally in the required way" is pretty reasonable. Is that the kind of reasoning you'd recommend we do instead / the shape of plan we aim to construct?
“A thought transfixed me: for the first time in my life I saw the truth as it is set into song by so many poets, proclaimed as the final wisdom by so many thinkers. The truth — that love is the ultimate and the highest goal to which man can aspire. Then I grasped the meaning of the greatest secret that human poetry and human thought and belief have to impart: The salvation of man is through love and in love.”
— Viktor Frankl (1905–1997), “Man’s Search for Meaning”
This is a solid take on what is necessary for a functional policy, especially the mention of questioning if both major powers would agree on something if there's a power discrepancy. Overall the call for more scrutiny is absolutely valid, I see many hypothetical "policies" but they seem highly dependent on specific actors taking specific steps, which makes the entire scenarios shaky, that's not to say a policy that relies on this impossible, as you stated, but it makes it a lot less viable in my opinion.
I also think there's an over assumption of the inherent goodness of businesses and nations, since they all want to win, which might mean short term benefits for their populations but long term harm to society with AI as a whole, and I think more polices should take this inherent selfishness/wish to succeed into account.
Who is the intended audience of these posts? I'm asking sincerely.
Because obviously you want to change policy *on the grandest scale imaginable*. And yet I earnestly don't know who the target actors/influencers are meant to be. The last I checked, no one this blog runs a branch of government. And the people who 'like and restack' aren't exactly famous (no offense to any of them).
Take this line:
> So, if you have policy proposals to make advanced AI go well, we challenge you to articulate them and then subject them to scenario scrutiny!
Great! And then what? Submit them? To who? To you? To hypothetical meetings in hypothetical halls of power?
I've pounded this drum again and again in this tiny rice-sized public-comment-section: Who are you actually talking to? Where/What is your interface to the non-AI community? Are you talking to individuals in "positions of concrete power"? Are you talking to more "grassroots" operatives and organizers? Are you talking to both the Left and Right? Are you talking to non-STEM people? Are there any skeptics who could still make useful allies?
***I think I'm allowed to keep pounding this drum as long as I see literally ZERO COVERAGE of AI Risk from any of the mainstream "Center to Left" news outlets I regularly watch, read, and listen-to. Not podcasts.***
(Unless you think you can change government policy on this topic *fully* under-the-radar of Mainstream News, Public Debate, Activism, and Election Cycles)
Just a reminder... the few mainstream magazine that *do* sometimes cover AI Risk are always skeptical-of [if not actively ridiculing] the "Doomers".
Our general strategy is: Our top priority is being correct, i.e. getting the ideas right. Our secondary priority is to articulate the ideas clearly and engagingly for a wide audience. Targeting specific powerful groups is maybe third priority. We do in fact talk with people in the government, with partisans on both sides, with miscellaneous other groups, etc. but it's not our main priority.
This strategy seems to have worked historically. I'm open to being convinced it's suboptimal though.
To answer your question about what to do with policy proposals & scenario scrutiny: We think they should typically be published, i.e. submitted to the public.
Not because I agree with her conclusions—I don't, actually. I'm generally supportive of AI 2027 and the broader community. But that article (and this mapping of the x-risk ecosystem https://www.aipanic.news/p/the-ai-existential-risk-industrial) helped me articulate something I've been struggling with: the epistemic environment that lay people encounter when trying to evaluate AI risk claims.
The author is Dr. Nirit Weiss-Blatt, a communication researcher whose "AI Panic" newsletter has reached hundreds of thousands of views and gets quoted in The Washington Post, WIRED, Forbes, and Scientific American as an authoritative voice on AI risk. Her documentation of funding flows and organizational structures is genuinely thorough. Yet her expertise is in media discourse analysis, not AI itself. And while she notes that reviewers "don't reject safety, they reject this unjustified totalizing worldview," it feels like tacked-on-lip-service, and the newsletter is literally named "AI Panic"... which doesn't exactly encourage nuanced engagement with AI Risk.
Here's what concerns me (not just about her work but about the entire ecosystem): intelligent lay-people face an impossible task navigating it. There's no clear hierarchy of credibility, no obvious legitimacy-granting institutions. One commenter on Weiss-Blatt's work... someone who took BlueDot AI Safety courses and participated in existential risk forecasting... made this critique:
"""Something that stands out in my mind is their dearth of publications in refereed journals. In neither of those X-risks activities in which I participated, did the extremists present refereed papers to support their contentions. Clearly, the AI extremists are adverse to peer reviewers. That said, for a view of what the peer-reviewed moderates are saying: https://forecastingresearch.org/publications."""
Then there's AI 2027 itself, which actually makes the point better than anything else could. It's supposedly a very carefully constructed scenario... informed by tabletop exercises, feedback from over 100 experts, and authors with strong track records. Yet it hasn't been traditionally peer-reviewed, and there's genuine debate about whether peer review even applies to this kind of work. Tyler Cowen says it should've been submitted to a journal. Zvi argues "this IS the peer review." Gary Marcus calls it narrative masquerading as forecasting. Karpathy pushes back on the core timelines. You've updated your own estimates since publication.
And the peer review question is just one dimension of the confusion. Which institutions should grant credibility? Which critics deserve serious engagement? Which media coverage reflects expert consensus?
***
And even if the goal is to just "be correct", there's a deeper problem. After 2008, some crash predictors gained credibility—but many others who were also technically 'right' were gradually filtered out once people realized they were just ideological cranks or broken clocks (like Austrian economists who predict crashes every year). That sorting process required exactly the kind of credibility-granting mechanisms I'm saying are currently absent.
***
On another note, I think that engaging qualified technical critics who disagree (people like Robin Hanson whom I mention just because I've seen the FOOM-debate mentioned a few times online in forums-and-threads) might be an extension of the core mission to "be correct," not just communication.
You mentioned you're not in the "direct communication business," and I appreciate you taking time to respond at all. But given how often the community emphasizes time pressure: being correct historically only translates to policy influence if there are functioning mechanisms for others to recognize that correctness before it's too late.
Thanks for this thoughtful reply & critique. I replied above to a different comment of yours, but I'll summarize here: Basically, I agree that critics will try to say concerns about AGI risk are noncredible sci-fi etc. They've been saying that for decades. But it's becoming increasingly difficult to say that with a straight face, because e.g. the world's biggest tech companies are explicitly saying they will build superintelligence soon & because literally the world's most-cited AI scientists are saying it could kill everyone. I'm predicting (and hoping...) that this trend will continue. More prestigious outlets will start acknowledging the problem every year. As for me, being prestigious doesn't seem like my comparative advantage.
At least, that's been my philosophy so far. You seem like a reasonable person & supportive too, so I'm especially interested in your opinion. What would you have me do?
[reiterating a point that's in my longer response, since I added it like 10 hours later, so don't know if you read it]
____
I suspect the implicit response is: "The strategy is being correct—if we're right about "2027", won't that vindicate us?" But being proven correct doesn't automatically translate to influence. After 2008, lots of people who predicted the crash got their 15 minutes of air-time.
There was this one investment broker, Peter Schiff, who I remember vividly because I was temporarily-seduced by his ideas (I was young and stupid). He'd been warning about the housing bubble on CNBC since 2006. And he was right. Technically. So networks brought him on constantly, where he advocated for his preferred policies... end the Fed, return to gold, brace for hyperinflation. But once people decided he was an ideological crank promoting economic suicide (and in fairness, he absolutely is), his bookings dropped off a cliff. He became persona non grata. Being vindicated on the crash didn't give him lasting credibility nor policy influence.
The parallel to now: if the AI risk community is already being framed as fringe or extreme... decried as panic, accused of astroturfing... then even being proven right about "particular events and timelines" might not be enough. You could be heard for 15 minutes but then ***all your further predictions and prescriptions*** could still be dismissed based on your reasoning, credentials, affiliations, etc.
because, in my non-peer-reviewed opinion, credibility/legitimacy mechanisms are currently a mess.
What do you think about Hinton & Bengio? There are plenty of prestigious AI experts saying the risks are real. Didn't seem like my comparative advantage to prestige-maximize.
"If the AI risk community is already being framed as fringe or extreme... decried as panic, accused of astroturfing..."
Theose accusations are false. E.g. a significant fraction of the general public, of ML researchers, of leading AI researchers (e.g. Hinton & Bengio, Ilya and Dario) agree the risks are real. Hopefully over time more people will come to recognize that. I know, the truth doesn't always win out, certainly not in time... but it's reasonable for me to hope for that, I think.
Daniel, you told everyone there's a year left. Well, maybe two - before significant irreversible bad things happen. Yet:
1 - "top priority is being correct, i.e. getting the ideas right" -are you saying you're not yet convinced we have two years left, at most? To use a parable, when you see beams begin to buckle, you don't want to research the best, most correct way to repair the structure, first you need to run to Lowe's to get some 2x4's to prop the thing up.
2 - "articulate the ideas [...] for a wide audience" - Scott A. is on your team, and has OOMs wider reach than this blog, and yet I don't see you regularly showing up at his open threads.
3 - "Targeting specific powerful groups is maybe third priority" - maybe?! you have a year or two left, if you want things to change within this time frame, who do you think can make this change possible? Santa Claus?
Look, as you probably guessed, I don't think some kind of a superhuman AI controlling robotic factories is coming in 2027 or 2028, but I don't matter - if you really believe in your forecasts, your actions don't seem to be aligned with this belief. That's an alignment problem that should be much easier to solve than the AI one.
1. I didn't tell everyone "there's a year left. Well, maybe two." That's uncharitable.
2 &3: You accuse me of not really believing what I'm saying? And your evidence for this is that I am not hanging around Scott's open threads, and not prioritizing hobnobbing with powerful people? Anything else? I'd be curious to hear what you'd be doing, if you had the beliefs I have.
1 - why is this "uncharitable"? Is this: https://blog.ai-futures.org/p/our-first-project-ai-2027 "not" telling "everyone" that there will be "the intelligence explosion in late 2027", or do you think said explosion is not a significant irreversible bad thing? Or does this not qualify as "a year or two left"? Which of these are uncharitable?
2/3 - I am telling you, as an outside skeptical observer, that your actions don't match the urgency of your message. I can't look into your head and see what you truly believe, that's not the point. I read words on your page that state your second priority being "articulate the ideas [...] for a wide audience", I see an easy path to reaching a much wider audience than you are reaching now, and I don't see you taking it.
What would I do? There's a reason I'm an engineer and not a public figure or an activist, but here's off the top of my head: form a PAC, an advocacy org, formulate what political action you propose to delay/avert the worse-case scenario from your forecast, reach out to your followers to call their congress creatures demanding action. Maybe start a tiktoc channel, whatever the young ones do these days to go viral. Something needs to be happening *now* if you want changes affecting 2027. As I keep repeating ad nauseum, with every passing day "2027" is getting more and more set in stone as the chips are moving through fabs, qual tests are being planned, systems are being prototyped, code is being written, etc. etc.
I'm just an anonymous guy typing words into comment boxes, feel free to ignore me, I don't believe in your 2027 scenario anyway, I'm not your target audience. But, just as an example, Jamie Fisher here does, and you're clearly not convincing him that you're acting with the urgency the issue deserves either.
The thing that was uncharitable was the implied level of confidence. We said all along that we don't know when AGI was coming, we gave probability distributions smeared out over the next few years. Our medians were later than 2027; mine was 2028, Eli's was 2030 or 2032 iirc.
Thanks for explaining what you would do. I don't think it's the best strategy, which is why I'm not doing it. E.g. I know nothing of tiktok, don't even have an account, and I feel like it's 'lowbrow' so to speak, not the right audience to target. As for forming a PAC/advocacy org: Again, not my skillset, my skillset is thinking about the future and trying to predict it. Other people are in fact doing advocacy, and probably will do a better job than me.
Re: acting with urgency: I feel like I'm in a double bind here. I'm simultaneously being accused (not by you, by other people) of panicking and overreacting, and (by you) of not panicking enough basically. I'm simultaneously being accused (by others) of being an extremist advocating for radical changes on the basis of insufficient evidence, and (by you) of not being radical enough. I'm going to continue to try to not let the haters get to me and weigh my options carefully and do what seems like the best compromise...
Well, this may be an indicator that the communication strategy is... not quite performing - I'm far from the only person attributing high confidence level to the 2027 number, it for better or worse has become your claim to fame. Maybe a tracker of how your forecast is evolving as more data come in would be handy on the front page?
Look, I'm in this weird position of being an enthusiastic Sci-Fi nerd, fascinated by the possibilities of AI, but also annoyed and frustrated by the utter insanity of people projecting robots completely taking over the economy in a space of few years. I'm a hardware guy, and it seems like y'all have some form of software/CS/algorithm background; which means you don't quite see how vastly different the realm of digits is from the realm of physical things. Yudkowsky is the worst manifestation of this, a classic intellectual idiot with a gift of gab, who knows nothing and sure of everything. But many similar cases all around.
Which is to say, no matter how fast LLMs are progressing, robotics will keep chugging along one physical prototype at a time; Moore's law will keep chugging along one wafer lot at a time, and heat management will keep developing one MEMS pump at a time, nothing will suddenly go FOOM, there's no way to make Moore's law accelerate in any meaningful sense, there's no way to double TSMC capacity next year, etc., etc.
So again, I'm not your problem - when I tell you you're not "radical enough" I'm only channeling concerns of people like Jamie here who use your forecast as another brick in their "AI will kill us in three years" building, and therefore why TF aren't you doing everything to stop it NOW if this is what you believe (do you?). They are the ones who'd benefit from better communication - and better forecasting, get someone kind of a "guy-like-me" with a couple of decades of hardware experience to work with you, to help you avoid nonsense and get some grounding on how fast things progress in "physical" world. Otherwise you'll keep finding yourself in this double-bind where your natural allies think you're dilly daddling while the world is about to burn, and your detractors think you're writing silly sci-fi about the machines taking over in two years.
To summarize: if there's no urgency, make it clear(er). If there is - make that clear(er). Otherwise you get the worst of both worlds.
This is a "good and galvanizing" presentation of AI dangers, imo, even if it doesn't directly mention "extinction". The dangers, imo, are framed similarly to the "slowdown ending" of the AI 2027 forecast, which is still very dystopian.
And it's on a popular and famous television show.
Tristan Harris is great. He even did a Netflix documentary years ago about social media.
(In case my point's unclear, Tristan Harris, manages to slip-in "Doomer Adjacent" scenarios without coming across as a Doomer. Whether or not Tristan personally believes in Extinction Risk... he's calling for a lot of the Same Damn Proposals as you. He's a critical ally. And you need more of them!)
I think the "is the world in equilibrium?" criterion really goes out the window with people like Trump. See: Tariffs, incoherent export control policies, and other moves seemingly driven by ego or whim instead of the interest of any nation or even large interest group.
For a policy perspective, I think it is EXTREMELY IMPORTANT to not apply rational actor theory to nations unless the nation has stably followed the predictions of that kind of theory.
To be specific, a rational theory would say "it doesn't matter if Xi Jinping calls Trump a 'pussy on AGI development' on national TV. The nations' incentives remain basically the same." But I guarantee that such a statement would MASSIVELY shift US policy and politics. Granted, such brazen public provocation is unlikely, but I'd imagine that some button pushing happens regardless.
I don't want to claim that the world is actually in equilibrium! However, I think it's still important to use this as a sanity check for plans. If a plan *only works if at least one major actor behaves significantly contrary to its incentives* then that's a mark against the plan. It's not a dealbreaker necessarily. Like, "This plan requires POTUS to behave irrationally, but here's an argument that he probably will behave irrationally in the required way" is pretty reasonable. Is that the kind of reasoning you'd recommend we do instead / the shape of plan we aim to construct?
All true. Unfortunately
under missing what's important for AI
“A thought transfixed me: for the first time in my life I saw the truth as it is set into song by so many poets, proclaimed as the final wisdom by so many thinkers. The truth — that love is the ultimate and the highest goal to which man can aspire. Then I grasped the meaning of the greatest secret that human poetry and human thought and belief have to impart: The salvation of man is through love and in love.”
— Viktor Frankl (1905–1997), “Man’s Search for Meaning”
If we only knew what love is...or Salvation...or Man.
This is a solid take on what is necessary for a functional policy, especially the mention of questioning if both major powers would agree on something if there's a power discrepancy. Overall the call for more scrutiny is absolutely valid, I see many hypothetical "policies" but they seem highly dependent on specific actors taking specific steps, which makes the entire scenarios shaky, that's not to say a policy that relies on this impossible, as you stated, but it makes it a lot less viable in my opinion.
I also think there's an over assumption of the inherent goodness of businesses and nations, since they all want to win, which might mean short term benefits for their populations but long term harm to society with AI as a whole, and I think more polices should take this inherent selfishness/wish to succeed into account.
Who is the intended audience of these posts? I'm asking sincerely.
Because obviously you want to change policy *on the grandest scale imaginable*. And yet I earnestly don't know who the target actors/influencers are meant to be. The last I checked, no one this blog runs a branch of government. And the people who 'like and restack' aren't exactly famous (no offense to any of them).
Take this line:
> So, if you have policy proposals to make advanced AI go well, we challenge you to articulate them and then subject them to scenario scrutiny!
Great! And then what? Submit them? To who? To you? To hypothetical meetings in hypothetical halls of power?
I've pounded this drum again and again in this tiny rice-sized public-comment-section: Who are you actually talking to? Where/What is your interface to the non-AI community? Are you talking to individuals in "positions of concrete power"? Are you talking to more "grassroots" operatives and organizers? Are you talking to both the Left and Right? Are you talking to non-STEM people? Are there any skeptics who could still make useful allies?
***I think I'm allowed to keep pounding this drum as long as I see literally ZERO COVERAGE of AI Risk from any of the mainstream "Center to Left" news outlets I regularly watch, read, and listen-to. Not podcasts.***
(Unless you think you can change government policy on this topic *fully* under-the-radar of Mainstream News, Public Debate, Activism, and Election Cycles)
Just a reminder... the few mainstream magazine that *do* sometimes cover AI Risk are always skeptical-of [if not actively ridiculing] the "Doomers".
Our general strategy is: Our top priority is being correct, i.e. getting the ideas right. Our secondary priority is to articulate the ideas clearly and engagingly for a wide audience. Targeting specific powerful groups is maybe third priority. We do in fact talk with people in the government, with partisans on both sides, with miscellaneous other groups, etc. but it's not our main priority.
This strategy seems to have worked historically. I'm open to being convinced it's suboptimal though.
To answer your question about what to do with policy proposals & scenario scrutiny: We think they should typically be published, i.e. submitted to the public.
I struggled to articulate the perfect response to your response... but then by waiting 24 hours, I literally found a blog post that almost does it for me. https://www.aipanic.news/p/why-the-doom-bible-left-many-reviewers
Not because I agree with her conclusions—I don't, actually. I'm generally supportive of AI 2027 and the broader community. But that article (and this mapping of the x-risk ecosystem https://www.aipanic.news/p/the-ai-existential-risk-industrial) helped me articulate something I've been struggling with: the epistemic environment that lay people encounter when trying to evaluate AI risk claims.
The author is Dr. Nirit Weiss-Blatt, a communication researcher whose "AI Panic" newsletter has reached hundreds of thousands of views and gets quoted in The Washington Post, WIRED, Forbes, and Scientific American as an authoritative voice on AI risk. Her documentation of funding flows and organizational structures is genuinely thorough. Yet her expertise is in media discourse analysis, not AI itself. And while she notes that reviewers "don't reject safety, they reject this unjustified totalizing worldview," it feels like tacked-on-lip-service, and the newsletter is literally named "AI Panic"... which doesn't exactly encourage nuanced engagement with AI Risk.
Here's what concerns me (not just about her work but about the entire ecosystem): intelligent lay-people face an impossible task navigating it. There's no clear hierarchy of credibility, no obvious legitimacy-granting institutions. One commenter on Weiss-Blatt's work... someone who took BlueDot AI Safety courses and participated in existential risk forecasting... made this critique:
"""Something that stands out in my mind is their dearth of publications in refereed journals. In neither of those X-risks activities in which I participated, did the extremists present refereed papers to support their contentions. Clearly, the AI extremists are adverse to peer reviewers. That said, for a view of what the peer-reviewed moderates are saying: https://forecastingresearch.org/publications."""
Then there's AI 2027 itself, which actually makes the point better than anything else could. It's supposedly a very carefully constructed scenario... informed by tabletop exercises, feedback from over 100 experts, and authors with strong track records. Yet it hasn't been traditionally peer-reviewed, and there's genuine debate about whether peer review even applies to this kind of work. Tyler Cowen says it should've been submitted to a journal. Zvi argues "this IS the peer review." Gary Marcus calls it narrative masquerading as forecasting. Karpathy pushes back on the core timelines. You've updated your own estimates since publication.
And the peer review question is just one dimension of the confusion. Which institutions should grant credibility? Which critics deserve serious engagement? Which media coverage reflects expert consensus?
***
And even if the goal is to just "be correct", there's a deeper problem. After 2008, some crash predictors gained credibility—but many others who were also technically 'right' were gradually filtered out once people realized they were just ideological cranks or broken clocks (like Austrian economists who predict crashes every year). That sorting process required exactly the kind of credibility-granting mechanisms I'm saying are currently absent.
***
On another note, I think that engaging qualified technical critics who disagree (people like Robin Hanson whom I mention just because I've seen the FOOM-debate mentioned a few times online in forums-and-threads) might be an extension of the core mission to "be correct," not just communication.
You mentioned you're not in the "direct communication business," and I appreciate you taking time to respond at all. But given how often the community emphasizes time pressure: being correct historically only translates to policy influence if there are functioning mechanisms for others to recognize that correctness before it's too late.
Thanks for this work. It matters.
Thanks for this thoughtful reply & critique. I replied above to a different comment of yours, but I'll summarize here: Basically, I agree that critics will try to say concerns about AGI risk are noncredible sci-fi etc. They've been saying that for decades. But it's becoming increasingly difficult to say that with a straight face, because e.g. the world's biggest tech companies are explicitly saying they will build superintelligence soon & because literally the world's most-cited AI scientists are saying it could kill everyone. I'm predicting (and hoping...) that this trend will continue. More prestigious outlets will start acknowledging the problem every year. As for me, being prestigious doesn't seem like my comparative advantage.
At least, that's been my philosophy so far. You seem like a reasonable person & supportive too, so I'm especially interested in your opinion. What would you have me do?
[reiterating a point that's in my longer response, since I added it like 10 hours later, so don't know if you read it]
____
I suspect the implicit response is: "The strategy is being correct—if we're right about "2027", won't that vindicate us?" But being proven correct doesn't automatically translate to influence. After 2008, lots of people who predicted the crash got their 15 minutes of air-time.
There was this one investment broker, Peter Schiff, who I remember vividly because I was temporarily-seduced by his ideas (I was young and stupid). He'd been warning about the housing bubble on CNBC since 2006. And he was right. Technically. So networks brought him on constantly, where he advocated for his preferred policies... end the Fed, return to gold, brace for hyperinflation. But once people decided he was an ideological crank promoting economic suicide (and in fairness, he absolutely is), his bookings dropped off a cliff. He became persona non grata. Being vindicated on the crash didn't give him lasting credibility nor policy influence.
The parallel to now: if the AI risk community is already being framed as fringe or extreme... decried as panic, accused of astroturfing... then even being proven right about "particular events and timelines" might not be enough. You could be heard for 15 minutes but then ***all your further predictions and prescriptions*** could still be dismissed based on your reasoning, credentials, affiliations, etc.
because, in my non-peer-reviewed opinion, credibility/legitimacy mechanisms are currently a mess.
What do you think about Hinton & Bengio? There are plenty of prestigious AI experts saying the risks are real. Didn't seem like my comparative advantage to prestige-maximize.
"If the AI risk community is already being framed as fringe or extreme... decried as panic, accused of astroturfing..."
Theose accusations are false. E.g. a significant fraction of the general public, of ML researchers, of leading AI researchers (e.g. Hinton & Bengio, Ilya and Dario) agree the risks are real. Hopefully over time more people will come to recognize that. I know, the truth doesn't always win out, certainly not in time... but it's reasonable for me to hope for that, I think.
Daniel, you told everyone there's a year left. Well, maybe two - before significant irreversible bad things happen. Yet:
1 - "top priority is being correct, i.e. getting the ideas right" -are you saying you're not yet convinced we have two years left, at most? To use a parable, when you see beams begin to buckle, you don't want to research the best, most correct way to repair the structure, first you need to run to Lowe's to get some 2x4's to prop the thing up.
2 - "articulate the ideas [...] for a wide audience" - Scott A. is on your team, and has OOMs wider reach than this blog, and yet I don't see you regularly showing up at his open threads.
3 - "Targeting specific powerful groups is maybe third priority" - maybe?! you have a year or two left, if you want things to change within this time frame, who do you think can make this change possible? Santa Claus?
Look, as you probably guessed, I don't think some kind of a superhuman AI controlling robotic factories is coming in 2027 or 2028, but I don't matter - if you really believe in your forecasts, your actions don't seem to be aligned with this belief. That's an alignment problem that should be much easier to solve than the AI one.
1. I didn't tell everyone "there's a year left. Well, maybe two." That's uncharitable.
2 &3: You accuse me of not really believing what I'm saying? And your evidence for this is that I am not hanging around Scott's open threads, and not prioritizing hobnobbing with powerful people? Anything else? I'd be curious to hear what you'd be doing, if you had the beliefs I have.
1 - why is this "uncharitable"? Is this: https://blog.ai-futures.org/p/our-first-project-ai-2027 "not" telling "everyone" that there will be "the intelligence explosion in late 2027", or do you think said explosion is not a significant irreversible bad thing? Or does this not qualify as "a year or two left"? Which of these are uncharitable?
2/3 - I am telling you, as an outside skeptical observer, that your actions don't match the urgency of your message. I can't look into your head and see what you truly believe, that's not the point. I read words on your page that state your second priority being "articulate the ideas [...] for a wide audience", I see an easy path to reaching a much wider audience than you are reaching now, and I don't see you taking it.
What would I do? There's a reason I'm an engineer and not a public figure or an activist, but here's off the top of my head: form a PAC, an advocacy org, formulate what political action you propose to delay/avert the worse-case scenario from your forecast, reach out to your followers to call their congress creatures demanding action. Maybe start a tiktoc channel, whatever the young ones do these days to go viral. Something needs to be happening *now* if you want changes affecting 2027. As I keep repeating ad nauseum, with every passing day "2027" is getting more and more set in stone as the chips are moving through fabs, qual tests are being planned, systems are being prototyped, code is being written, etc. etc.
I'm just an anonymous guy typing words into comment boxes, feel free to ignore me, I don't believe in your 2027 scenario anyway, I'm not your target audience. But, just as an example, Jamie Fisher here does, and you're clearly not convincing him that you're acting with the urgency the issue deserves either.
The thing that was uncharitable was the implied level of confidence. We said all along that we don't know when AGI was coming, we gave probability distributions smeared out over the next few years. Our medians were later than 2027; mine was 2028, Eli's was 2030 or 2032 iirc.
Thanks for explaining what you would do. I don't think it's the best strategy, which is why I'm not doing it. E.g. I know nothing of tiktok, don't even have an account, and I feel like it's 'lowbrow' so to speak, not the right audience to target. As for forming a PAC/advocacy org: Again, not my skillset, my skillset is thinking about the future and trying to predict it. Other people are in fact doing advocacy, and probably will do a better job than me.
Re: acting with urgency: I feel like I'm in a double bind here. I'm simultaneously being accused (not by you, by other people) of panicking and overreacting, and (by you) of not panicking enough basically. I'm simultaneously being accused (by others) of being an extremist advocating for radical changes on the basis of insufficient evidence, and (by you) of not being radical enough. I'm going to continue to try to not let the haters get to me and weigh my options carefully and do what seems like the best compromise...
Well, this may be an indicator that the communication strategy is... not quite performing - I'm far from the only person attributing high confidence level to the 2027 number, it for better or worse has become your claim to fame. Maybe a tracker of how your forecast is evolving as more data come in would be handy on the front page?
Look, I'm in this weird position of being an enthusiastic Sci-Fi nerd, fascinated by the possibilities of AI, but also annoyed and frustrated by the utter insanity of people projecting robots completely taking over the economy in a space of few years. I'm a hardware guy, and it seems like y'all have some form of software/CS/algorithm background; which means you don't quite see how vastly different the realm of digits is from the realm of physical things. Yudkowsky is the worst manifestation of this, a classic intellectual idiot with a gift of gab, who knows nothing and sure of everything. But many similar cases all around.
Which is to say, no matter how fast LLMs are progressing, robotics will keep chugging along one physical prototype at a time; Moore's law will keep chugging along one wafer lot at a time, and heat management will keep developing one MEMS pump at a time, nothing will suddenly go FOOM, there's no way to make Moore's law accelerate in any meaningful sense, there's no way to double TSMC capacity next year, etc., etc.
So again, I'm not your problem - when I tell you you're not "radical enough" I'm only channeling concerns of people like Jamie here who use your forecast as another brick in their "AI will kill us in three years" building, and therefore why TF aren't you doing everything to stop it NOW if this is what you believe (do you?). They are the ones who'd benefit from better communication - and better forecasting, get someone kind of a "guy-like-me" with a couple of decades of hardware experience to work with you, to help you avoid nonsense and get some grounding on how fast things progress in "physical" world. Otherwise you'll keep finding yourself in this double-bind where your natural allies think you're dilly daddling while the world is about to burn, and your detractors think you're writing silly sci-fi about the machines taking over in two years.
To summarize: if there's no urgency, make it clear(er). If there is - make that clear(er). Otherwise you get the worst of both worlds.
https://www.youtube.com/watch?v=675d_6WGPbo
This is a "good and galvanizing" presentation of AI dangers, imo, even if it doesn't directly mention "extinction". The dangers, imo, are framed similarly to the "slowdown ending" of the AI 2027 forecast, which is still very dystopian.
And it's on a popular and famous television show.
Tristan Harris is great. He even did a Netflix documentary years ago about social media.
(In case my point's unclear, Tristan Harris, manages to slip-in "Doomer Adjacent" scenarios without coming across as a Doomer. Whether or not Tristan personally believes in Extinction Risk... he's calling for a lot of the Same Damn Proposals as you. He's a critical ally. And you need more of them!)