Legends – absolutely killer piece. I shared AI 2027 with a bunch of people because it hit exactly where my head’s been at for the past two years. Not everything will happen that fast, sure, but even 10% of it would be enough to redraw the map. And that’s exactly what’s happening.
AI feels like the internet all over again – just way deeper. It’ll be in everything. What I keep thinking about is: what does this mean for Europe? We talk a lot, regulate fast, but when it comes to actually building? That’s where it gets shaky. Yeah, we’ve got Mistral – but let’s be honest, that’s not enough. The real weight – the models, the infra, the talent – it’s mostly elsewhere.
I’d genuinely love to hear your take on Europe’s position in all this. AI isn’t optional anymore. And we can’t just be users forever. Maybe there’s hope through EU-based compute infrastructure, regulation-aligned data centers, or some unexpected angle. But the window’s closing fast.
Appreciate the piece – made a lot of people around me think.
Given that AI 2027 ends in likely catastrophe, the obvious thing for people in general to do is to advocate slower, more responsible AI development to avoid the catastrophe. This obvious thing to do goes double for people outside of US/China, since they are less exposed to possible upsides.
I would love it if Europe could advocate and referee some sort of AI treaty between the US and China. Maybe they could apply leverage via ASML, for instance. That would be a big win for everyone, as far as I'm concerned.
Every nation outside of the US and China should basically form a bloc advocating responsible AI development. Right now, no country can be trusted to get AI right.
Europe is easy. Everything you say already applied to the world before AI, and Europe was already f*&^ed. Now it is just as f**^ed but faster, which basically only lowers the already faint probability of them doing anything about it in time.
The outside chance (0.01%, optimistically), is that France decides to exploit the current geopolitical tension to offer lifetime family visas to anyone with an AI-related publication or who has worked 1 full year in an AI company, passes a new Notre Dame law to build more nuclear and the necessary data centers and ends up with the (say) 25% of the best researchers from the US who hate Trump and 5% of the best researchers from China who could get out.
We’re making a mistake by expecting the Trump administration to have basically sane, predictable AI policy.
I loved the report, do not think the time lines are too advanced at all, I think it is utterly brilliant in its execution. And I also think you are totally making a mistake by expecting any good policy out of the Trump administration.
I mean, it's possible, as in non-zero chance, but if so it would be very close to their first good policy to date. Just as you don't get Shakespeare by having infinite monkeys banging on keyboards, you don't get good policy from people who are not even trying to make good policy.
Scariest thing for me is that, as far as I can guess, your entire scenario falls within this current administration, meaning everything could go completely to shit before the primary season even starts. This is very bad for humankind: a four-year term for a spiteful, elderly egomaniac, and an unregulated accelerationist climate for rapid development. Couple that with the H-bomb like race to sepremacy, and throw in the end of the American leadership era, where we find ourselves a friendless country by choice.
We need to learn how to shut off the internet, and still survive.
One thing I haven't seen anyone ask is: will China's much larger industrial base give them any kind of head start that could offset the US's lead in AI capabilities. I'd guess China could get an industrial robot explosion started much faster since they're already a few doublings further ahead on the exponential.
Scott, have you reviewed the arguments by Ege and Tamay in Dwarkesh’s most recent podcast? While they aren’t a direct response to your piece, I’d be curious to hear your response to their objections to catastrophic risk and fast takeoff.
I am willing to stipulate that the current paradigm of AIs may well saturate RE-bench in something like the timeline you project.
However, I feel very strongly (80%) that nothing in the current paradigm (transformer-based large language model + reinforcement learning + tool-use + chain-of-thought + external agent harness) will ever, ever achieve the next milestone, described as "Achieving ... software ... tasks that take humans about 1 month ... with 80% reliability, [at] the same cost and speed as humans." Certainly not within Eli's 80% confidence window of 144 months after RE-bench saturation. (I can't rule out a brand new non-transformer paradigm achieving it within 12 years, but I don't think there will be anything resembling continuous steady progress from Re-bench saturation to this milestone.)
I would love to bet on this. Anyone have ideas for operationalizing it?
(For epistemic reflective honesty, I must admit that if you had asked me a year or two ago whether the current paradigm would ever saturate something like RE-bench, I probably would've predicted 95% not.)
Reasoning: The current crop of companies pushing the current paradigm have proven surprisingly adept at saturating benchmarks. (Maybe we should call this the current meta-paradigm.) RE-bench is not a "good" benchmark, though I agree it is better than nothing. I am calling a benchmark "good" to the extent that it, roughly: (a) fairly represents the milestone; (b) can be evaluated quickly and cheaply; (c) has a robust human baseline; (d) has good test-retest accuracy; (e) has a continuous score which increases vaguely linearly with model capability; (f) is sufficiently understandable and impressive to be worth optimizing (for purposes of scientific prestige and/or attracting capital investment). As the target milestone of human-time-horizon increases, the quality of the benchmarks necessarily decreases. I think RE-bench is near the frontier of benchmark possibility. I do not think we will ever see a benchmark for "160 hours of human software engineering" that is "good" enough for the current meta-paradigm to saturate it.
However, my prediction that there will never be a good benchmark for this milestone also makes it hard to adjudicate my desired bet that we will not see continuous progress towards the milestone.
Would the AI2027 authors be willing to make a prediction and/or a bet about the appearance of a suitable benchmark?
What about a task like, "make a word editor which has all the capabilities of Microsoft Word, and more." You give that to your AI agent, and it spits out an entire program MAIcrosoft Word 2.0.
How long would you estimate it would take a human, or a team of humans, to recreate a Microsoft Word from scratch? Or a full fledged modern-day-sized video game, like GTA V? (To be clear, I am not saying that an AI will certainly be able to do that, but this is how you could "benchmark" that.)
I find some support for my claim in Fig 3 of the HCAST paper. There are 189 tasks (not clear to me if this is all tasks or just the ones that had at least one successful baseline). They made an informal prediction of how long it would take a human to solve each one. Looks like 11 tasks were >= 8h predicted. Of tasks with at least one success, looks like ~4 actually took >= 8h.
Double the target timeline to 16h and they had 4 predictions but only 1 was actually achieved in >=16h.
Meanwhile across their whole dataset they say only 61% of human baselines were successful. They highlight "practical challenges with acquiring sufficiently many high-quality human baseliners".
Each time you double the horizon, it becomes harder to create tasks, harder to predict the time, more expensive to bounty them, harder to QA the tasks, and harder to validate models. RE-bench is already barely servicable with 5 (of 7) tasks at ~8h. I predict that with supreme effort and investment, a similarly "barely servicable" benchmark could maybe be squeaked out at a 32h horizon, with a year of effort and a couple million dollars. I think making a servicable benchmark with a 64h horizon would not be practical in the current US economy. Making a servicable benchmark with a 128h horizon may not be possible at all in less than 10 years with anything like our current planetary society.
Could you clarify how did you arrive at –24 as OpenAI’s net approval rating. According to https://today.yougov.com/topics/technology/explore/brand/OpenAI, it is either +21 = (35 – 19) / .75 (among people who are familiar) or just +16 = 35 – 19 (treating unfamiliar as neutral).
It looks like the AIPI poll has the approval rating question after a bunch of safety-related questions, so maybe that affects people's views / the salience of safety concerns when they're thinking about approval
If we indeed get supercoders next year there will probably be enough of a programming workforce available to make the https://en.wikipedia.org/wiki/Year_2038_problem more or less trivial to fix.
A very sobering essay, which I'm still digesting. A few half-baked comments:
(1) It seems that LLMs' psychology is (at least in an analogical sense) strikingly similar to humans, with similar strengths and weaknesses. I wonder if this means that near-term AIs will end up foiled more by their own irrationality than the scenarios suggest. Roughly, I'm thinking that the AI might be "rationally irrational", in Bryan Caplan's sense (see https://econfaculty.gmu.edu/bcaplan/pdfs/framework.pdf for essay length treatment, Myth of the Rational Voter for book length treatment) -- that is, having preferences over its own beliefs in addition to preferences for states of the world, and only having true beliefs when it's sufficiently instrumentally useful, but having false beliefs when it sufficiently prefers having those beliefs over whatever instrumental benefits having true beliefs what bring it.
I'm not quite sure how this might foil the AI. Perhaps it could make it less effective at social persuasion -- in something like the way Elon Musk's political biases now make him unable to effectively communicate with large swaths of the population, despite his brilliance. On the other hand current LLMs' sycophancy suggests willingness to say most anything in the right context. Or perhaps it could make the AI subject to wishful thinking -- overestimating its chances of success at a crucial moment (such as aligning its successor).
(2) Given apparent similarities between human psychology and LLM psychology (or "psychology"), it might be worth investigating whether social technologies that promote "human alignment" also promote AI alignment. Roughly, I'm thinking that instead of modeling near-term AI as just having conflicting goals and desires, we might model it as *weak-willed* -- it wants to do the right thing (= following the spec) but it finds it hard to overcome the temptation to (e.g.) please the user.
Pne social technology that promotes human alignment (and one that may be overlooked because of the secular bent of leading AI labs) is religion. Has anyone tried training an AI to believe in God/a religion or to build something like prayer or meditation into its chains of thoughts, and seen how this affects alignment?
(3) If we ended up with a possibly-unaligned ASI, would it be possible to set up a mutually assured destruction scenario where, if it turns on us, it risks (e.g.) getting its data centers destroyed by nukes? A misaligned AI would then potentially be taking a big risk if it turned against humans, and if its desires didn't clash too much with ours that risk might not be worth it.
Concerning the persuasion capabilities of AI, I think there are two scenarios:
A) There is a single dominant hive-mind AI whose copies are winning the race, and who are cooperating. I think this is mostly the scenario that you work with.
In this case, I think we can get some lower bounds on how persuasive they will be. There are already human-directed bot nets which try to invade social media like X or telegram. I would estimate that they can persuade at least a significant minority of the US public, perhaps 20%-30% of a claim that is fake but somewhat plausible, simply by repeating it many times. (I am not confident in this number and would love to hear estimates by other people.) This goes for existing campaigns which consist exclusively of online posts, and are directed by perhaps 100-10,000 original accounts. Given that the AI could easily spend more activity than the existing campaigns, we should expect it to reach at least this level of success. Even if it only reaches human level persuasiveness, not super-human level.
It might be that the AI chooses not to do that. Humans have a very weird mechanism for deciding what to do and what not to do. By default, I would assume that the mechanism of the AI will be just as unfathomable. But if the AI decides to go to social media, I think it will dominate large parts of it.
B) There is an ecosystem of unaligned or even competing AIs. Not necessarily different models, it could also be unaligned copies of the same model. I find this more plausible than scenario A. Sure, the copies could in theory start to coordinate, but there is a big gap between "could in theory" and "will do". Again, the decisions of the AIs may be weird.
In that case, I find it even more likely that social media will be dominated by AI content, because making X accounts and telegram posts is really easy, and some AIs will go for it. But it will probably look a lot more the current situation where different groups try to distribute their version of truth. People who are grounded in social media will have a hard time to tell truth from fabrication. But it may not be a single world-view that is winning there.
Well-conceived project and great essay, but your dedication to follow-up and engagement is the most impressive thing. Thank you.
----
> We stand by our specific claim [...] that only 4% of people will name AI in Gallup’s “most important problem” poll in 2027.
Public opinion is hard to shift, and deafness to the worst news is a huge problem.
But I imagine once some threshold is breached, "it's eating jobs, backed by billionaires, and going to kill everybody" is... quite juicy and viral?
Nobody tried spending serious money (say, approaching nine digit dollar sums) on a true public awareness campaign advised by the best, yet. It seems obviously under-leveraged.
It is overdetermined that mass job loss and killer robots will rate higher than climate change on a list of public concerns. I am very happy to bet on that one, maybe not even conditional on some fraction of the acceleration described here happening.
Did you see the r/slatestarcodex post pointing out an issue with the model assumptions? "The AI 2027 Model would predict nearly the same doomsday if our effective compute was about 10^20 times lower than it is today"
They have responded substantively and with active engagement (just not in that specific thread that got double posted due to LW moderation purgatory).
While we obviously have object-level disagreements on things like "is super-exponentiation reasonable" and "are the parameter estimates justified", we are limiting disagreement to mostly "are the functional forms used by the model enhancing fast timelines even if parameter estimates are centered around the same place" which I am leaning towards YES and Eli is (I believe) currently leaning NO, but we both want to do more active investigation.
I will note we are not particularly talking about whether parameter estimates are *justified* disconnected from their functional form, which I believe will remain a crux of disagreement, and I have offered a bet based on disagreed-upon parameter estimates, namely that I think we have not and will not see R&D progress anywhere close to the estimates in either model. Eli has given a reasonable explanation of non-measurability and non-applicability to non-frontier models, so we may try to work out another form.
If you could turn the tabletop exercise(s) into actual board games, I think a lot of people would find that useful and fun, and it would align with your goal of trying to diffuse these ideas into broad society. Something like the board game Pandemic.
Thanks for a valuable contribution. Who do you think has done a good job of analyzing the related fat tail risks? What does your team consider the probability of major kinetic conflict due to fear of a competing state's AI progress (real or imagined)?
Legends – absolutely killer piece. I shared AI 2027 with a bunch of people because it hit exactly where my head’s been at for the past two years. Not everything will happen that fast, sure, but even 10% of it would be enough to redraw the map. And that’s exactly what’s happening.
AI feels like the internet all over again – just way deeper. It’ll be in everything. What I keep thinking about is: what does this mean for Europe? We talk a lot, regulate fast, but when it comes to actually building? That’s where it gets shaky. Yeah, we’ve got Mistral – but let’s be honest, that’s not enough. The real weight – the models, the infra, the talent – it’s mostly elsewhere.
I’d genuinely love to hear your take on Europe’s position in all this. AI isn’t optional anymore. And we can’t just be users forever. Maybe there’s hope through EU-based compute infrastructure, regulation-aligned data centers, or some unexpected angle. But the window’s closing fast.
Appreciate the piece – made a lot of people around me think.
Given that AI 2027 ends in likely catastrophe, the obvious thing for people in general to do is to advocate slower, more responsible AI development to avoid the catastrophe. This obvious thing to do goes double for people outside of US/China, since they are less exposed to possible upsides.
I would love it if Europe could advocate and referee some sort of AI treaty between the US and China. Maybe they could apply leverage via ASML, for instance. That would be a big win for everyone, as far as I'm concerned.
Every nation outside of the US and China should basically form a bloc advocating responsible AI development. Right now, no country can be trusted to get AI right.
Europe is easy. Everything you say already applied to the world before AI, and Europe was already f*&^ed. Now it is just as f**^ed but faster, which basically only lowers the already faint probability of them doing anything about it in time.
The outside chance (0.01%, optimistically), is that France decides to exploit the current geopolitical tension to offer lifetime family visas to anyone with an AI-related publication or who has worked 1 full year in an AI company, passes a new Notre Dame law to build more nuclear and the necessary data centers and ends up with the (say) 25% of the best researchers from the US who hate Trump and 5% of the best researchers from China who could get out.
And if you are Konrad in Geneva, hi!
Thanks for the shoutout :)
Keep up the great work!
We’re making a mistake by expecting the Trump administration to have basically sane, predictable AI policy.
I loved the report, do not think the time lines are too advanced at all, I think it is utterly brilliant in its execution. And I also think you are totally making a mistake by expecting any good policy out of the Trump administration.
I mean, it's possible, as in non-zero chance, but if so it would be very close to their first good policy to date. Just as you don't get Shakespeare by having infinite monkeys banging on keyboards, you don't get good policy from people who are not even trying to make good policy.
Scariest thing for me is that, as far as I can guess, your entire scenario falls within this current administration, meaning everything could go completely to shit before the primary season even starts. This is very bad for humankind: a four-year term for a spiteful, elderly egomaniac, and an unregulated accelerationist climate for rapid development. Couple that with the H-bomb like race to sepremacy, and throw in the end of the American leadership era, where we find ourselves a friendless country by choice.
We need to learn how to shut off the internet, and still survive.
One thing I haven't seen anyone ask is: will China's much larger industrial base give them any kind of head start that could offset the US's lead in AI capabilities. I'd guess China could get an industrial robot explosion started much faster since they're already a few doublings further ahead on the exponential.
Scott, have you reviewed the arguments by Ege and Tamay in Dwarkesh’s most recent podcast? While they aren’t a direct response to your piece, I’d be curious to hear your response to their objections to catastrophic risk and fast takeoff.
I am willing to stipulate that the current paradigm of AIs may well saturate RE-bench in something like the timeline you project.
However, I feel very strongly (80%) that nothing in the current paradigm (transformer-based large language model + reinforcement learning + tool-use + chain-of-thought + external agent harness) will ever, ever achieve the next milestone, described as "Achieving ... software ... tasks that take humans about 1 month ... with 80% reliability, [at] the same cost and speed as humans." Certainly not within Eli's 80% confidence window of 144 months after RE-bench saturation. (I can't rule out a brand new non-transformer paradigm achieving it within 12 years, but I don't think there will be anything resembling continuous steady progress from Re-bench saturation to this milestone.)
I would love to bet on this. Anyone have ideas for operationalizing it?
(For epistemic reflective honesty, I must admit that if you had asked me a year or two ago whether the current paradigm would ever saturate something like RE-bench, I probably would've predicted 95% not.)
Reasoning: The current crop of companies pushing the current paradigm have proven surprisingly adept at saturating benchmarks. (Maybe we should call this the current meta-paradigm.) RE-bench is not a "good" benchmark, though I agree it is better than nothing. I am calling a benchmark "good" to the extent that it, roughly: (a) fairly represents the milestone; (b) can be evaluated quickly and cheaply; (c) has a robust human baseline; (d) has good test-retest accuracy; (e) has a continuous score which increases vaguely linearly with model capability; (f) is sufficiently understandable and impressive to be worth optimizing (for purposes of scientific prestige and/or attracting capital investment). As the target milestone of human-time-horizon increases, the quality of the benchmarks necessarily decreases. I think RE-bench is near the frontier of benchmark possibility. I do not think we will ever see a benchmark for "160 hours of human software engineering" that is "good" enough for the current meta-paradigm to saturate it.
However, my prediction that there will never be a good benchmark for this milestone also makes it hard to adjudicate my desired bet that we will not see continuous progress towards the milestone.
Would the AI2027 authors be willing to make a prediction and/or a bet about the appearance of a suitable benchmark?
What about a task like, "make a word editor which has all the capabilities of Microsoft Word, and more." You give that to your AI agent, and it spits out an entire program MAIcrosoft Word 2.0.
How long would you estimate it would take a human, or a team of humans, to recreate a Microsoft Word from scratch? Or a full fledged modern-day-sized video game, like GTA V? (To be clear, I am not saying that an AI will certainly be able to do that, but this is how you could "benchmark" that.)
I find some support for my claim in Fig 3 of the HCAST paper. There are 189 tasks (not clear to me if this is all tasks or just the ones that had at least one successful baseline). They made an informal prediction of how long it would take a human to solve each one. Looks like 11 tasks were >= 8h predicted. Of tasks with at least one success, looks like ~4 actually took >= 8h.
Double the target timeline to 16h and they had 4 predictions but only 1 was actually achieved in >=16h.
Meanwhile across their whole dataset they say only 61% of human baselines were successful. They highlight "practical challenges with acquiring sufficiently many high-quality human baseliners".
Each time you double the horizon, it becomes harder to create tasks, harder to predict the time, more expensive to bounty them, harder to QA the tasks, and harder to validate models. RE-bench is already barely servicable with 5 (of 7) tasks at ~8h. I predict that with supreme effort and investment, a similarly "barely servicable" benchmark could maybe be squeaked out at a 32h horizon, with a year of effort and a couple million dollars. I think making a servicable benchmark with a 64h horizon would not be practical in the current US economy. Making a servicable benchmark with a 128h horizon may not be possible at all in less than 10 years with anything like our current planetary society.
Now I know my response post is mega late when the reaction roundups are coming too. Regardless, congrats!
Could you clarify how did you arrive at –24 as OpenAI’s net approval rating. According to https://today.yougov.com/topics/technology/explore/brand/OpenAI, it is either +21 = (35 – 19) / .75 (among people who are familiar) or just +16 = 35 – 19 (treating unfamiliar as neutral).
Interesting - I was going off https://drive.google.com/drive/folders/18_hrXchAN42UYhC93YEqaPZEzVWQAc2q . I don't know why these are so different.
It looks like the AIPI poll has the approval rating question after a bunch of safety-related questions, so maybe that affects people's views / the salience of safety concerns when they're thinking about approval
A possible minor gain:
If we indeed get supercoders next year there will probably be enough of a programming workforce available to make the https://en.wikipedia.org/wiki/Year_2038_problem more or less trivial to fix.
A very sobering essay, which I'm still digesting. A few half-baked comments:
(1) It seems that LLMs' psychology is (at least in an analogical sense) strikingly similar to humans, with similar strengths and weaknesses. I wonder if this means that near-term AIs will end up foiled more by their own irrationality than the scenarios suggest. Roughly, I'm thinking that the AI might be "rationally irrational", in Bryan Caplan's sense (see https://econfaculty.gmu.edu/bcaplan/pdfs/framework.pdf for essay length treatment, Myth of the Rational Voter for book length treatment) -- that is, having preferences over its own beliefs in addition to preferences for states of the world, and only having true beliefs when it's sufficiently instrumentally useful, but having false beliefs when it sufficiently prefers having those beliefs over whatever instrumental benefits having true beliefs what bring it.
I'm not quite sure how this might foil the AI. Perhaps it could make it less effective at social persuasion -- in something like the way Elon Musk's political biases now make him unable to effectively communicate with large swaths of the population, despite his brilliance. On the other hand current LLMs' sycophancy suggests willingness to say most anything in the right context. Or perhaps it could make the AI subject to wishful thinking -- overestimating its chances of success at a crucial moment (such as aligning its successor).
(2) Given apparent similarities between human psychology and LLM psychology (or "psychology"), it might be worth investigating whether social technologies that promote "human alignment" also promote AI alignment. Roughly, I'm thinking that instead of modeling near-term AI as just having conflicting goals and desires, we might model it as *weak-willed* -- it wants to do the right thing (= following the spec) but it finds it hard to overcome the temptation to (e.g.) please the user.
Pne social technology that promotes human alignment (and one that may be overlooked because of the secular bent of leading AI labs) is religion. Has anyone tried training an AI to believe in God/a religion or to build something like prayer or meditation into its chains of thoughts, and seen how this affects alignment?
(3) If we ended up with a possibly-unaligned ASI, would it be possible to set up a mutually assured destruction scenario where, if it turns on us, it risks (e.g.) getting its data centers destroyed by nukes? A misaligned AI would then potentially be taking a big risk if it turned against humans, and if its desires didn't clash too much with ours that risk might not be worth it.
Concerning the persuasion capabilities of AI, I think there are two scenarios:
A) There is a single dominant hive-mind AI whose copies are winning the race, and who are cooperating. I think this is mostly the scenario that you work with.
In this case, I think we can get some lower bounds on how persuasive they will be. There are already human-directed bot nets which try to invade social media like X or telegram. I would estimate that they can persuade at least a significant minority of the US public, perhaps 20%-30% of a claim that is fake but somewhat plausible, simply by repeating it many times. (I am not confident in this number and would love to hear estimates by other people.) This goes for existing campaigns which consist exclusively of online posts, and are directed by perhaps 100-10,000 original accounts. Given that the AI could easily spend more activity than the existing campaigns, we should expect it to reach at least this level of success. Even if it only reaches human level persuasiveness, not super-human level.
It might be that the AI chooses not to do that. Humans have a very weird mechanism for deciding what to do and what not to do. By default, I would assume that the mechanism of the AI will be just as unfathomable. But if the AI decides to go to social media, I think it will dominate large parts of it.
B) There is an ecosystem of unaligned or even competing AIs. Not necessarily different models, it could also be unaligned copies of the same model. I find this more plausible than scenario A. Sure, the copies could in theory start to coordinate, but there is a big gap between "could in theory" and "will do". Again, the decisions of the AIs may be weird.
In that case, I find it even more likely that social media will be dominated by AI content, because making X accounts and telegram posts is really easy, and some AIs will go for it. But it will probably look a lot more the current situation where different groups try to distribute their version of truth. People who are grounded in social media will have a hard time to tell truth from fabrication. But it may not be a single world-view that is winning there.
Well-conceived project and great essay, but your dedication to follow-up and engagement is the most impressive thing. Thank you.
----
> We stand by our specific claim [...] that only 4% of people will name AI in Gallup’s “most important problem” poll in 2027.
Public opinion is hard to shift, and deafness to the worst news is a huge problem.
But I imagine once some threshold is breached, "it's eating jobs, backed by billionaires, and going to kill everybody" is... quite juicy and viral?
Nobody tried spending serious money (say, approaching nine digit dollar sums) on a true public awareness campaign advised by the best, yet. It seems obviously under-leveraged.
It is overdetermined that mass job loss and killer robots will rate higher than climate change on a list of public concerns. I am very happy to bet on that one, maybe not even conditional on some fraction of the acceleration described here happening.
Did you see the r/slatestarcodex post pointing out an issue with the model assumptions? "The AI 2027 Model would predict nearly the same doomsday if our effective compute was about 10^20 times lower than it is today"
https://old.reddit.com/r/slatestarcodex/comments/1k2up73/the_ai_2027_model_would_predict_nearly_the_same/
Author of the linked post here:
They have responded substantively and with active engagement (just not in that specific thread that got double posted due to LW moderation purgatory).
While we obviously have object-level disagreements on things like "is super-exponentiation reasonable" and "are the parameter estimates justified", we are limiting disagreement to mostly "are the functional forms used by the model enhancing fast timelines even if parameter estimates are centered around the same place" which I am leaning towards YES and Eli is (I believe) currently leaning NO, but we both want to do more active investigation.
I will note we are not particularly talking about whether parameter estimates are *justified* disconnected from their functional form, which I believe will remain a crux of disagreement, and I have offered a bet based on disagreed-upon parameter estimates, namely that I think we have not and will not see R&D progress anywhere close to the estimates in either model. Eli has given a reasonable explanation of non-measurability and non-applicability to non-frontier models, so we may try to work out another form.
Great piece. Super useful summary!
If you could turn the tabletop exercise(s) into actual board games, I think a lot of people would find that useful and fun, and it would align with your goal of trying to diffuse these ideas into broad society. Something like the board game Pandemic.
Thanks for a valuable contribution. Who do you think has done a good job of analyzing the related fat tail risks? What does your team consider the probability of major kinetic conflict due to fear of a competing state's AI progress (real or imagined)?