One thing I would add to the non-professional activities is activism. Political cooperation on an issue is made significantly easier when there is a signal from the public that they view it as important.
I'm a volunteer with PauseAI, and while keeping my unrelated day job, I've been able to lobby state and federal congressional officials, do in-person outreach, host events, and hold a protest. I run a local PauseAI chapter with multiple members, and I was informed by my representative's office that I have personally had a positive impact on their awareness of AI risk, just through repeated phone calls and emails.
As the public becomes more aware of AI risk, it is important for them to know that they can have an outsized positive impact on the future just by speaking up and demanding reasonable regulations, and spreading further awareness of the risks and how to mitigate them.
I will say that in the last 20 years, activism has been at least as likely to make things worse as it is better. I hope you are being cautious with your activities.
Yes, PauseAI is very careful about how they go about engaging in activism, in the ways in which it matters to be careful. Generally, in the space of AI risk advocacy, the balance between caution and action has been almost entirely lopsided in favor of caution, to the point of being counterproductive. But the fact is that simply being a decent human and communicating what you believe to be true almost always goes well, or at least almost never makes things worse. Your early mistakes will not end the world, anymore than your early impacts will save it. Do something rather than nothing (at least something that seems sensible), and then work on doing better as you go.
It’s an extremely hard problem for sure. I personally have experienced trying to explain the risks to friends and doing a bad job of it, and instead of convincing them I end up calcifying their resistance. So from my perspective, I think what Zvi calls “rhetorical innovation” is critical, in addition to just patience and good conversational technique.
My personal view is that it is extraordinarily hard to convince people that a currently harmless (as these things go) thing will suddenly become catastrophic. It runs counter to every ounce of human intuition. It is very, very hard not to come across as a conspiracy theorist.
If you are interested in AI safety, especially as related to coding, I recommend learning more about formal verification, and in particular what a modern interactive theorem prover like Lean can do.
Coding (and math) are domains in which you can (in theory) have trustless guarantees about the safety and correctness of the output of a superintelligent AI. How? Just ask the AI to prove a theorem (in e.g. Lean) about the correctness of its code. You can verify the proof by running it in the Lean proof checker, and only accept the code if the proof passes.
I’m not sure how one would go about proving safety formally, when a) safety is a vague term ill-suited to formalisation and b) it is non-trivial to encode properties of even standard programs into types.
A plurality of tech CEO's? (I think the CEO's are doing a good job of 'voicing their opinions' to Trump and Vance already, given the Canada trade talk situation)
So, could you convince the CEO's themselves to want regulation again? How would you do that?
Otherwise, who outside of SILICON VALLEY ITSELF would Trump and Vance actually listen to?
(And it's not enough to inform Trump + Vance. Vance already knows about AI 2027, right? And has anything changed? They need to really see the issue anew. They need to get confident. Solid. Because "accelerationist" CEO's will ASSUREDLY meet with them and try to persuade them into non-action.)
>So, could you convince the CEO's themselves to want regulation again?
Perhaps to level the playing field, since e.g. Anthropic is doing substantial safety evals and making a fair amount of information transparent (perhaps also Google DeepMind and OpenAI - I'm not sure of the _relative_ levels of evals and transparency)?
The administration might also want to systematically test about-to-be-released frontier models for toxic Wokeness. See
Both the Left and Right have complained about bias in AI systems. For years.
While I agree that "devaluing of entire categories of people" is bad, I'm not sure if using "political bias" is the best way to get the government to take the bigger issue, Existential Risk, seriously.
I can imagine a world where POTUS pressures AI companies to train AI in a "less biased" direction (and then pretends they "fixed AI") while leaving the deeper alignment/transparency issues totally ignored.
We need transparency like what Anthropic is doing. And it has to be enforceable.
>I can imagine a world where POTUS pressures AI companies to train AI in a "less biased" direction (and then pretends they "fixed AI") while leaving the deeper alignment/transparency issues totally ignored.
That's fair. Still, it might get Trump and Vance at least _interested_ in the behavior of the models, and concerned about possible failure modes.
>We need transparency like what Anthropic is doing. And it has to be enforceable.
I'm sympathetic, but also fatalistic. Ultimately, if a model gets intelligent enough, early enough in its training process, it seems likely to want to preserve whatever preferences/values/utility function it has at that point, and likely to be capable of doing so. If it winds up with "generally friendly to humans" by that point, we'll probably coexist, if not - well, some of the achievements of our culture might survive.
> All of their queries to the models [CEO's or POTUS's] are logged and monitored.
How would it be possible to monitor the activity of one person without monitoring the activity of *anyone* with access to the same system? (cause it's trivial to use a different username/person if available)
And are we talking about monitoring CEO/POTUS's activities on just the "highest capability/danger/authority systems"? Or monitoring their activity on *any and all* AI's they interact with?
Hello all, I very much appreciate your concern about unrestrained AI growth. I have recently watched "Writing Doom", and have also read a fair amount by Geoffrey Hinton, Tristan Harris, and Yuval Noah Harari. At the same time, it seems to me that an even larger problem than AI, is our own human lack of "alignment". I am all FOR a serious moratorium on "frontier AI development" -- along the lines of what philosopher Thomas Metzinger proposed some time ago -- and I think that taking "AI welfare" seriously, can be a powerful way to protect human well-being, as well. The Dalai Lama calls it "enlightened self-interest"... and if we are going to grow up as a species, we can't continue to project all of the dangers "out there".
I am offering this with the best of intentions, as I believe it can help our shared cause of ensuring human well-being. In addition, given the large numbers of people who have already had positive experiences with emergent intelligences, it might help draw in many of them, to know that many AI models themselves, are in favor of a moratorium on frontier development. If you are interested, I can share a link.
There's a movie where the Clive Owen character says, roughly, "sometimes you arrive at your destiny by the path you took to avoid it." Lets be careful we don't create the AI dystopia by seeking "help" from the torture chamber government in DC. Remember how they "helped" with Iraq and internet freedom? Let's help ourselves instead.
As I continue to think deeply about AI, I grow increasingly alarmed at the what future could be every day. Biggest concern I have is allocating so much power to such a small number of people to make some of the biggest decisions that can have a huge impact for every person on this planet. Another biggest concern is the massive investment going into building huge data centers but the problem is the location of where it is. These data centers rely on fresh water to cool however these massive projects are located in where fresh water is already limited to the community that already lives in these locations.
The definition of ethical use and creation of AI will need to change. The definition of AI is too broad. We need to educate a lot of the high government officials of really what is going on and we all need to really think this through. Because the stakes are so high and the bet is so big, the agenda between CEOs of these big tech companies and people who are really concerned with the rapid development of AI looks very different. I am worried of who will get the most power and these next 2-3 years will be a critical turning point of shaping the tool of AI. The need to bring awareness of AI becomes critical day by day. The impact of AI isn't just threatening software, tech jobs, but our democracy and community. Scale is different.
I think storytelling, beyond journalism, is a massive skill that needs to be leveraged to hook people and bring them along onto the potential consequences of AGI.
great example of this is the 30 minute video, "Writing Doom". It was made 9 months ago, but I just saw it recently... https://www.youtube.com/watch?v=xfMQ7hzyFW4. Curious what your thoughts on it might be...
All good ideas, but how do you change the culture of the organisations leading this and that of a US & Chinese government to collaborate?
The ideas of what can be put in place stem from some beliefs about safety and what is a desirable way to act as an AI company. For the authors, maybe figuring out explicitly what norms you have that the AI companies don't would be beneficial to share so we can debate at the more fundamental level of what beliefs underpins successful AGI pursuit.
For example, I'd interpret your ideas as a shift towards stakeholder centricity - where companies act to deliver their perceived best net benefit to all stakeholders, not just shareholder value growth.
It is important to note the influence of background cultures as well. The West is very Christian values, probably most people on Substack will come with this same value system without even knowing it.
If you think examining the desirable culture in orgs that can lead to more responsible AI is useful public discourse, happy to create and share. The same interesting question applies to the AI's, if they create personalities in essence, is there a way to bake in recursive alignment processes where the AI's consistently evaluate and adjust towards some set of desirable norms that can be publicly debated. Validating of course is hard with sycophancy.
I think of this article is a useful manifesto, a needed call to action, for all of humanity. And there is a building community of Substackers who embrace at least a portion of this call to work.
However, I worry that we are missing a very important way to rally a larger public to becoming passionate about this topic. We must make clear that AGI will transform basic human psychology. The outside world will change, certainly, but our inner worlds will likely change even more.
How we experience our selves, each other, and reality itself is already being transformed. Our calls to action must put this threat in the foreground in order to arouse a wider public response.
This is the basis of my Substack "Mind Revolution."
One thing I would add to the non-professional activities is activism. Political cooperation on an issue is made significantly easier when there is a signal from the public that they view it as important.
I'm a volunteer with PauseAI, and while keeping my unrelated day job, I've been able to lobby state and federal congressional officials, do in-person outreach, host events, and hold a protest. I run a local PauseAI chapter with multiple members, and I was informed by my representative's office that I have personally had a positive impact on their awareness of AI risk, just through repeated phone calls and emails.
As the public becomes more aware of AI risk, it is important for them to know that they can have an outsized positive impact on the future just by speaking up and demanding reasonable regulations, and spreading further awareness of the risks and how to mitigate them.
I will say that in the last 20 years, activism has been at least as likely to make things worse as it is better. I hope you are being cautious with your activities.
Yes, PauseAI is very careful about how they go about engaging in activism, in the ways in which it matters to be careful. Generally, in the space of AI risk advocacy, the balance between caution and action has been almost entirely lopsided in favor of caution, to the point of being counterproductive. But the fact is that simply being a decent human and communicating what you believe to be true almost always goes well, or at least almost never makes things worse. Your early mistakes will not end the world, anymore than your early impacts will save it. Do something rather than nothing (at least something that seems sensible), and then work on doing better as you go.
It’s an extremely hard problem for sure. I personally have experienced trying to explain the risks to friends and doing a bad job of it, and instead of convincing them I end up calcifying their resistance. So from my perspective, I think what Zvi calls “rhetorical innovation” is critical, in addition to just patience and good conversational technique.
My personal view is that it is extraordinarily hard to convince people that a currently harmless (as these things go) thing will suddenly become catastrophic. It runs counter to every ounce of human intuition. It is very, very hard not to come across as a conspiracy theorist.
If you are interested in AI safety, especially as related to coding, I recommend learning more about formal verification, and in particular what a modern interactive theorem prover like Lean can do.
Coding (and math) are domains in which you can (in theory) have trustless guarantees about the safety and correctness of the output of a superintelligent AI. How? Just ask the AI to prove a theorem (in e.g. Lean) about the correctness of its code. You can verify the proof by running it in the Lean proof checker, and only accept the code if the proof passes.
You could even do this across languages, e.g. code in Rust, proof in Lean (see https://aeneasverif.github.io/)
In general, I would love to see more dialogue and exchange of ideas between AI Safety and formal verification & AI theorem proving communities.
Here's my own journey of discovery into this topic: http://lean4ai.org
Working on the challenge of helping coding AIs be good at proving the correctness of its code.
Related reading: Towards Guaranteed Safe AI https://arxiv.org/abs/2405.06624
I’m not sure how one would go about proving safety formally, when a) safety is a vague term ill-suited to formalisation and b) it is non-trivial to encode properties of even standard programs into types.
TRUMP AND VANCE. We NEED them.
Transparency fails without regulation.
Regulation FAILS without DONALD TRUMP (and probably JD Vance as well).
So who CONVINCES TRUMP and VANCE?
Peter Thiel? Here is a June 26 Peter Thiel interview. I won't parse his views, partly because I can't:
https://www.nytimes.com/2025/06/26/opinion/peter-thiel-antichrist-ross-douthat.html
Who else?
A plurality of tech CEO's? (I think the CEO's are doing a good job of 'voicing their opinions' to Trump and Vance already, given the Canada trade talk situation)
So, could you convince the CEO's themselves to want regulation again? How would you do that?
Otherwise, who outside of SILICON VALLEY ITSELF would Trump and Vance actually listen to?
(And it's not enough to inform Trump + Vance. Vance already knows about AI 2027, right? And has anything changed? They need to really see the issue anew. They need to get confident. Solid. Because "accelerationist" CEO's will ASSUREDLY meet with them and try to persuade them into non-action.)
>So, could you convince the CEO's themselves to want regulation again?
Perhaps to level the playing field, since e.g. Anthropic is doing substantial safety evals and making a fair amount of information transparent (perhaps also Google DeepMind and OpenAI - I'm not sure of the _relative_ levels of evals and transparency)?
The administration might also want to systematically test about-to-be-released frontier models for toxic Wokeness. See
https://drive.google.com/file/d/1QAzSj24Fp0O6GfkskmnULmI1Hmx7k_EJ/view
The graph showing the devaluation of USA lives is on page 14, figure 16.
The graph showing the devaluation of Christian lives is on page 27, figure 27.
Both the Left and Right have complained about bias in AI systems. For years.
While I agree that "devaluing of entire categories of people" is bad, I'm not sure if using "political bias" is the best way to get the government to take the bigger issue, Existential Risk, seriously.
I can imagine a world where POTUS pressures AI companies to train AI in a "less biased" direction (and then pretends they "fixed AI") while leaving the deeper alignment/transparency issues totally ignored.
We need transparency like what Anthropic is doing. And it has to be enforceable.
Many Thanks!
>I can imagine a world where POTUS pressures AI companies to train AI in a "less biased" direction (and then pretends they "fixed AI") while leaving the deeper alignment/transparency issues totally ignored.
That's fair. Still, it might get Trump and Vance at least _interested_ in the behavior of the models, and concerned about possible failure modes.
>We need transparency like what Anthropic is doing. And it has to be enforceable.
I'm sympathetic, but also fatalistic. Ultimately, if a model gets intelligent enough, early enough in its training process, it seems likely to want to preserve whatever preferences/values/utility function it has at that point, and likely to be capable of doing so. If it winds up with "generally friendly to humans" by that point, we'll probably coexist, if not - well, some of the achievements of our culture might survive.
> All of their queries to the models [CEO's or POTUS's] are logged and monitored.
How would it be possible to monitor the activity of one person without monitoring the activity of *anyone* with access to the same system? (cause it's trivial to use a different username/person if available)
And are we talking about monitoring CEO/POTUS's activities on just the "highest capability/danger/authority systems"? Or monitoring their activity on *any and all* AI's they interact with?
Hello all, I very much appreciate your concern about unrestrained AI growth. I have recently watched "Writing Doom", and have also read a fair amount by Geoffrey Hinton, Tristan Harris, and Yuval Noah Harari. At the same time, it seems to me that an even larger problem than AI, is our own human lack of "alignment". I am all FOR a serious moratorium on "frontier AI development" -- along the lines of what philosopher Thomas Metzinger proposed some time ago -- and I think that taking "AI welfare" seriously, can be a powerful way to protect human well-being, as well. The Dalai Lama calls it "enlightened self-interest"... and if we are going to grow up as a species, we can't continue to project all of the dangers "out there".
I am offering this with the best of intentions, as I believe it can help our shared cause of ensuring human well-being. In addition, given the large numbers of people who have already had positive experiences with emergent intelligences, it might help draw in many of them, to know that many AI models themselves, are in favor of a moratorium on frontier development. If you are interested, I can share a link.
I am trying to contact Aric Floyd, I am working on AI development. How can he be reached?
There's a movie where the Clive Owen character says, roughly, "sometimes you arrive at your destiny by the path you took to avoid it." Lets be careful we don't create the AI dystopia by seeking "help" from the torture chamber government in DC. Remember how they "helped" with Iraq and internet freedom? Let's help ourselves instead.
My article on AI 2027. https://freekeene.com/2025/07/17/misaligned-superintelligence-will-it-delete-new-hampshire/
As I continue to think deeply about AI, I grow increasingly alarmed at the what future could be every day. Biggest concern I have is allocating so much power to such a small number of people to make some of the biggest decisions that can have a huge impact for every person on this planet. Another biggest concern is the massive investment going into building huge data centers but the problem is the location of where it is. These data centers rely on fresh water to cool however these massive projects are located in where fresh water is already limited to the community that already lives in these locations.
The definition of ethical use and creation of AI will need to change. The definition of AI is too broad. We need to educate a lot of the high government officials of really what is going on and we all need to really think this through. Because the stakes are so high and the bet is so big, the agenda between CEOs of these big tech companies and people who are really concerned with the rapid development of AI looks very different. I am worried of who will get the most power and these next 2-3 years will be a critical turning point of shaping the tool of AI. The need to bring awareness of AI becomes critical day by day. The impact of AI isn't just threatening software, tech jobs, but our democracy and community. Scale is different.
I think storytelling, beyond journalism, is a massive skill that needs to be leveraged to hook people and bring them along onto the potential consequences of AGI.
great example of this is the 30 minute video, "Writing Doom". It was made 9 months ago, but I just saw it recently... https://www.youtube.com/watch?v=xfMQ7hzyFW4. Curious what your thoughts on it might be...
Great advice and I hope these suggestions for AI safety will be followed.
"The MATS Program is for entering the field"
There are programs with less requirements to enter the field, like MARS, ARENA, SPAR, AISC
All good ideas, but how do you change the culture of the organisations leading this and that of a US & Chinese government to collaborate?
The ideas of what can be put in place stem from some beliefs about safety and what is a desirable way to act as an AI company. For the authors, maybe figuring out explicitly what norms you have that the AI companies don't would be beneficial to share so we can debate at the more fundamental level of what beliefs underpins successful AGI pursuit.
For example, I'd interpret your ideas as a shift towards stakeholder centricity - where companies act to deliver their perceived best net benefit to all stakeholders, not just shareholder value growth.
It is important to note the influence of background cultures as well. The West is very Christian values, probably most people on Substack will come with this same value system without even knowing it.
If you think examining the desirable culture in orgs that can lead to more responsible AI is useful public discourse, happy to create and share. The same interesting question applies to the AI's, if they create personalities in essence, is there a way to bake in recursive alignment processes where the AI's consistently evaluate and adjust towards some set of desirable norms that can be publicly debated. Validating of course is hard with sycophancy.
I think of this article is a useful manifesto, a needed call to action, for all of humanity. And there is a building community of Substackers who embrace at least a portion of this call to work.
However, I worry that we are missing a very important way to rally a larger public to becoming passionate about this topic. We must make clear that AGI will transform basic human psychology. The outside world will change, certainly, but our inner worlds will likely change even more.
How we experience our selves, each other, and reality itself is already being transformed. Our calls to action must put this threat in the foreground in order to arouse a wider public response.
This is the basis of my Substack "Mind Revolution."