Reading List: Why Superintelligence May Be Less of a Risk Than We Think
Some reading pointers
Some intuitive arguments I find compelling:
Some more formal ones (both presented at the AGI-26 conference):
The Explosion Afraid of Itself (paywalled; trying to find a full-text copy & will link it here)
And one that’s somewhere between formal and intuitive:
Anti-Singularity: Toward Harmony with Machines by Harmless AI
(also published also as Anti-Yudkowsky…)
A great read that goes through a bunch of foundational concepts from top to bottom.
The formal arguments rely on being more precise in what kind of system we’re talking about. Rather than terms like “AGI”, “superintelligence”, “singularity” etc., the formal concepts come from dynamical systems theory, neuroscience, active inference and algorithmic information theory. By being more rigorous in physically defining things, we can tease apart what the terms mean and get more clarity on what terms make sense to use together. This is possible even if the concepts haven’t been fully turned into math yet (“You see, the only difference between poetry and a technical solution is the level of detail”). E.g. from one of the intuitive posts above:
In any case, while it’s been years since I’ve had the thought to write a thesis on why Yudkowsky and others are wrong on the risks of superintelligence (but never made the time!), plenty of others have been writing about this in the meantime. In the last couple years the topic has entered more of the mainstream, and a quick search for “why AI won’t kill us all” yields plenty of results. I haven’t read all these, but will continue perusing such posts and see what other arguments folks have made and whether I can integrate them into my framework…
When someone’s writing about the topic on Psychology Today, you know it’s gone mainstream! :D