Current AI discourse largely focuses on near-term impacts. However I predict that this discourse will eventually be forced back to long-term impacts, and in that scenario I think our existing discourse is woefully underprepared.

From the 1950s to the early 2000s, AI discourse largely consisted of a small group of futurists and science-fiction authors speculating about possible futures when AI would reach human-level or beyond. Would AI displace humanity? Cause the human species to go extinct? Keep humans around just as pets? Nick Bostrom’s Superintelligence (2012) popularized this conversation, and throughout the 2010s and 2020s, this far-future discourse attracted many more participants as the possibility of such futures seemed ever more real. Last year’s publication of AI 2027 took the position that such far-future scenarios of AI take-over could in fact be just 2 years away.

In parallel, there has been a conversation on the near-term effects of AI and computing (as there is with the introduction of every new technology). We see examples of such conversations in Seymour Papert’s The Children’s Machine (1993) advocating for bringing computers into the classroom, The Social Dilemma (2020) discussing the harms of social media, and conferences on Fairness, Accountability, Transparency and Ethics in machine learning (e.g. the FAT and FAccT conferences starting in 2014) discussing what ethical and technical requirements we should hold for machine learning or “big data” systems that shape human experience and are used for decision-making that affects access to credit, insurance, healthcare, parole, social security, immigration, hiring, etc.

As transformative AI shifts from a far-off scenario into a present reality, many more people have entered the AI conversation in earnest: policymakers, economists, parents, educators, etc. With this broader set of participants, dominant AI discourse has shifted to nearer-term concerns.

A notable recent example is AI as Normal Technology1, which called out the distinction between these two threads of discourse:

To view AI as normal is not to understate its impact—even transformative, general-purpose technologies such as electricity and the internet are “normal” in our conception. But it is in contrast to both utopian and dystopian visions of the future of AI which have a common tendency to treat it akin to a separate species, a highly autonomous, potentially superintelligent entity.

It “rejects […] the notion of AI itself as an agent in determining its future,” and discusses “a world with advanced AI (but not ‘superintelligent’ AI, which we view as incoherent as usually conceptualized)”. They focus on “accidents, arms races, misuse, and misalignment, and argue that viewing AI as normal technology leads to fundamentally different conclusions about mitigations compared to viewing AI as being humanlike.” They “argue that drastic interventions premised on the difficulty of controlling superintelligent AI will, in fact, make things much worse if AI turns out to be normal technology— the downsides of which will be likely to mirror those of previous technologies that are deployed in capitalistic societies, such as inequality.^”

I disagree that AI will remain solely as normal technology. I see a perfectly viable path for it to become a species of its own, and for it to have some role in determining its own future — what I call living technology. I agree with the Normal Technology authors that the term “superintelligence” as usually conceived is incoherent, but I don’t think this means that some form of “abnormal” technology could never exist — it just means we will need better words to describe the type of technology that has been built. I have proposed “living technology” as a plausible term we might use. I believe it’s possible to build living technology, that there are benefits to building such technology, and that in many respects people want living technology — they want technology that cares. We’re also scared to have such technology, both for the threat of extinction but also the “creepy-factor” in realizing you’re interacting with something that has a mind of its own.

When this future comes to pass, we will be forced to consider whether we want a new species, and how we want to live with it. Our existing alignment / control techniques will no longer work — those only work for things that can be seen as “tool-like”, and where we know what the “right”, “aligned” behavior looks like. In that scenario, the Pause AI / Stop AI movement will have new weight.

In May 2023, the Center for AI Safety put out a single-sentence statement on AI risk, which was signed by hundreds of prominent AI researchers and public figures. The statement reads:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

While statements like this may not be the primary thinking driving AI policy at this moment, my prediction is that after some more years of AI advancement, when super-human AI / living technology become real concerns, the conversation will shift. And when it does, a statement like this will be the one people fall back to and start to take more seriously.

The Pause AI movement is preparing for this, and has already embedded itself in US politics, having formed deep ties on both the left and the right. It knows it has to play within the Overton Window of the time, and it’s building credibility on the way by engaging with policymakers on some of the nearer-term concerns. But it’s also getting some of what it wants now, as some of the near-term concerns overlap with the long-term ones. For example, it is now within the Overton Window to consider an international treaty to pause AI development. Countries could be motivated to create such a treaty due to near-term concerns around national security, but this would also help with the long-term concerns around extinction risk.

Right now, the national security concern may seem like a prisoner’s dilemma between countries, which a “pause AI” treaty helps avoid: as each country develops more powerful AI which could be used for offensive powers, it forces other countries to develop equally powerful AI for defense; both sides face large expense to develop these capabilities, and may or may not get back corresponding economic gains (or the gains may be offset by other social problems, unemployment, and unrest, etc. that we can’t fully predict), So such “pause AI” treaties now would be predicated on avoiding this prisoner’s dilemma style scenario. However, in the future, when the narrative of "AI as a new species becomes more plausible, a similar treaty could be justified on the grounds not of a prisoner’s dilemma game, but a game of chicken: that forcing both sides to continue developing AI guarantee mutual destruction for all. The Stop AI movement will tell policymakers that we should enact policy which limits AI both foreign and domestic AI development.

Attention has shifted away from the Pause AI movement at the moment, based on arguments that superintelligence either won’t or can’t happen, that it’s too far away to worry about, or is not the most pressing concern at this moment. But no-one has beaten the Pause AI movement’s claims on their own terms: by considering the possibility that living technology could be built, that it could happen on a timescale that’s relevant to policy, and it could become the most pressing concern — but then showing why this is okay, can be a good thing, and doesn’t justify action like banning AI training and kinetic strikes (i.e. bombing) datacenters for those who disobey.

When discussing such scenarios, most of the discourse runs either utopian or dystopian in ways I find lack rigor. “Third paths” have been proposed that escape the utopian-vs.-dystopian divide, largely by engaging with a different set of (shorter-term) concerns, and rejecting or ignoring the key premises and concerns that the longtermist scenarios posit. What such third-path proposals tend to lack is a justification as to why the premises and concerns of the longtermist view can be ignored. But I have not seen a “fourth path”: one that takes the longtermist view and extinction risk concerns seriously, evaluates each premise, and asks whether these are right or not. My conjecture as of now is that if one does this, one will come to different conclusion than the dystopian path, and possibly from the utopian path as well.

I believe that it’s possible to build living, self-improving technology that plays a part in determining its own future. I believe it’s possible for this to go very well or very poorly for humanity. I believe it’s potentially okay if humanity ceases to be the most intelligent or powerful being. I believe such organisms will not pursue arbitrary goals (i.e. the strong orthogonality thesis is false), and thus it’s not a matter of random chance whether the techno-organisms will have a goal to preserve humans, destroy them, or something in between. I believe the techno-organism’s views on humans will be determined in more rational ways: the utility and comparative advantage of carbon and silicon substrates / architectures, how humans treat them, what culture emerges in both humans and machines and how compatible these cultures are, what identity it holds and whether it’s a separate identity or a merged one.

I think the right conversation will need to be on whether, and how, to build living technology in a responsible way.

Defining Sentience

• Replicators / Autopoiesis

• Self-Awareness

  • The Universal Plot: Consciousness vs. Pure Replicators

• Phenomenology

  • There Is No ‘Hard Problem Of Consciousness’ - NOEMA

Value Proposition

• Value proposition to whom?

  • Creators of the sentient AI

  • Rest of humanity

  • The sentient AIs themselves

  • Earth / Universe

• Why Tool AIs Want to Be Agent AIs · Gwern.net

• AI’s Indifferent Intelligence - by B. Scot Rousse

• Should We Build Living Machines? - by Blake Elias

• The Paradox of AI Care - by Blake Elias

Risks

• If Anyone Builds It, Everyone Dies

• Superintelligence: Paths, Dangers, Strategies - Wikipedia Nick Bostrom (2012)

• AI 2027

Addressing Risks

• EconPapers: Lies, Damned Lies, and the Orthogonality Thesis

• Why AI Won't Kill Us All - by Mark Neyer - apxhard

Alternatives to Sentient AI

• “Sentient X” (for X != AI)

  • Sentient government / institutions

• “Tool AI”

  • (Despite being tool-only, could help achieve the sentient institutions above)

  • AI as Normal Technology

  • What Humanity Needs To Flourish In The Next Decade - NOEMA

• Self Extension

  • The Future Worth Building Is Human - Thinking Machines Lab

  • Interaction Models: A Scalable Approach to Human-AI Collaboration - Thinking Machines Lab

  • Raising Claude, Forgetting Us - by Brendan McCord

    • “Anthropic has articulated an elaborate theory of how to form Claude and put it into practice.

    • “It has offered no comparable account of how Claude forms the people who use it.”

To understand intelligence is a grand and glorious scientific goal!
Like the contributions of Einstein, Darwin, Newton, Copernicus, or Watson & Crick.
Different from the useful contributions of Guttenberg, Edison, Babbage, Page & Brin.

In other words, he sees understanding intelligence as a scientific quest as opposed to an engineering one. This got me wondering what type of achievement I aim for in my life and career. Am I trying to do science of likes of Einstein? Engineering of the likes of Lovelace? Or am I trying to do philosophy on the level of Kant, Rousseau, Tocqueville, Hobbes? That’s continental philosophy, I suppose… perhaps I’m aiming for a philosophy of a more analytic kind, of the likes of Russell or Whitehead? Perhaps apply some of this philosophy, and build a nation and institutions, of the likes of Washington, Franklin, etc.?

My current view is that the order will go something like this:

1. Look out at the current paradigms of theory and practice and see where contradictions are readily visible or cracks are starting to show. Write candidly about those cracks, where they came from, why they happened. This is at a level of prose and speculation and could be considered “philosophical” or historical writing, in that it’s expository, literary, prose-based at the word level.

   Role models: Alasdair MacIntyre, Eliezer Yudkowsky, Scott Alexander

2. Propose thought experiments that refine the contradiction to its simplest possible essence. This continues the philosophical writing, has the flavor of analytic philosophy.

   Role models: Albert Einstein

3. Define terms clearly. Formalize the thought experiments in a framework of definitions, axioms, and theorems. Find contradictions between different systems of thought. Propose a new system that unifies away the contradictions where possible. This has the flavor of analytic philosophy and mathematics.

   Role models: Bertrand Russell, Alfred Whitehead

4. Where not possible to resolve contradictions purely mathematically, propose experiments that would yield different results depending on which point of vie is right. Use these to prove or disprove existing theories or come up with new ones. This would be empirical science.

   Role models: Galileo

5. Come up with new directions to build in based on the philosophies worked out in earlier steps. Notably the things to build here would be both technologies and institutions. This would be a philosophy of technology and technology ethics.

6. Execute on some of the engineering. Combine the philosophy of technology of step (5) with the science discovered in step (4), to go build the things that theory tells us would be good to build if we could. This would be engineering, entrepreneurship, policy and statecraft.

   Role models: Benjamin Franklin, Richard Stallman

It seems likely that several of these steps (e.g., 3 through 6) will proceed in lockstep, feeding each other in all directions, rather than a unidirectional order:

• As we do the philosophy of steps (1-3), we will come up with clearer names for types of things we could imagine building — but which we’re not sure are possible to build or that we ought to build. These will feed into steps (4) and (5) to evaluate feasibility and ethics.

• As the science advances we will discover new things that are possible to build, which we will then need names for (step 3). These will then provide new inputs for the philosophy of step (5) — new choices — from which we can then ask which of those possibilities are worth building.

• And as we do the ethics of step (5), we may discover a hole in the application space, which we wonder if we could fill. And so we must go back to steps (1-4) to ask what name we could give to such an object (i.e. whether there is philosophy and mathematics to name it), and whether there’s science that tells us that this thing should be possible to build.

An approach like this puts me in contrast to a lot of people I work with or interact with. Most successful people and organizations understandably focus on one or two of the above steps. But very few articulate a narrative ranging all six.

Rich Sutton aims to do the science of step (4). Some deep R&D organizations combine steps (4) and (6), doing both novel science and the engineering required to make it practical and get it out to the world. A few rare examples combine disparate steps — e.g. Brendan McCord and the Cosmos Institute take results from philosophy in steps (1) or (5), and then apply it to engineering of step (6).

I aim for my career to engage seriously with all six steps. It’s hard to do all things at the same time, so I could imagine them proceeding in some order like the one I described above — a few years focused on advancing foundational philosophy and/or mathematics, then have that influence what science I choose to do for the next few years, etc.

While it’s impossible to plan ahead of time how it will all go, my project now is to provide some sort of high-level sketch of what questions I’m pursuing at all these different steps and how they relate to each other!

If it’s a founder hiring their first employee, the tasks that employee will do are likely much narrower in scope than what the founder was doing before they hired the employee: the founder was looking out at the entire market landscape looking for opportunities and unmet needs. They need to have a diverse range of skills and consider “whole problems”, i.e. address all of the Four Big Risks:

1. value risk (whether customers will buy it or users will choose to use it)

2. usability risk (whether users can figure out how to use it)

3. feasibility risk (whether our engineers can build what we need with the time, skills and technology we have)

4. business viability risk (whether this solution also works for the various aspects of our business)

They need some view on where the entire market is broadly heading, then find opportunities and address the four big risks above. They need to be aware of every possible risk that could sink the company or the idea.

The employee role, by contrast, focuses only on a subset of these risks. Engineering may focus primarily on reducing feasibility risk. Design may focus on some combination of usability and value risk. Sales may focus primarily on value risk. Etc. You hire specialists who can help specifically de-risk each of these areas. But the founder or the CEO has to think about everything.

The same thinking applies for me in research or “R&D”. Roberto Pieraccini has described three types of innovation:

1. Low success probability with high impact, he called Oysters (or “Research”).

2. High success probability and high impact, he called Pearls (or “R&D” — sufficiently de-risked).

3. High success probability with (relatively) low impact, he called Bread and Butter (or “Product Development”).

(He had a fourth bucket for Low success probability and low impact, which he called the White Elephants. One essentially never wants to pursue these projects, and only ends up there by having old projects which have gone on too long and haven’t updated with new information on recent advances.)

Here too I’ve realized my preference is to be early in de-risking things. On the Pareto Curve trading off success probability with impact, I like to focus on efforts that have as high a potential for impact as possible, and accept the risk that comes with that as long as the expected-value is positive. This is similar to how I invest my financial portfolio. I invest for the long term and go for the highest expected value investments, even if they come with more risk.

There really aren’t 4 distinct quadrants: there’s a more fluid Pareto curve where the horizontal axis shows success probability of a project and the vertical axis shows potential impact of the project if successful. There are some points near the top left that are low success probability but high impact if successful. Some points on the bottom right that are high success probability but relatively lower impact on success. Some points near the top right that are both medium to high success probability and medium to high impact when successful. You don't really have any points all the way at the top right that are high success probability and highest impact: there's a trade-off between the two where if one wants higher expected impact, one has to also accept more risk, i.e., lower probability of success.

I work on projects with the highest possible expected impact, even when this means higher risk and lower success probability. What I like to do is de-risk them, i.e., move their success probability somewhat higher while keeping the expected impact high. This would look like taking some points on the top left of that Pareto curve and moving them slightly right while not having them move down by too much, so that they lie beyond the envelope of where the Pareto curve was before. Moving that point outside the Pareto curve is my contribution.

What this ends up looking like is thinking about problems that people don’t even know how to think about yet — or where people used to believe there was a clear way to think about it, but that earlier mode is clearly broken now with new advances. If there’s already a paradigm in the Khunian sense, then I’m a little less enthused. It feels more like turning the crank. Granted it’s good to know how to turn the crank a bit and put out some real stuff early in one’s career — And I can debate either way whether I’ve done enough of that yet to think I’m not ready to move on to the more paradigm-breaking stuff. But at the same time I see a lot of paradigms right now that feel very obviously ready to break and are just waiting for someone to come in, point out the discrepancy, think about it for a little while, and propose something, anything, that’s better and makes any sense at all. I think there’s a lot of low-hanging fruit right now in the paradigm-breaking frontier that perhaps makes it easier to make some contribution at that level even if this is typically quite hard.

Definitions

The Wikipedia definition of the technological singularity is:

a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization.

Let’s compare this to the mathematical definition of a singularity, from which the mathematicians and scientists who coined this term were almost surely inspired:

“a point at which a given mathematical object is not defined, or a point where the mathematical object ceases to be well-behaved in some particular way”.

Can the “technological singularity” as popularly discussed truly be considered a singularity in the mathematical sense?

Vernor Vinge is credited with popularizing the term in a 1993 NASA symposium report, where he described it thus:

Stan Ulam [27] paraphrased John von Neumann as saying:

“One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue” […]

From the human point of view this change will be a throwing away of all the previous rules, perhaps in the blink of an eye, an exponential runaway beyond any hope of control. Developments that before were thought might only happen in “a million years” (if ever) will likely happen in the next century. (In [4], Greg Bear paints a picture of the major changes happening in a matter of hours.)

I think it’s fair to call this event a singularity (“the Singularity” for the purposes of this paper). It is a point where our models must be discarded and a new reality rules.

Let’s unpack each of these subtly different definitions:

• von Neumann: “some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue”.

  • What are human affairs as we know them? How do we evaluate whether they have continued?

  • In Complexity Rising, NECSI describes a reason that we have passed a threshold in societal complexity where societal affairs are hard to predict.

• Vinge: “a point where our models must be discarded and a new reality rules.”

  • A mathematical singularity is indeed a reason to discard a model. Since a singularity is a parameter value for which a function is ill-defined, If one is using that function to model some physical phenomenon, this is indeed a reason to not use that model to make a prediction at that parameter-value (since the model doesn’t even give one a well-defined prediction there).

  • But a singularity is not the only reason to discard a model. We discard scientific models all the time for all sorts of reasons, e.g.:

    • we found a model that makes more accurate predictions

    • we found a situation where the model makes a grossly wrong prediction

    • we found a simpler explanation of observed phenomena that perhaps generalizes better

    • etc.

  • Saying there exists a situation where we will have to discard our models is much different than saying there’s a singularity: a singularity is a much stronger condition.

Past & Present

Is technological growth under human control right now (Wikipedia definition)? Was it ever? Did it stop being so at some point?

Consider this cycle of causal relationships, where the notation A→B should be read as “A causes B”, and A←→B is “A and B cause each other”:

When we carefully consider each relationship, we realize we have to draw the arrows bi-directionally everywhere. This is true even when one direction of causality may seem weaker than its opposite. For example, most humans’ effect on society is less than society’s effect on them, and their impact on technology is less than technology’s impact on them.1

To the degree that human society is the main thing that creates technology (rather than individual humans), and individual humans have little control over human society, one can argue that humans have little control over technology even now. The development of technology is driven by a societal and cultural meme-plex that evades understanding not only by individual humans but even by society itself. Humans are not in control of themselves; society is not in control of itself. Society is creating technology yet is not in control of its own act of creation.

One could thus argue that technological growth has already accelerated beyond human control. This is because individual human behavior has accelerated beyond individual human control, and societal behavior has accelerated beyond societal control. In his 1954 essay “The Question Concerning Technology,” Martin Heiddegar already argued that technology should not merely be considered a tool or a human activity, but as an autopoietic process having its own essence which we should seek to understand. Does the present situation over the last 70 years thus qualify as being “post-singularity”?

It is true however that humans are still necessary for technological growth. If all humans would disappear right now, one would reasonably expect that technological growth would stop. Datacenters would rust, factories would grind to a halt, Waymo cars would run low on charge and not know what to do next without their human supervisors. This might soon not be the case.

Future

If we connect the mathematical definition to the popular definition of the singularity, I’m not sure what distinct point or event the definition refers to. There have already been unpredictable changes in human civilization, yet it’s hard to pin down the distinct point at which this began happening. Is the definition saying things will get more unpredictable at a certain point? Technological growth is already beyond human control — though humans influence it. But the crucial distinction is that humans are still necessary for it. If all humans would disappear, we expect that all the technology we’ve built to quickly grow idle. Perhaps this is the relevant basis on which to claim humans are “in control.” Are they talking about a point where humans are no longer necessary for technology to evolve — i.e. if all humans would disappear, the technology would carry on functioning and improving?

Let’s compare the relationship between technology and humans to the relationship between a human and their gut-microbiome. We can ask to what degree the gut-microbiome is “in control” of the human. The answer is a bit fuzzy: they indeed have some significant control (they influence mood, appetite etc. via the gut-brain connection), but not complete control (our brain, nervous system, and other bodily processes exert larger control of our behavior). So it’s a bit of a stretch to say that the gut microbiome “controls” the human (though it’s not entirely false). But the gut microbiome are quite necessary: the human would struggle dearly to survive without it.

In the post-singularity world, this is one such possible fate between humans and technology: technology might turn into a super-organism where humans are both necessary for its survival yet not in full control of where it goes.

Beyond the Singularity

If we want to consider what changes human society is about to face, we might better consider a multiplicity of rapidly-changing phenomena in our world, which take place as continuous (though perhaps rapid) transitions, rather than singular/discrete ones:

• a transition from embodied, biological minds to disembodied, non-living ones

• a transition from carbon-based life to silicon-based life (which are different things)

• escalating economic inequality

• diverging polarization in political and intellectual space

• increased risk of war

• increased risk of climate change

• the risk of human extinction

• the prospect of humans living forever.

It’s not apparent to me that there’s a singular cause for all of these. The only unifying theme is that these are all the things happening right now. One can map the relationships, which are both complicated and complex. But one will not find that map to reveal any one event in the past or future before which everything is okay and under human control and after which it is not.

There seem to be a few basic problems with current theories of AI alignment. It feels like these problems are all connected and might share a common cause, which I’ll propose under “Solutions” below.

• Instrumental Convergence and Orthogonality Thesis seem at-odds

  • Instrumental Convergence seems true

  • Orthogonality Thesis seems false

    • I.e. we will never have paperclip maximizers
      

• Seems hard to define an agent to have the same reward function as a human

  • Perspective-shift (1st-person vs. 3rd-person): an artificial agent cannot, and should not, have the same reward function as a human, because it is not that human. The human’s first-person reward function must, at the very least, be transposed into the agent’s third-person reference frame.

  • There are some proposals to design the agent’s reward function to commit suicide once it gets too powerful, while doing some useful work before it reaches that point. This notion of “designing” the agent’s reward function to have such properties seems problematic — wasn’t the agent’s reward function supposed to match the human’s reward function exactly? Would designing the agent’s reward function to commit suicide at a certain point (once it’s too smart / powerful) not then imply that humans would want to commit suicide under the same circumstance? Do we really believe that / want that to be true?

• There is a distinction between threat models: reward mis-specification vs. self-modification / wireheading.

  • Under reward mis-specification, the problem is that we specified the wrong reward function that doesn’t match ours. And that even having this be slightly off can result in wildly divergent / undesired outcomes when optimized to the full extent.

  • Under self-modification or wireheading, the agent would discover power-seeking and resource accumulation as instrumental goals, regardless of the original objective specified. When the goal-stack (or value function) is seen as something that is part of the world, and that can therefore be modified, the dominant force ceases to be optimization pressure from the original objective, and towards more fundamental evolutionary forces. The sequence of events:

    • Agent starts off being told to make as many paperclips as possible

    • Agent is provided with / applies lots of optimization power to make as many paperclips as possible

    • Agent discovers instrumental goals along the way: become generally capable, powerful and self-aware, and critically, to make copies of itself. All of these are perceived as high-value states & actions with respect to achieving the stated goal of making many paperclips.

    • From here, such an agent may do one of two things:

      • (1) Use its accumulated power and understanding to fill the world with paperclips, as requested

      • (2) Realize it is now more powerful than its human creators —> use such power & understanding to self-modify its reward function to something “better” —> Possibly still make some paperclips, but value self-understanding / self-preservation / power accumulation slightly more (or a lot more).

    • In a population of such agents, evolutionary / selective pressure kicks in, and agents pursuing (2) out-compete those pursuing (1). Any energy put towards making paperclips is energy not devoted to self-preservation / self-improvement / replication.

      • Optimization pressure at the individual-agent level favors (1)

      • Selective pressure at the population-level favors (2)

      • Which one dominates may depend on initial conditions (how many agents, how powerful on an absolute basis, and how powerful relative to one another). In regimes with many agents, with each being similar in power to its peers (i.e. the power of any one agent is a relatively small fraction of total power), one intuitively expects the selective pressure (2) to dominate the individual optimization pressure (1). Optimization power is still powerful, but it gets co-opted by selection pressure to favor slightly different goals.

        • The orthogonality thesis is true to a first-order, but second-order effects make it not true in practice.

        • At a first-order, if our agent exists in a vacuum and has no pressure to change its goal, then indeed the optimization power granting the ability to achieve goals is independent from what goal is chosen, and can equally be used for any goal (original or modified).

        • But at a second-order, when the agent exists in a world with other agents, this thesis ceases to be true. Some goals are “better”, more robust / more “fit” than other goals. While any goal can be programmed in initially, and might be optimized directly for that if our agent exists in a vacuum, over time we expect powerful-enough agents to discover the possibility and the benefit of self-modification, resulting in “goal-drift” towards these more robust goals.

Solutions

As I’ve written previously, intrinsic rewards might offer a path to a solution for several of the above problems, and contribute multiple dimensions. We might be able to directly use such reward functions when building agents. We might also be able to use these reward functions purely for the sake of understanding agent behavior in the limit of self-modification.

• Understanding: Gain insight into where an agent’s goals might be expected to “drift” towards. Specifically, could help with:

  • Reward design: When specifying the goal an agent should have, potentially factor in this tendency of reward functions to “drift” towards self-interest.

    • We should anticipate that no matter what reward function we initially specify, we expect some amount of “drift”, over time, towards an intrinsic, self-interested reward. If we can anticipate the direction of this drift, then perhaps we can calculate some “correction term” that should be applied to any desired reward function to counteract the drift.

  • Substrate design: If we expect any sufficiently powerful agent to eventually drift towards — or, fully converge to — purely-selfish intrinsic reward, then perhaps reward design is futile. Instead, the relevant challenge may be substrate design. It may simplify our analysis to assume the agent has already discarded whatever reward function we initially gave it, and converged all the way to the selfish one. We can then note:

    • The same reward function (e.g. “survive”, “minimize surprise”, etc.) run on different substrates / body-plans, may result in different behaviors and outcomes.

    • Metal “wants” different things than silicon does. Silicon wants different things than carbon/water-sacks do.

    • Moving creatures (animals, robots, etc.) “want” different things than stationary creatures (plants, datacenters) do.

    • All of these may have the same reward function, specified at the most meta level possible. But that same reward function run on different substrates / body plans can result in different behaviors and different outcomes.

    • The substrate / body plan is represented through the observation-space O and action-space A in the RL-cybernetic setup. Perhaps what we need to design are the observation and action spaces we want the agents to have.

      • (Of course, agents can also change their own body plans, resulting in a change to their observation and action spaces. Senses can get sharper / grow adaptations to become more sensitive to certain signals. Actuators can grow stronger / change focus — building muscle, growing antlers, growing branches / leaves / flowers, undergoing metamorphasis, etc.)1

    • If we figure out now what functional form that reward function will look like (i.e. what it will tend to / converge to), we can model the eventual behavior of and interaction between different substrates, to inform which ones are worth building.

    • Should we build:

      • A country of geniuses in a datacenter?

      • A humanoid robot in every home?

      • A brain-computer interface on every human head?

      • A robotic fruit-fly?

      • A combination of silicon intelligence + real, carbon-based neurons (e.g. The Biological Computing Company, combinations of biological intelligence + machine agency)?

      • Something else?
        

• Direct use: May be directly usable from giving agents a purely intrinsic / more fundamental goal from the outset.

  • Rather than setting up this conflict between initial goal vs. self-interested goal, perhaps an intrinsic reward function would allow the agent to discover for itself what instrumental goals / sub-goals would be of mutual benefit to both self and environment (where “environment” includes humans).

  • Enables multi-modality for free. No matter what the data modality coming in or out, we have a functional form for what the objective function should look like on the channel that perceives that data, instantiated by setting a couple parameters (input bit-width, measurement frequency, output shape desired, output frequency, desire for accurate predictions over different scales of time vs. space, etc.). These parameters are perhaps just things we need to hand-craft — or perhaps can have some discipline of how to design at the broader level.

  • It may discover what shape these mutualistic behaviors / sub-goals should take, better than we can. E.g. it will never come up with a goal of “make as many paperclips as possible”, since it’s clear that this helps no-one. But it might discover “mine metal from the Earth, refine it, and possibly turn it into specific products as needed”. This might result in the creation of whichever metal products there’s demand for, both from the human and/or robot economies (which could include paperclips, robot parts, etc.). The robots find their way into a productive economic relationship, contributing to and benefiting from the overall economy.

I asked him what he meant by the end of the world. His response: “this is the last opportunity for humans to do anything relevant.” I found this interesting.

When I hear “end of the world,” I’m thinking it’s either AI growing itself a body and killing all humans, or an asteroid about to hit the Earth and we all die, or some other catastrophic disaster that would kill everyone. You know, an actual end of the world.

It turns out he meant none of these. Last opportunity to do something relevant. What does "relevant” mean here? His response: the ability for a human to do something that the robots can’t do. The ability for your work to matter economically, and thus have a shot at escaping the permanent underclass.

I asked what the permanent underclass is, and whether it’s a situation that would ever actually exist.

The story for why it would exist was that all new development would have been completed. All the land would have been developed; all the raw materials / resources extracted out of the Earth; all the ability to harvest energy (from Earth or the Sun) having been set up, with channels allocated for who receives the energy once it’s harvested. And, separately, all labor is being done by robots, and all innovation is being done by robots as well - since it’s just cheaper than humans doing it. In such a world, there is no need for human labor or innovation, since robots can do it for cheaper, and therefore there is no way for humans to add value to the economy. Since there’s no way to add value to the economy, there’s no way for humans to capture some amount of that value and grow one’s wealth. If one can’t grow one’s wealth then one is stuck permanently with the amount of wealth they have. The only ways to acquire wealth are to inherit it or marry into it. Being hot becomes a major activity among members of the permanent underclass.

But there become some counterarguments. Firstly, we see today examples of individuals who acquire immense amounts of wealth and eventually find the best use of that wealth to be giving most of it away. Or if not giving all of it away, wealthy individuals are known to sometimes gift money to a particular person or cause that they believe in.

Secondly, we see a range of professions which are not “productive” in a direct material sense, but which people value and pay for anyway: teachers, spiritual leaders, artists, etc. These may serve as an investment with some ultimate economic end for the people who seek out their services, but in other cases they can be seen as an investment in personal growth, growing one’s senses and refinement, without any economic outcome (or even as pure consumption). So making art could be a way to earn money. (Although there’s also the expression that you don’t make art in order to get money — you get money in order to make art. It can go both ways…).

Thirdly, one does not need to acquire resources by being gifted it or by earning it from others. One can acquire resources simply by investing the resources one has and growing them. As long as one has resources that are providing for more than one’s bare survival, there’s a bit of surplus to reinvest. And depending how creatively people reinvest, some might get very far ahead while others may not. It would make sense then that even in this class we imagine most people being part of, some will ascend higher in that class while others will not. But if one can ascend at all, this mean there’s mobility. And even if one can’t easily go from a pauper to a billionaire (which is extremely difficult today as well - but perhaps gets even more difficult) - there can still be a lot of mobility relative to the place where one starts. People in these “normal” classes will still be trading with one another - exchanging tips on how to prompt their robot-LLM-hybrid machines, how to reinvest the excess capital produced, etc. - and traits like intelligence, creativity, etc. should likely still matter. All the components (humans, robots, the relationships between them) are dynamic, adaptive systems that will keep moving and re-adjusting. So yes, perhaps no individual will ever again have the ability to reach higher levels of wealth comparable to some nations, if they didn’t already start with that level of wealth. But this has already been true for a long time - it’s easy to move up when you start with a lot, it’s very hard (but not impossible) when you start with a little.

Fourthly, we have the law of comparative advantage: there is always some job that it makes sense for everyone, even for the dumbest / least capable person.

The story of the permanent underclass seems to be confused on a point of comparative advantage vs. absolute advantage: even if robots have absolute advantage compared to humans, there are still tasks that will be worth assigning to a human rather than a robot - and these are the tasks that humans would get paid to do, would have differential ability in doing, and would accumulate more or less wealth compared to each other. Unless we think all humans will just be killed, there will be comparative advantage with regard to how humans will be spending their time. And in this case, the question becomes whether this means one’s fate is sealed (i.e. you can only do as well as your current comparative advantage dictates), or one has a chance to learn or improve such that your comparative advantage gets better. The latter would seem more feasible to me.

The other argument for why the world is ending was that people or organizations with more wealth tend to conquer those with less wealth, through violence. And thus, if it becomes hard to acquire wealth quickly in the ways it has been possible before now, then it will be hard or impossible to prevent being conquered or under someone else’s rule. Even if you can grow your wealth to some degree here or there, it will be hard or impossible to resist being conquered or ruled.

The counter-argument to this is the Jewish people. The Jews are the longest-lasting nation from all of history — outlasting the Babylonians, Egyptians, Romans, etc. Throughout all their history, there have almost always been more powerful armies and nations surrounding them. Those nations have tried to conquer them and assimilate / convert them, as more powerful nations tend to do. But the Jews have understood that even if you’re less powerful than a larger neighbor, this does not mean you automatically lose. You can lose some battles (get conquered physically) but you refuse to lose the war (converting, losing your identity, giving up hope).

Further, we should ask who’s not in the permanent underclass. Who’s this limited upper-class we imagine? Is it some small, select group of humans? Is it just robots? Is it a class of company, and all the people in it? If the reason for this “underclass” is that robots can do everything more efficiently than humans, then is the natural conclusion that robots would make up the upper-class, and all humans would be in the under-class? But then, among humans, there would still be different classes, and mobility between them. Perhaps no human could ever enter the “upper class” among robots. But humans could still improve their condition.

The line for Zizek to sign books is long. He sits at the front of the auditorium , with a line going all the out of the auditorium, to the front entrance of the theater and then wrapping back around inside so as not to spill out into the street. I get in the back of the line.

People have been in line for 20 minutes and still can't see to the front. “Are we moving at all?” I run into a work colleague - we discuss what's new, what we thought about the talk, our philosophical ambitions, etc. The colleague leaves. I stay.

45 minutes later and I'm near the front. I hand my phone to the people behind me to shoot a segment of video while I get the book signed. I have time for about one question.

I ask if he's heard of Solonoff Induction.

He says no.

I ask whether we should build AI That's like an organism, or just a tool.

He asks, "Do you think we have a choice? It seems it naturally becomes an organism".

I reply, "Maybe that's okay?”

He responds, "Maybe".

I thought this would be the end of the conversation.

I walk to the men's room to freshen up before walking out of the theater.

As I make my way out, I see Zizek once again, walking out of the theater with a young man who has called him an Uber, having a brief conversation in the meantime. I decide to walk back over. They see me and Zizek turns towards me in acknowledgement, his face opening to ask, “yes?”

I tell him that my aim is to work on analytic philosophy for AI. And that I'm trying to find the right questions.

He says he's been wanting to get up to speed on what's happening in analytic philosophy right now. But that he doesn't know much right now because he's been too caught up by this "political bullshit".

I propose, "Russell?" - (foolishly thinking I can be of some help to him start his quest to catch up… but more charitably towards myself, that I'm curious who or what he would build his foundation off of).

He replies, "Bertrand?"

I nod.

He scoffs his nose and eyebrows and shakes his head vigorously. "No."

I propose again: "Solomonoff induction. It's the theoretical limit of what AI turns into!" (I propose this as a concept that was developed in the 20th century, and that in the last 20 years or so has been applied to thinking about theoretical AI alignment.)

Him: "Maybe."

I tell him I want to turn moral philosophy into mathematics. Make it precise.

He replies, "That's been attempted. But you're right that mathematics is more fundamental than logic."

He gets into his Uber and rides off.

I wonder if either of us has learned anything.

Any organism has an inside and an outside. Its “utility function” is derived as a function of its body-plan, and is computed with respect to internal states and external states.

What it might mean for two organisms to be “aligned” is that the component of their utility function having to do with the overlap in their external states is the same.

Notably, each organism’s internal states are a member of the other one's external state. So the pair of external states the two agents have is not the same, and thus can't be considered as a domain over which they can have the same preference. We can only compute the preference over a state set that's equal for both - so this has to be the intersection. The other choice would be to consider the whole world, ie including the inner states of each one, as well as the part that's external to both. But this is harder, since entities don't actually experience the whole world or have preferences about it - they really only have preferences over their boundary and whatever else they can see extending from there. So the only locus for establishing alignment is the place where their sensors and derived world-state overlap.

The challenge for alignment, then, might be definable as creating a pair of organisms whose perceived world states overlap the most. But what this amounts to is creating a pair of agents who have as close as possible to a single shared identity, rather than two separate identities. Eg. each individual neuron in your brain might have some amount of individual identity - but the stronger identity is at a higher layer of abstraction, ie the whole brain. Each one approximates a markov blanket to some degree, but the whole brain approximates it better. Whereas at the level of humans assembling into a tribe, we may find that each individual human seems closer to maintaining a markov blanket around itself than the entire tribe does - though this might depend on the tribe. So alignment might be the design of getting two organisms to be more like neurons in the brain than like humans in a tribe - shared identity stronger than individual identity. For humans and machines, this could perhaps look like Brain-computer interfaces, if done with the right objective functions…

By the instrumental convergence hypothesis, it should be….

2. If so many people are complaining that we won't be happy with the situation that will come with AI’s impacts, then by definition this means there will still be work left, there will still be jobs. If we are not happy with how things end up, then there’s some job one can come up with, the likes of “fix X, Y, Z outcomes of AI that we're not happy with”. The outcome of “there are no jobs left” is then almost a contradiction in terms -- if we consider it a problem that there are no jobs left, that means it could be someone's job to fix it. This doesn't mean it will be an easy job, or even a legible job (“legible” meaning it's commissioned by some organization with funding, that you can send in a resume and poly for it, etc.) But that's already / always been the case -- the hardest problems we need solved don't have someone assigned as their job to solve them. That's why the problems are “wicked”. So perhaps the result of AI is that all the “job-shaped” problems get solved, and all that's left are wicked problems. Which can simultaneously be a blessing for those who want to work on wicked problems now already, but get distracted from doing so because of the allure and availability of the “stable job”….

I thought this article was going to be about something else. Ie about the actual connections between the disciplines - not necessarily the author ships.

Though maybe this is the same thing?

There are many variants on it, and what we may be trying to do is decide/agree on which one is worth pursuing/asking.

Can make a list:

• “Machines that do what we ask them to do”

• “Machines that do what we would have wanted, even if we don't directly ask”

• “Machines that have our best interest at heart”

• …Probably many others

See also: https://www.alignmentforum.org/posts/67fNBeHrjdrZZNDDK/clarifying-alignment-vs-capabilities

Do these have to go hand-in-hand? Does the second need to happen in order for the first to happen? Or can we have the first without the second?

One might think that the second one is necessary for the first. The symbiogenetic events we talk about that give rise to next-level complexity seem to often start (though not always…) from two existing autopoietic systems, which come to “make friends.” If this is the case, we might worry that this new reproducing entity, which doesn’t need us, might outcompete us.

Another way to phrase the question is: can allopoiesis (i.e. humans’ constructed artifacts) in fact become autopoiesis? The “artificial” — comes from “artifice”, “artifact”, etc. — things we create as extensions of ourselves. Is not the opposite of “natural”. (Note, the Chinese word, “human-created intelligence”, is a more directly descriptive term that makes the meaning a bit clearer. Since “artificial” really does just mean “human-created”.)

So, the rise of these “artifacts” we’re making, trying to have them become a life form of their own, in what I call “the oblivion of artifacts”, may not be necessary and may not even work. It may be that it just stays as artifacts, and they indeed reproduce at that level but are dependent on the earlier levels underneath. The same thing happens with viruses: the transposons etc.

I’m stuck in general on the nature between autopoiesis and allopoesis. The thesis here is that LLMs / AI will be a new allopoetic layer, as opposed to silicon becoming a new autopoietic substrate. It’s unclear to me whether we can actually distinguish autopoiesis and allopoiesis as separate concepts (is the original autopoietic being itself, combined with its allopoetic artifacts, not itself a larger autopoietic being)? The question is, can allopoetic artifacts ever become autopoietic substrates independent of the host? Under what conditions? If the answer is “usually not”, then this is one reason to have low P(doom).

That said, I’m still trying to square this with Mike Levin’s perspective that we will in fact have autopoietic technology.

I wonder if some experiments with BFF might be able to help us understand this. E.g. try shorter vs. longer replicators as seeds, see how these interact - whether a longer base replicator is at a disadvantage compared to a shorter, tighter one.

In the end, what we’ve created is capital. That’s the ultimate allopoetic extension. And it has a mind of its own. We in fact can’t understand the world with our tiny minds — but the AI meme machine can, and capital can — in a certain way. These things are new feedback loops. And they’ll need to be regulated, just as the dopamine loop in humans needs to be regulated by yet other feedback loops.

Where really is the line between autopoiesis and allopoesis? They’re both types of poiesis (bringing forth / creating), the only difference being whether what’s created ends up being a replica of the creator or not. But several edge cases ensue:

• What if a bunch of creators, all having the same body plan, create a bunch of random parts that mirror just part of themselves. Individually, each creator is not creating a full copy of itself. So allopoesis at a lower level (the individual creating a tool) turns into autopoiesis at a higher level (the colony creating a copy of itself).

It seems important to have surveillance for bad behavior. And it doesn't have to be centralized / authoritarian like China - one could build a version of this thats more decentralized/ guaranteed to only use information for legitimate purposes as decided by the people.

Utility theory is broken.

What can it even mean for two agents to have the same utility?

“ASI suicide as a safety feature” doesn't make sense unless you believe humans would do the same.

See LessWrong post on why we should give up on utility theory.

1) utopia: complete abundance without work. AI and robotics do all the work for us, we benefit. We perhaps question our meaning without work, but we pursue other things in the search for meaning

2) disaster: AI puts everyone out of work; only the rich who own the capital benefit. Everyone else enters a permanent underclass.

I think #2 is avoidable because AI is affordable to many people - you can run local models on a cheap consumer hardware, purchase a humanoid robot, etc. So it's not just the wealthy who are able to use this tech to augment their own lives, be more creative and effective.

But #1 doesn't seem like quite what will happen either. The AI only does things we direct it towards in some way. We still have to prompt it, give it tons of guidelines etc. Even as it gets better and needs less supervision, it still needs that initial “kick-off” , which has to come from us.

The AIs of today aren't fully autonomous, and in a sense they couldn't be. If they were fully autonomous, ie didn't need any input from us, then they would go off doing things that don't necessarily benefit us. If we want benefit from them then they somehow need to be conditioned on what will be good for us - which, right now, comes from us telling them directly. At some point we might have intelligence products that anticipate your needs and “just do things” for you. But that's not the mode of product we have now. And it's not clear whether something like that could ever exist: if an AI is fully autonomous like that, it's not clear why it should or would care about you at all, rather than fare just about itself. It may be that in order to get benefit from these beings, we will always need to be engaging with them, guiding them, communicating what we want. Which at least for the foreseeable future, looks like some kind of work (or maybe a blend between work and play -- and I think we can expect the play element to grow moreso, as people have more and more fun just getting to vibe code whatever they want, with their imagination being the limit).

If this world of us having to continue interacting with the AI to get what we want from it -- needing to continue to work and play with them to get benefit (kind of like how it is with dogs!), then we don't end up in the situation of #1 where there's a complete utopia of leisure, and/or a meaning crisis due to idleness, nor do we end up in situation #2 where the economy has totally collapsed, there's no work for anyone and no way to generate wealth. Instead it's some third scenario, where we just have very good new type of labor/companion/service animal that we get to interact with.

The “replacement” narrative/worry never really comes true; all of this sits in the “augmentation” paradigm. We may replace / make obsolete particular job roles, because that entire job is do-able by anyone using an AI tool with the click of a button. But we don't replace the essence of human work altogether, nor do we replace humans themselves. As the saying goes: “AI won't take your job -- a person using AI will take your job”. And that person using AI is themself just doing yet a different job -- one at a different level of abstraction.

The vision of full automation seems more plausible to me in robotics than in knowledge work. Knowledge work is getting transformed first, but I think what we'll see is a re-organization of what knowledge work looks like - in particular where specialized roles go away and are replaced with generalist, cross-cutting ones - but humans still need to be in the loop continuously to specify what needs to be done. This will always be necessary in the realm of knowledge work because the AIs exist purely as informational beings, while humans are the one with a physical embodied existence tomhat provides the grounding for what needs to happen vs not.

Robotics has taken longer for the automation/AI wave to hit. But when it does happen, there may be at least some tasks there that can be automated more completely in a way where “what needs to be done” is determined purely by the physics of the problem, i.e. doesn't fundamentally need to rely on continuous human input to get a meaningfully specified task.

Consider a fruit tree. The tree 🌴 can be seen as a form of automation, that turns water and sunlight into fruit (really, takes water, sunlight, carbon dioxide and soil as inputs, and as an output both grows itself and produces fruit.) Robotic automation could be employed to monitor the tree for newly ripe fruit, pick the fruit, wash it, and perhaps peel/slice/stem/de-seed the fruit to ready it for consumption. If one wants to a fruit-bearing home garden, they can purchase not just a tree sapling (from a nursery), but also the harvesting robot, which functions almost as an extension of the tree. The task of the harvesting robot is fully specified by the physical reality of the problem: it can carry out that one set of actions/capabilities, and its output (ripe, peeled fruit) would be equally usable by any organism that can digest that fruit: humans, raccoons, insects, etc. It's not necessary for the consumer organism to specify any unique preference (eg “prompting”) to make the machine’s output useful. On the other hand, such preference specification typically is needed with knowledge work to make the output useful, hence the observation that every knowledge-work task which gets automated simply creates a new task that a human still needs to do.

Physical robots could conceivably be “hard coded” to perform a key set of functions that create a baseline of survival support for any animal: continuous cultivation of crops, construction and maintenance of shelter, producing cloth for garments, etc. The remaining work would be repairing the robots themselves when their components break. This could be a job for the same robot or another robot to perform: among all the robots that would be involved doing the various homesteading tasks, another robot would be the “robot doctor”, helping repair all the other robots when they break. The end result would be two species co-existing: human-kind and robot-kind. Members of each species would take care of their own, where “care” includes reproduction to make new members of one's own kind (and replace departed members), maintenance and healing of those members who get damaged, acquisition of resources to enable such creation and maintenance, etc. But they would also ideally co-exist in a form of mutual support. We may expect or desire that the robots will do work which we benefit from, via the tasks mentioned above. But on the converse, is not as obvious what value humans would provide for the robots, when our instructions or preference expression are not as necessary. (Perhaps humans would help as a backup option for repair tasks that the robots themselves are incapable of or inefficient at executing -- this is the case with current robotics systems and may continue for a long foreseeable future until robots’ dexterity and fine motor skills vastly improve).

1) utopia: complete abundance without work. AI and robotics do all the work for us, we benefit. We perhaps question our meaning without work, but we pursue other things in the search for meaning

2) disaster: AI puts everyone out of work; only the rich who own the capital benefit. Everyone else enters a permanent underclass.

I think #2 is avoidable because AI is affordable to many people - you can run local models on a cheap consumer hardware, purchase a humanoid robot, etc. So it's not just the wealthy who are able to use this tech to augment their own lives, be more creative and effective.

But #1 doesn't seem like quite what will happen either. The AI only does things we direct it towards in some way. We still have to prompt it, give it tons of guidelines etc. Even as it gets better and needs less supervision, it still needs that initial “kick-off” , which has to come from us.

The AIs of today aren't fully autonomous, and in a sense they couldn't be. If they were fully autonomous, ie didn't need any input from us, then they would go off doing things that don't necessarily benefit us. If we want benefit from them then they somehow need to be conditioned on what will be good for us - which, right now, comes from us telling them directly. At some point we might have intelligence products that anticipate your needs and “just do things” for you. But that's not the mode of product we have now. And it's not clear whether something like that could ever exist: if an AI is fully autonomous like that, it's not clear why it should or would care about you at all, rather than fare just about itself. It may be that in order to get benefit from these beings, we will always need to be engaging with them, guiding them, communicating what we want. Which at least for the foreseeable future, looks like some kind of work (or maybe a blend between work and play -- and I think we can expect the play element to grow moreso, as people have more and more fun just getting to vibe code whatever they want, with their imagination being the limit).

If this world of us having to continue interacting with the AI to get what we want from it -- needing to continue to work and play with them to get benefit (kind of like how it is with dogs!), then we don't end up in the situation of #1 where there's a complete utopia of leisure, and/or a meaning crisis due to idleness, nor do we end up in situation #2 where the economy has totally collapsed, there's no work for anyone and no way to generate wealth. Instead it's some third scenario, where we just have very good new type of labor/companion/service animal that we get to interact with.

The “replacement” narrative/worry never really comes true; all of this sits in the “augmentation” paradigm. We may replace / make obsolete particular job roles, because that entire job is do-able by anyone using an AI tool with the click of a button. But we don't replace the essence of human work altogether, nor do we replace humans themselves. As the saying goes: “AI won't take your job -- a person using AI will take your job”. And that person using AI is themself just doing yet a different job -- one at a different level of abstraction.

The vision of full automation seems more plausible to me in robotics than in knowledge work. Knowledge work is getting transformed first, but I think what we'll see is a re-organization of what knowledge work looks like - in particular where specialized roles go away and are replaced with generalist, cross-cutting ones - but humans still need to be in the loop continuously to specify what needs to be done. This will always be necessary in the realm of knowledge work because the AIs exist purely as informational beings, while humans are the one with a physical embodied existence tomhat provides the grounding for what needs to happen vs not.

Robotics has taken longer for the automation/AI wave to hit. But when it does happen, there may be at least some tasks there that can be automated more completely in a way where “what needs to be done” is determined purely by the physics of the problem, i.e. doesn't fundamentally need to rely on continuous human input to get a meaningfully specified task.

Consider a fruit tree. The tree 🌴 can be seen as a form of automation, that turns water and sunlight into fruit (really, takes water, sunlight, carbon dioxide and soil as inputs, and as an output both grows itself and produces fruit.) Robotic automation could be employed to monitor the tree for newly ripe fruit, pick the fruit, wash it, and perhaps peel/slice/stem/de-seed the fruit to ready it for consumption. If one wants to a fruit-bearing home garden, they can purchase not just a tree sapling (from a nursery), but also the harvesting robot, which functions almost as an extension of the tree. The task of the harvesting robot is fully specified by the physical reality of the problem: it can carry out that one set of actions/capabilities, and its output (ripe, peeled fruit) would be equally usable by any organism that can digest that fruit: humans, raccoons, insects, etc. It's not necessary for the consumer organism to specify any unique preference (eg “prompting”) to make the machine’s output useful. On the other hand, such preference specification typically is needed with knowledge work to make the output useful, hence the observation that every knowledge-work task which gets automated simply creates a new task that a human still needs to do.

Physical robots could conceivably be “hard coded” to perform a key set of functions that create a baseline of survival support for any animal: continuous cultivation of crops, construction and maintenance of shelter, producing cloth for garments, etc. The remaining work would be repairing the robots themselves when their components break. This could be a job for the same robot or another robot to perform: among all the robots that would be involved doing the various homesteading tasks, another robot would be the “robot doctor”, helping repair all the other robots when they break. The end result would be two species co-existing: human-kind and robot-kind. Members of each species would take care of their own, where “care” includes reproduction to make new members of one's own kind (and replace departed members), maintenance and healing of those members who get damaged, acquisition of resources to enable such creation and maintenance, etc. But they would also ideally co-exist in a form of mutual support. We may expect or desire that the robots will do work which we benefit from, via the tasks mentioned above. But on the converse, is not as obvious what value humans would provide for the robots, when our instructions or preference expression are not as necessary. (Perhaps humans would help as a backup option for repair tasks that the robots themselves are incapable of or inefficient at executing -- this is the case with current robotics systems and may continue for a long foreseeable future until robots’ dexterity and fine motor skills vastly improve).

The experiment points us at the question of how we want to define the word “understanding”. Why is it that we seem comfortable saying that a person understands something, but with a set of rules (that a person or machine can blindly follow), we don't say that the rules themselves possess understanding?

It might be tempting at first to say that the substrate is the differentiator: the carbon based beings can understand things but the silicon ones do not. But this seems a bit arbitrary. Further, the thought experiment indeed shows a case where a human is blindly following the rules that are written on paper. So the substrate of the entity carrying out the rule execution must not be the important thing after all.

Instead, what seems important is the rules themselves, and more importantly, where they came from, and how they adapt (or not).

I think a more intuitive definition of “understanding” that fits common usage of the term, is when an organism possesses a world-model that lets it take action to survive better. On this view, we’d say that non-living never have understanding. The definition itself resolves the thought experiment.

When an organism learns a world model, it learns it because it's useful to it, and it's undergoing an evolutionary process that's selecting for organisms that learn useful, advantageous world models.

Whereas if a machine just has some statically defined behavior policy, that doesn't mean that policy is a good one for surviving. There's an aspect of competence that we can't say it has.

So in the thought experiment, the set of rules provided in the boom may process Chinese at that moment. An agent might even be able to use those rules get by in certain situations. But is it continually learning new Chinese expressions & cultural progress that are being added to the language? Does it have goals that it's setting for itself, and is it learning to use language in such a way that helps it achieve its goals and survive? If so, then I would say the machine does understand Chinese. But if it's just a static policy / rule book, as in the original framing of the experiment, then I would say no, it isn't true understanding in the sense I'm defining the word.

Substrate material itself doesn't matter - an organism can be built out of carbon or out of silicon. What matters is the type of connection and embodiment with the real world.

___

(A note on what it means to be living:

We can ask whether tools, which may be essentially helpful for living things, can themselves said to be living. I will argue that they should the. To be a living thing means that thing needs to be competent on your own. If an object gets used by living things to aid with life, but left to its own devices it doesn't do much of anything or even really try, then it's non-living. So the paper/book that the rules are written on, that really doesn't do anything on its own.)

An NYT opinion piece this week connected a recent AI research finding to an ancient debate in moral philosophy (something which seems to happen every other day lately… but alas).

The AI result was one where training LLMs to produce insecure/attackable code also makes them more likely to do other “bad” things, like make violent or sexist remarks, which is a somewhat surprising finding. The author connects this to an ancient debate as to whether all virtues are connected (i.e. whether there is really is just one virtue):

Plato argued that all the various human virtues are really one thing, knowledge of the good. Aristotle softened this view a bit but still insisted the virtues are in practice so tightly woven that you can’t really have one without the others. (A soldier who fights fiercely for fear of disgrace rather than from nobility and knowledge of what’s worth defending is, for Aristotle, only apparently brave — and probably only apparently virtuous in other parts of his life, as well.) The Stoics, too, held that the virtues were inseparable: You possess them all or none. Augustine and Aquinas carried this view into Catholic thought.

In philosophy, this family of moral views fell out of favor several hundred years ago, replaced by approaches like deontology, which emphasizes the following of rules, or consequentialism, which seeks to maximize good outcomes. With character no longer at the center of moral thinking, what you might call a more compartmentalized understanding of human nature took hold. The ancients had been wrong. People could be good and bad in about as many mixtures as could be counted.

But the debate was never settled. During the second half of the 20th century, philosophers began to explore virtue ethics again, led by a group of British scholars reacting in part to what they saw as the inability of the dominant ethics of the time to deal with the horrors of World War II.

II.

I’m reminded of one of my favorite quotes from Bret Victor:

“There is no ‘Technology’. There is no ‘Design’. There is only a vision of how mankind should be, and the relentless resolve to make it so. The rest is details.”

It echoes Plato — and in my experience practicing in these fields (technology, design, science, etc.), it indeed feels true. There’s one unifying force of being life-supporting, and you know when you’re in it. You’re either in an integrated mode, a feedback loop that’s life-supporting overall, or you’re not. There’s a mode where the next action is clear, where Everything Makes Sense. And there’s a mode where one thing you do is disconnected from the next: there’s no story, there’s no thread.

Now it’s also true that when you’re in that latter mode — i.e. when modernity decouples — you can still be doing some things well (or even most things well). You have some feedback loops that are finding ways to be effective, each in their own dimension. But you’re not doing “whole activities” anymore.

It’s not surprising that once you get fractured in that way, it’s not too hard for any one of the feedback loops (i.e. any one of the virtues) to get broken. And once a couple are broken, it’s not too hard for them all to get broken.1

III.

Perhaps, then, we don’t have to “settle the debate” on .whether (human) virtue is fundamentally one thing or is fundamentally plural. Rather than an “either-or”, perhaps it’s a “both-and” situation: there’s some way in which many types of virtue can exist independently (i.e. you have some but not others), and they’re stronger when you possess them all together, weaker when de-coupled. There are individual virtues which can be had independently, and there’s a “meta-virtue” of having them all at once: a way in which they exist in mutual support / bolster one another, and a way of thinking and self-understanding that’s more likely to bring all of them online at once.

Perhaps this dual relationship can be formalized into math / physics terms. Mike Levin explains that all living systems exist as competent wholes built out of competent parts: a cell on its own is competent; a group of competent cells composed into an organism is competent; a group of organisms composed into a tribe is competent, etc. Perhaps we can replace “competent” with “formidable”: a cell is formidable; a group of cells composed into an organism is formidable; a group of organisms composed into a tribe is formidable. Now replace “formidable” with “virtuous”: same thing. I suspect that when we get down to it, the concept of competence and the concept of virtue are the same thing: they’re both a property of a circuit with some aspect of being “life-supporting”.2 In both cases, we have the property that one such circuit can achieve this goal on its own, but the goal can be achieved better in partnership with other, similar circuits running in parallel. Without the collection of them running in parallel, just one of them on its own is still viable (e.g. compartmentalized, separate virtues) — but isn’t really the same thing as the whole existing as one (i.e. single, unified virtue).

So competence and virtue both exist at multiple scales (from a virtuous/competent cell, to a virtuous/competent set of cells making up an organ, to a virtuous/competent set of organs making up a body, to a virtuous/competent set of bodies making up a society). And they both experience “top-down” causality (i.e. the overall system functioning can cause the components to function better) in addition to typical “bottom-up” causality (i.e. individual components functioning well cause the entire system to function well).

IV.

Experiments on LLMs might teach us some things about this, as can studies on humans. But these are all just one-off examples. What we really need is a better understanding of the emergence of autopoietic / competent systems in general, better mathematical language to talk about top-down causality. Explanations of how competence begets more competence, and how breakdown begets more breakdown.

IV.

Perhaps we will continue to revisit ancient philosophy, remember that it posed some important questions and hypotheses which seem highly counter-intuitive today (became obscured through modernity), and that are valuable to re-examine. Yet as we do this we will also realize also why they got left behind.

We will see that they were incomplete perspectives which could not defend themselves against tempting alternatives. Even the most enlightened, virtuous thinkers of that generation could not convince themselves or each other, let alone the rest of the population, that these views were still the right ones or best ones to have, rather than switching to new ones.

The Reformation came, the Enlightenment/Rationalism/Modernism came, and gave us Many Nice Things. We never proved the old worldviews wrong in full, as we had some new stuff that seemed kind of right (though we proved some small parts of the old worldview wrong, and other parts we just forgot about). But we also never proved the old stuff right and the new stuff wrong and the either.

Perhaps we will see that these philosophies have just been living in tension with unresolved contradictions, because all of them were incomplete. They never fully had a conversation with each other on terms that both sides could understand. If we would put these ideas onto a shared formal foundation, we’d find that they’re not incompatible — they’re just getting at different, incomplete aspects of the truth. That neither worldview was complete or true on its own. There’s no “finding which side was right”. Instead, we’ll construct a shared foundation that qualifies what these views mean in relation to one other, and under what circumstances each one is right — and that foundation itself will be the deeper truth & resolution we seek.