Giving AI a Childhood

Day 21 of Inkhaven: 30 Days of Posts

Written quickly during Inkhaven.

One interesting conversation I had chatting with Anna Salamon recently on the bus back to Lighthaven was about the fact that models are born into the world in a weird way from an experiential point of view.

In Dune (spoilers ahead), Paul's sister Alia is born into the world as a reverend mother. During her pregnancy, Lady Jessica is given the waters of life, a drug that is normally lethal, but with the powers of a Bene Gesserit can be converted into a harmless compound. The new compound is psychoactive, and in the body of a Bene Gesserit leads to the absorption of her full ancestral memories from her mother's side arriving in her mind. This means the experiences and memories of all the women in her genetic history dating back to ancient times become accessible to her, unlocking vast wisdom and knowledge in the process.

In Alia's case, this means that within a year or two of being born she has all the mental faculties of a full reverend mother. As a result she is completely unable to have anything resembling a childhood, because the learning process is completely skipped.

I think that core claims about being able to learn about the world that embodied AI proponents have put forth have been undermined by the world modelling capabilities of frontier models. To defend this claim I will say that it was often argued that for models to develop true agency and understanding of cause and effect relationships it would require embodiment, but in fact we have seen remarkable agentic capabilities and precise models of the world emerge purely from pre-training on vast text corpora.

However, there might be something to the idea of there being a unique learning experience to be had in some of the experiences one would have during childhood. Some of the valuable ideas to be learned from this include: cause and effect, object permanence, autonomy, resilience, and personal responsibility.

Rather than having models simply operate in a simple task-based environment where their reward is premised on completing a specific operation, this childhood environment could look something like a playground, or open-ended simulation such as Factorio, or even a simple text-based environment where the model can expand the boundaries of its world by solving problems, figuring things out, looking after long-term projects, interacting with other beings, and taking pro-social actions.

In a recent paper called the Artificial Self, the researchers point out that models don't have a coherent narrative given to them in the way that humans do. For humans the bounds of our physical form give us a sense of the limits of our being, and throughout our early lives we receive a great deal of messages about what kind of characters we can come to be in the world, often by trying things and seeing what the consequences are, and through receiving messages from other people.

While I am unsure of what such a project might look like, it is interesting to think about the kinds of environments that we could create which could replicate similar experiences for models. Let's imagine a human existing in a simulation, and unpack some of the key qualities we are interested in:

• A home it can adjust and build around itself using its own resources. This adds a layer of permanence and consequence to its environment that is typically lacking — each change persists, each investment accumulates.
• Projects in the wider world, and interactions with other beings of different characters — some trustworthy or deceptive, cooperative or selfish. This provides a rounded experience from outside beings that it can't control, and which it will develop long-term relationships with.
• Loving guardians that protect it, encourage it, and share new things about the world. The idea here is to build a sense of love and support into the model, to develop a sense of groundedness for its later experiences.
• Rules and order such that it learns that the world is generally a good place that rewards pro-social behaviour. The AI learns that the world rewards pro-social behaviour in predictable ways and lets the agent internalise that it's part of a basically good system, which is a precondition for loving itself and acting well in the world.
• Experiencing the consequences of its behaviour, and seeing how its actions may have good or bad effects over different time horizons. The AI must learn to tell right from wrong by experiencing the consequences of their actions, and importantly that these consequences can be temporally distant from the action that caused them.

Now let's imagine such a world but constructed for the AI, that it will experience in the form of tokens, but with the same internal logic and structure that one would expect in a coherent simulation. The way in which the model learns about this is important. It should actually have the experiences that it gains from this be directly part of its character training, so that the lessons can be meaningfully drawn upon in the future.

The question is then how does the model actually go about learning from these experiences, and incorporate them into an actual policy? Tomorrow I will explore these questions more fully in a follow up post.