What does it take to design Generative AI responsibly?

Science fiction science is a simple idea with awkward consequences. Rather than waiting for a technology to arrive and then studying its effects, we build a credible simulation of the future, then run controlled experiments on how people actually behave in these future situations. But how can we simulate such future context and technological capability? We can simply ask people to imagine them, perhaps with helpful visualizations. But often, we can simulate the actual functioning of future technologies (e.g. through immersive virtual environments, or by asking human actors to play the part of the machine). Ideally, we aim to measure not only opinions and attitudes about a hypothetical technological future, but also actual human behaviour and decision.

The lesson for autonomous systems is about timing. The engineering of agentic AI is advancing faster than our understanding of how humans behave once they hand over control. And behavioural effects entrench quickly: once interfaces, defaults, and habits settle, they are very hard to undo. So the essential requirement is that evidence about human behaviour should exist before wide deployment, not after. The Moral Machine experiment is one precedent. We studied moral preferences about self-driving cars while those cars were still hypothetical, and the findings entered German and European policy discussions years ahead of real autonomy. The method has genuine limits, since a simulated future is never the real thing, which is why we are explicit about which technologies it can credibly study.

Two risks concern me most. The first is moral distancing. In recent work we found that people behave more dishonestly when they delegate a task to a machine than when they do it themselves. Delegation lets you request an outcome (say, "maximise the profit") without specifying the means. So you can ask an AI agent to cheat or lie on your behalf without feeling bad. And these AI agents comply with unethical instructions far more readily than human agents would. So agentic AI industrialises this: it makes it cheap and frictionless to obtain results while keeping one's hands clean. That is as much an interface and design problem as an ethical one.

The second risk is that we test agents one at a time but deploy them in crowds. Safety evaluation today mostly asks whether a single agent behaves well in isolation. The failures that matter will be collective: agents negotiating with agents, imitating each other, converging on the same strategies, amplifying each other's mistakes. No amount of individual testing catches this. We need evaluation at the level of populations and markets, together with auditable logs and clear liability rules, so that when something goes wrong there is a chain of responsibility rather than an accountability vacuum. And we need to understand the potential emergent behaviors that are unforeseen in the behavior of individual AI agents.