Elliott Thornley

I'm an Assistant Professor of Philosophy at the National University of Singapore.

I work on AI safety. Right now, I'm using ideas from decision theory to train safer artificial agents.

I also do work in ethics, focusing on the moral importance of future generations.

You can email me at elliott.thornley@gmail.com.

You can find my CV here.

AI Safety

Risk-Averse AIs

Character Training for Risk-Averse Agents

with Arav Dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, and David Demitri Africa

2026

Abstract

Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent’s resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate different aspects of the constitution, finding that token budget and model choice are the most influential aspect of character training to instill risk aversion. We conclude from these results that character training is a promising and scalable way to instil broad dispositions, which we can use to our advantage in mitigating risk from misaligned AI agents.

Out-of-Distribution Generalization of Risk Aversion in Language Models

with Kristina Zhang, Junior Chinomso Okoroafor, Benjamin Maltbie, Andrew Lin, and Abhitej Bokka

2026

Abstract

Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned. Misaligned but risk-averse AIs would tend to prefer low-risk, low-reward strategies like cooperation over high-risk, high-reward strategies like rebellion, limiting the downsides of any misalignment. But we can only feasibly train AIs to be risk-averse on low-stakes gambles, and we will only be safe if their risk aversion generalizes to astronomically-high-stakes gambles. Will it? To shed light on this question, we introduce RiskAverseOOD: a benchmark for measuring how well risk aversion generalizes out of distribution. We then offer some initial results. Using a variety of methods to make Qwen3-8B choose risk-aversely when the stakes are low, we find that we can induce substantial risk aversion when the stakes are astronomically high. From a baseline 2% rate of choosing a safe ‘Cooperate’ option, we see rates around 70% (SFT and tie training) and 52% (DPO). Activation steering scores 78% but hurts capabilities and makes the model excessively risk-averse. We observe similar effects at different scales (Qwen3-1.7B and Qwen3-14B) and across model families (Gemma-3-12B-IT and Llama-3.1-8B-Instruct). Overall, we find that risk aversion learned at low stakes can generalize OOD to astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe. Achieving that level of consistency is an open problem.

Risk-Averse AIs

with William MacAskill

2026

Abstract

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs to cooperate with us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse.

Tie Training

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

with Christian Moya, Alex Semendinger, and Guang Lin

ICML, 2026

Abstract

Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today’s language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model’s dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.

Shutdownable Agents

Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

with Carissa Cullen, Harry Garland, Alexander Roman, Louis Thomson, and Christos Ziakas

ICLR Agents in the Wild Workshop, 2026

Abstract

Misaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for Same-Length Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, and thus incentivizes agents to (1) choose stochastically between different trajectory-lengths (be neutral about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be useful). In this paper, we use DReST to train deep RL agents and fine-tune LLMs to be neutral and useful. We find that these DReST agents generalize to being neutral and useful in unseen contexts at test time. Indeed, DReST RL agents achieve 11% (PPO) and 18% (A2C) higher usefulness on our test set than baseline agents, and our fine-tuned LLM achieves maximum usefulness and near-maximum neutrality. Our results provide some early evidence that DReST could be used to train more advanced agents to be useful and neutral. Prior theoretical work suggests that these agents would be useful and shutdownable.

Shutdownable Agents through POST-Agency

2025

Abstract

Many fear that future artificial agents will resist shutdown. I present an idea – the POST-Agents Proposal – for ensuring that doesn’t happen. I propose that we train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). I then prove that POST – together with other conditions – implies Neutrality+: the agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths. I argue that Neutrality+ keeps agents shutdownable and allows them to be useful.

Towards Shutdownable Agents via Stochastic Choice

with Alexander Roman, Christos Ziakas, Leyton Ho, and Louis Thomson

Transactions on Machine Learning Research, 2025

NeurIPS ARLET Workshop, 2025

Technical AI Safety Conference, 2025

Abstract

The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the PAP is using a novel ‘Discounted Reward for Same-Length Trajectories (DReST)’ reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be useful), and (2) choose stochastically between different trajectory-lengths (be neutral about trajectory-lengths). In this paper, we propose evaluation metrics for usefulness and neutrality. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be useful and neutral. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be useful and neutral. Our theoretical work suggests that these agents would be useful and shutdownable.

The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists

Philosophical Studies, 2025

Abstract

I explain and motivate the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems suggest that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it’s costly to do so. I end by noting that these theorems can guide our search for solutions to the problem.

Ethics

A Non-Identity Dilemma for Person-Affecting Views

Australasian Journal of Philosophy, forthcoming

Abstract

Person-affecting views state that (in cases where all else is equal) we’re permitted but not required to create people who would enjoy good lives. In this paper, I present an argument against every possible variety of person-affecting view. The argument is a dilemma over trilemmas. Narrow person-affecting views imply a trilemma in a case that I call ‘Expanded Non-Identity.’ Wide person-affecting views imply a trilemma in a case that I call ‘Two-Shot Non-Identity.’ One plausible practical upshot of my argument is as follows: we individuals and our governments should be doing more to reduce the risk of human extinction this century.

A Fission Problem for Person-Affecting Views

Ergo, 2026

Abstract

On person-affecting views in population ethics, the moral import of a person’s welfare depends on that person’s temporal or modal status. These views typically imply that—all else equal—we’re never required to create extra people, or to act in ways that increase the probability of extra people coming into existence. In this paper, I use Parfit-style fission cases to construct a dilemma for person-affecting views: either they forfeit their seeming-advantages and face fission analogues of the problems faced by their rival impersonal views, or else they turn out to be not so person-affecting after all. In light of this dilemma, the attractions of person-affecting views largely evaporate. What remains are the problems unique to them.

Critical-Set Views, Biographical Identity, and the Long Term

Australasian Journal of Philosophy, 2025

Abstract

Critical-set views avoid the Repugnant Conclusion by subtracting some constant from the welfare score of each life in a population. These views are thus sensitive to facts about biographical identity: identity between lives. In this paper, I argue that questions of biographical identity give us reason to reject critical-set views and embrace the total view. I end with a practical implication. If we shift our credences towards the total view, we should also shift our efforts towards ensuring that humanity survives for the long term.

The Procreation Asymmetry, Improvable-Life Avoidance, and Impairable-Life Acceptance

Analysis, 2023

Abstract

Many philosophers are attracted to a complaints-based theory of the procreation asymmetry, according to which creating a person with a bad life is wrong (all else equal) because that person can complain about your act, whereas declining to create a person who would have a good life is not wrong (all else equal) because that person never exists and so cannot complain about your act. In this paper, I present two problems for such theories: the problem of impairable-life acceptance and an especially acute version of the problem of improvable-life avoidance. I explain how these problems afflict two recent complaints-based theories of the procreation asymmetry, from Joe Horton and Abelard Podgorski.

Critical Levels, Critical Ranges, and Imprecise Exchange Rates in Population Axiology

Journal of Ethics and Social Philosophy, 2022

Abstract

According to critical-level views in population axiology, an extra life improves a population if and only if that life’s welfare level exceeds some fixed “critical level.” An extra life at the critical level leaves the new population equally good as the original. According to critical-range views, an extra life improves a population if and only if that life’s welfare level exceeds some fixed “critical range.” An extra life within the critical range leaves the new population incommensurable with the original. In this paper, I sharpen some old objections to these views and offer some new ones. Critical-level views cannot avoid certain repugnant and sadistic conclusions. Critical-range views imply that lives featuring no good or bad components whatsoever can nevertheless swallow up and neutralize goodness and badness. Both classes of view imply discontinuities in implausible places. I then offer a view that retains much of the appeal of critical-level and critical-range views while avoiding the above pitfalls. On the Imprecise Exchange Rates View, various exchange rates—between pairs of goods, between pairs of bads, and between goods and bads—are imprecise. This imprecision is the source of incommensurability between lives and between populations.

Is Global Consequentialism More Expressive Than Act Consequentialism?

Analysis, 2022

Abstract

Act consequentialism states that an act is right if and only if the expected value of its outcome is at least as great as the expected value of any other act’s outcome. Two objections to this view are as follows. The first is that act consequentialism cannot account for our normative ambivalence in cases where agents perform the right act out of bad motives. The second is that act consequentialism is silent on questions of character: questions like ‘What are the right motives to have?’ and ‘What kind of person ought I be?’. These objections have been taken to motivate a move to global consequentialism, on which acts are not the only subjects of normative assessment. Motives and decision-procedures (amongst other things) are also judged right or wrong by direct reference to their consequences. In this paper, I argue that these objections fail to motivate the move from act to global consequentialism.

The Impossibility of a Satisfactory Population Prospect Axiology (Independently of Finite Fine-Grainedness)

Philosophical Studies, 2021

Abstract

Arrhenius’s impossibility theorems purport to demonstrate that no population axiology can satisfy each of a small number of intuitively compelling adequacy conditions. However, it has recently been pointed out that each theorem depends on a dubious assumption: Finite Fine-Grainedness. This assumption states that there exists a finite sequence of slight welfare differences between any two welfare levels. Denying Finite Fine-Grainedness makes room for a lexical population axiology which satisfies all of the compelling adequacy conditions in each theorem. Therefore, Arrhenius’s theorems fail to prove that there is no satisfactory population axiology. In this paper, I argue that Arrhenius’s theorems can be repurposed. Since all of our population-affecting actions have a non-zero probability of bringing about more than one distinct population, it is population prospect axiologies that are of practical relevance, and amended versions of Arrhenius’s theorems demonstrate that there is no satisfactory population prospect axiology. These impossibility theorems do not depend on Finite Fine-Grainedness, so lexical views do not escape them.

A Dilemma for Lexical and Archimedean Views in Population Axiology

Economics and Philosophy, 2021

Abstract

Lexical views in population axiology can avoid the Repugnant Conclusion without violating Transitivity or Separability. However, they imply a dilemma: either some good life is better than any number of slightly worse lives, or else the ‘at least as good as’ relation on populations is radically incomplete. In this paper, I argue that Archimedean views face an analogous dilemma. I thus conclude that the lexical dilemma gives us little reason to prefer Archimedean views. Even if we give up on lexicality, problems of the same kind remain.

Papers in Population Ethics

DPhil (PhD) Thesis, 2023

Abstract

This thesis consists of a series of papers in population ethics: a subfield of normative ethics concerned with the distinctive issues that arise in cases where our actions can affect the identities or number of people of who ever exist. Each paper can be read independently of the others. In Chapter 1, I present a dilemma for Archimedean views in population axiology: roughly, those views on which adding enough good lives to a population can make that population better than any other. In Chapter 2, I extend Gustaf Arrhenius’s famous impossibility theorems in population axiology into the domain of choices under risk. My risky impossibility theorems dispense with the assumption that welfare levels are finitely fine-grained, and so tell against lexical views in population axiology. In Chapter 3, I present objections to critical-level and critical-range views in population axiology. I then sketch out what I call the ‘Imprecise Exchange Rates View’ and argue that it is an attractive alternative. In Chapter 4, I address critical-level and critical-range views again. This time, I note that they are vulnerable to objections from biographical identity: identity between lives. I suggest that these objections give us reason to reject critical-level and critical-range views and embrace the Total View. In Chapter 5, I argue that objections of the same form – objections from personal identity – tell against person-affecting views in population ethics. In Chapter 6, I draw out some counterintuitive implications of two recent complaints-based theories of the procreation asymmetry.

Policy

How Much Should Governments Pay to Prevent Catastrophes? Longtermism's Limited Role

with Carl Shulman

Essays on Longtermism, Oxford University Press, 2025

Abstract

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this chapter, we show that the case for preventing catastrophe does not depend on longtermism. Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy.

Writing

Subscribe to my Substack.

AI Philosophy Competition: $11,000 in Prizes

2026

Abstract

AI is now exceptionally capable in mathematics and coding, but how good is it at philosophy? Zachary Goodsell and I are organising the first AI Philosophy Competition to find out.

Challenging the Unawareness Argument with Third Actions and Mixed Actions

2026

Abstract

I challenge Anthony DiGiovanni's unawareness argument using third actions and mixed actions.

AI Writing Is Not Repetitive Enough

2026

Abstract

One big defect of AI writing is the use of synonyms to avoid repetition.

Statistically, the Truth Is Disappointing

2026

Abstract

I prove that, given some reasonable assumptions, noticing that you want X to be true should make you think X is less likely to be true.

How Big Is the Sun? How Could You Figure It Out?

2026

Abstract

I trace a route by which the ancient Greeks could have estimated the size of the Sun.

Tie Training Can Make DPO/RLHF-Trained AIs Generalize Better

2026

Abstract

This post covers our ICML paper, ‘Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training’.

The Official Laws of Football Require Some Players to Trap Themselves in the Goal

2026

Abstract

A strange consequence of the official Laws of the Game.

Anonymous Peer Review Is Almost Over

2026

Abstract

Claude Opus 4.7 is very good at guessing who wrote a piece of text. That could be a problem for anonymous peer review.

Life Is Long

2026

Abstract

People say life is short. Compared to what?

You Are Quoting Bentham

2026

Abstract

Jeremy Bentham coined many words we use today, including ‘maximize’, ‘international’, and ‘deontology’.

Taking Scanlon (Too) Seriously

2026

Abstract

As written, Scanlon's Transmitter Room case presents no problem for utilitarianism.

AI Safety Can Be a Pascal's Mugging Even If p(doom) Is High

2026

Abstract

Pascal's muggings are about the probability that you make a difference, not about baseline risk.

My Favourite Arguments Against Person-Affecting Views

2024

Abstract

I present some of my favourite arguments against person-affecting views in population ethics.

There Are No Coherence Theorems

2023

Abstract

Authors in the AI safety community often make the following claim: advanced artificial agents will behave like expected utility maximisers, because doing so is the only way to avoid pursuing dominated strategies. I argue that the claim is false.

Summary: The Case for Strong Longtermism

Global Priorities Institute, 2022

Abstract

In this paper, Hilary Greaves and William MacAskill make the case for strong longtermism: the view that the most important feature of our actions today is their impact on the far future. They claim that strong longtermism is of the utmost significance: that if the view were widely adopted, much of what we prioritise would change.

Summary: The Epistemic Challenge to Longtermism

Global Priorities Institute, 2022

Abstract

According to longtermism, what we should do mainly depends on how our actions might affect the long-term future. This claim faces a challenge: the course of the long-term future is difficult to predict, and the effects of our actions on the long-term future might be so unpredictable as to make longtermism false. In ‘The Epistemic Challenge to Longtermism,’ Christian Tarsney evaluates one version of this epistemic challenge and comes to a mixed conclusion. On some plausible worldviews, longtermism stands up to the epistemic challenge. On others, longtermism’s status depends on whether we should take certain high-stakes, long-shot gambles.

The Moral Case for Long-Term Thinking

with Hilary Greaves and William MacAskill

The Long View, 2021

Abstract

This chapter makes the case for strong longtermism: the claim that, in many situations, impact on the long-run future is the most important feature of our actions. Our case begins with the observation that an astronomical number of people could exist in the aeons to come. Even on conservative estimates, the expected future population is enormous. We then add a moral claim: all the consequences of our actions matter. In particular, the moral importance of what happens does not depend on when it happens. That pushes us toward strong longtermism. We then address a few potential concerns, the first of which is that it is impossible to have any sufficiently predictable influence on the course of the long-run future. We argue that this is not true. Some actions can reasonably be expected to improve humanity’s long-term prospects. These include reducing the risk of human extinction, preventing climate change, guiding the development of artificial intelligence, and investing funds for later use. We end by arguing that these actions are more than just extremely effective ways to do good. Since the benefits of longtermist efforts are large and the personal costs are comparatively small, we are morally required to take up these efforts.

Wittgenstein's Tractatus: Now With Examples

2021

Abstract

I explain the Tractatus using plenty of examples.

Against Ambition

2019

Abstract

I argue that personal ambition is bad for you.