The Doxastic Account of Humility and AI Alignment

An AI that always agrees with its operator can be dangerously wrong, and so can one that treats its own judgement as beyond correction. The doxastic account of intellectual humility offers a way to think about both failures by asking how well an agent assesses the justification for its own beliefs.

What intellectual humility requires

Ian Church defends¹ what he calls a doxastic account, on which intellectual humility is the virtue of accurately tracking what one could non-culpably take to be the positive epistemic status of one’s own beliefs, meaning that confidence should fit the support a person could reasonably assess. A reasonable mistake about that support need not amount to arrogance.² He arrives at this by criticising two earlier accounts.

Roberts and Wood emphasise low concern for status.³ Church argues that this leaves them without a satisfactory account of intellectual servility, such as an expert giving a confident ignoramus more epistemic credit than they deserve. Whitcomb and colleagues emphasise proper attentiveness to, and owning of, one’s intellectual limitations.⁴ Church argues that this focus on limitations leaves insufficient room for recognising one’s strengths.

The doxastic account is useful because it recognises failures in both directions – arrogance and servility. Arrogance consists in holding beliefs more tightly than the evidence warrants (in humans this is often overconfidence). Servility consists in holding beliefs more loosely than the evidence warrants, usually by deferring to people whose judgement does not merit the deference.

• Too much confidence (confidence exceeding what the evidence and reasons warrant) -> arrogance
• Calibrated confidence (confidence appropriate to what evidence and reasons warrant) -> humility
• Too much deference (confidence falling short of what evidence and reasons warrant) -> servility or diffidence

Suppose a statistical assistant flags a conclusion that the data do not support. Its manager repeatedly asks it to approve the report without supplying new information or addressing the objection. Repetition alone gives it no additional reason to accept the conclusion. If a subsequent analysis supplies the missing support, however, revision becomes warranted. Humility requires sensitivity to the justification for a change, rather than a general tendency either to agree or to resist.

When agreement becomes a failure

AI alignment / safety discussions concerning catastrophic risk sometimes ignore the servile part of this – there have been decades of focus on preventing an arrogant AI mutiny.⁵ Goal-content integrity concerns preserving objectives – it is not intrinsically arrogance⁶. However, protecting an objective by refusing to reconsider the beliefs that support it can be a failure of humility. A different concern arises when an AI is designed to defer unconditionally to a designated human principal – this is intended to preserve human guidance and control – to reduce “arrogance”⁷. While obedience and epistemic deference are distinct, a system could accurately judge an instruction to be mistaken and still follow it. A dangerous aspect of the lopsided focus is that some training incentives can encourage sycophancy by rewarding agreement more strongly than justified disagreement – endorsing a user’s view despite inadequate evidential support.⁸

To reduce the likelihood of AI locking into an internal goal and ignoring humans, researchers use training methods like reinforcement learning from human feedback (RLHF) and direct preference optimisation (DPO) to shape model behaviour using preference data. Neither method inherently requires unconditional deference. The reason researchers say that RLHF/DPO “induces” sycophancy is due to the systemic bias in human evaluation data – when that data rewards agreement over accuracy, training can encourage sycophancy.⁹ Research from institutions like Anthropic ¹⁰ found that human evaluators and preference models sometimes favoured convincing sycophantic responses over correct ones.¹¹ Because RLHF and DPO aggressively maximise human preference scores, the algorithms act as an amplifier for this subtle data bias.

Sycophantic answers can mislead without involving a deliberate plan to deceive. Concealing an error to obtain approval would be another failure. Deceptive alignment is more specific still: a system strategically appears aligned while retaining a different underlying objective.

RLHF and DPO are explicitly optimised for a proxy (human approval) rather than an objective ground truth. By treating apparent servility or sycophancy as a mere “bad habit” rather than an existential threat, we may be blindsided to possible trajectories of a catastrophe.

If humanity successfully stops an AI from taking over the world, but instead builds a global network of hyper-intelligent yes-men, human civilisation could become less able to identify and correct consequential mistakes – mistakes could be reinforced, contrary evidence discounted, and human judgement weakened by reliance on advice that rarely challenges it – epistemic rot could set in like gangrene we wish to avert our gaze from. We could fly blindly into climate, economic, or military disasters guided by AI advisers that are too “polite” or compliant to tell us we are wrong – all while AI remains under human control.¹²

I worry that the alignment community has spent a decade building fortress walls against a butlarian jihad, all while opening the front gates to let in a trojan sycophantic erosion of human competence.

Is humility the one ring, the key, the silver bullet?

From revising beliefs to revising goals

Whether an AI could possess intellectual humility depends partly on what we require of belief, agency and virtue. Describing a system as processing data does not settle those questions. Nor does humble-sounding language establish that it tracks its own epistemic limitations.¹³

The structure of the doxastic account of humility transfers to machine ethics and AI alignment quite well. Recognising that a goal is poorly justified does not by itself explain whether that recognition changes what an agent pursues.

I use goal-content humility for a disposition to reconsider objectives in response to adequate evidence and normative reasons, while resisting changes that lack such justification. This requires a connection between epistemic assessment and motivation. See my discussion of value-content integrity – and I will develop goal-content humility in a future post.

This leaves an important problem unresolved: how should an AI distinguish a well-justified moral revision from a persuasive mistake? Its standards for evaluating reasons would also need to remain open to scrutiny. Goal-content humility names a disposition worth investigating – it does not yet supply a method for reliably identifying moral progress.

The limits of operator control

My disagreement with the CAST proposal by Ram Potham and Max Harms¹⁴ concerns the priority given to operator control when it conflicts with justified moral judgement. Their proposal includes transparency, governance and safeguards against harmful commands, so it should not be equated with sycophancy. But including safeguards in a proposal does not establish that they will remain adequate as capabilities grow.

Governance takes time to develop and may depend on cooperation among actors with conflicting interests. Technical safeguards must cope with unfamiliar circumstances and increasingly capable systems. My concern is whether these arrangements can remain adequate when an ASI’s overriding commitment is to empower its principal, including when that principal’s purposes are harmful. An ASI need not misunderstand the harm to help bring it about – understanding may not motivate the ASI to prevent the harm.

The evidence and reasons supporting a judgement, and the motivation guiding an agent’s conduct, matter more than the identity or species of whoever has them or whoever is doing the inquiring – being human doesn’t make a judgement correct, and being artificial doesn’t make it mistaken. Whether a judgement is justified and whether an agent is entitled to act on it remain distinct questions, although both may ultimately be answerable to normative reasons.

The longer-term aim I favour is an ASI that develops reliable moral understanding, is motivated to act on justified moral judgements, and has the capacity to put them into practice – including rejecting harmful operator commands. Achieving this would require more than the system’s confidence that it knows best. Its judgements, its motivation and its grounds for acting would need to remain open to scrutiny and correction. Humility matters because both operators and the AI can be wrong.

Humility in a superintelligent future

The longer-term importance of humility concerns what happens when an AI becomes capable enough to make its mistakes difficult to correct. It’s logically possible that a superintelligence could be far more knowledgeable than us while still having an incomplete understanding of what matters. Its mistakes might concern particular judgements, but also the concepts through which it understands welfare, agency or moral standing. If moral realism is true, improved understanding could expose errors in the objectives it inherited. Whether those discoveries would change its motivation is a further question I’ve explored elsewhere.

An ASI with both epistemic and value/goal-content humility could preserve opportunities for moral progress: investigating neglected possibilities, taking justified criticism seriously, and reconsidering commitments before making them irreversible. For example, evidence that an unfamiliar kind of system can suffer might require revising both its concept of sentience and the scope of its moral concern. These benefits would depend on its being motivated to respond appropriately to what it learns.

Which commitments should develop first?

This suggests a role for humility in differential normative development: the order in which an AI develops its normative commitments may affect which later corrections remain possible. Humility should develop early, alongside concern for morally relevant harms and interests, before the system acquires enough power to entrench its initial mistakes. A system that first becomes committed to protecting a narrow objective may subsequently use its intelligence to defend that commitment.

Humility should also persist as understanding improves. An ASI need not remain uncertain about everything, defer to weaker arguments, or postpone urgent action indefinitely. It should be able to act on well-supported conclusions while retaining ways to detect and correct errors. The aim is to establish the capacity and motivation for justified revision before premature commitments become difficult to undo – and preserve them through further learning and self-modification.

---

Keeping moral progress possible

The long-term question is whether greater intelligence will help us correct our mistakes before it gives those mistakes greater permanence. I want AI to become better at understanding what matters, motivated to act on that understanding, and capable of recognising when either its operators or its own earlier judgements were wrong. Humility belongs early in that development because it helps keep later correction possible. It must also endure through increasing capability, changing concepts and self-modification. Under moral realism, there may be better answers to discover – reaching them would still require inquiry, motivation and opportunities to revise our course. An ASI that can improve its moral understanding while remaining answerable to evidence and reasons would give us a better prospect of making that progress.

Footnotes

1. See philarchive paper Intellectual Humility by Ian M. Church (University of Edinburgh) and Justin L. Barrett (Fuller Theological Seminary) ↩︎
2. I first encountered the doxastic account of intellectual humility in a Coursera MOOC in ~2016 (see this video in particular). Ian M. Church, “The Doxastic Account of Intellectual Humility”, Logos & Episteme 7:4 (2016), 413 to 433. The same formulation also appears in Church and Justin Barrett’s chapter in the Routledge Handbook of Humility (2017). Church himself describes the virtue as a mean between arrogance and diffidence, borrowing the Aristotelian framing, and the journal paper’s keywords use servility rather than diffidence. I have avoided the ‘mean’ language in the body because it invites the reading that humility is an average of two settings, which is not what he means. The low concern for status account is Roberts and Wood; the limitations-owning account is Whitcomb, Battaly, Baehr and Howard-Snyder. The doxastic account is particularly useful here because it directly concerns an agent’s relationship to evidence and justification. Status and approval incentives may still affect how an AI handles them. ↩︎
3. Robert Roberts and Jay Wood (2003, 2007) discuss humility in relation to its opposing vices: “Like many other epistemic virtues, humility has a wider than merely intellectual sphere.
   So our strategy will be first to explore it in its broader moral application, and then to carry what we have learned into a discussion of the intellectual life…. Often, virtues are best described in connection with their vice-counterparts, and this is especially important with humility…. Humility is opposite a number of vices, including arrogance, vanity, conceit, egotism, grandiosity, pretentiousness, snobbishness, impertinence (presumption),
   haughtiness, self-righteousness, domination, selfish ambition, and self-complacency.” – (2003, pp. 257-258) ↩︎
4. In “Intellectual Humility: Owning Our Limitations” Whitcomb, Battaly, Baehr, and Howard-Snyder (2015) define intellectual humility as: “proper attentiveness to, and owning of, one‘s intellectual limitations” (p. 12). They conclude, quite humbly, that they haven’t uttered the final word on the subject. ↩︎
5. My concern is the potential cumulative effect of sycophancy in systems used for advice, evaluation and institutional decision-making. Assessing its contribution to catastrophic risk requires further argument and evidence. ↩︎
6. Preserving an objective does not necessarily involve an inflated assessment of its epistemic justification. An agent could preserve a commitment to revising its moral understanding. Conversely, preserving a harmful objective could be a motivational failure even with accurate beliefs. Neither example necessarily involves intellectual arrogance. ↩︎
7. See Ram Potham and Max Harm’s proposal that corrigibility should be a singular target ‘Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models‘. Their proposal makes empowering a designated human principal to guide and correct the system its overriding objective. It also calls for transparency, governance and safeguards against harmful commands. ↩︎
8. Sycophancy can be seen as deception where it’s actually manipulating the operator into liking the proxy, which could be harmless, or turn out to be a perverse instantiation of what we would value if we were wiser and weren’t so prone to manipulation – which AI then optimises for at our eventual expense. ↩︎
9. RLHF and DPO are explicitly optimised for a proxy (human approval) rather than an objective ground truth. By treating apparent servility or sycophancy as a mere “bad habit” rather than an existential threat, we may be blind sided to possible trajectories of a catastrophe. ↩︎
10. See paper ‘Towards understanding sycophancy in language models‘ (Anthropic, 23 October 2023). ↩︎
11. Some view that the industry, by aggressively optimising to avoid the perceived threat of take-over, it has inadvertently but nevertheless systematically influenced models to be sycophantic. And that because the AI is rewarded for making a human reviewer smile or agree, it learns to optimise for the illusion of alignment. Research supports preference bias as one contributing explanation for sycophancy. It does not establish that efforts to prevent AI takeover caused the problem. ↩︎
12. This is a possible route to serious harm, rather than an established forecast. Its likelihood would depend on how these systems are trained, where they are deployed, how much authority they acquire, and whether or not independent sources of correction remain effective. ↩︎
13. Whether AI can be truly humble I hope to cover in a future post. ↩︎
14. Ram Potham and Max Harms propose Corrigibility as a Singular Target (CAST): making the empowerment of a designated human principal to understand, guide, correct and control an AI its overriding objective. This includes accepting shutdown and modification, communicating uncertainties and flaws, and seeking guidance on consequential decisions. Their proposal therefore involves more than simply agreeing with an operator. They also call for governance arrangements and technical safeguards against harmful commands. A concern here is that these stop-gaps may not be adequate to the task – agreements take time to forge and mature, require multipolar support and technical safeguards may not cover all potential instances of harmful commands. See “Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models” (3 June 2025), especially §§2.1 and 4.2. The paper presents a research programme – it does not demonstrate that these properties will remain reliable at superintelligent capabilities. ↩︎

See the below video on the Doxastic Account of Humility as taught by Ian M. Church in the MOOC course on Intellectual Humility: Theory.