Paper Finds Self-Rewarded Moral Argument Training Boosts Consistency, Not Validity

Researchers behind a new preprint posted to arXiv on October 21, 2026 set out to test whether training a language model to argue well about moral dilemmas would also make its judgments better. The premise sounds reasonable: a model that can construct a stronger case for a position might also be more likely to land on the right one. The study finds the opposite split holds instead. Argument quality training made the model’s verdicts far more stable and consistent, but it did not make those verdicts more sound by the measures the authors used to check validity.

The method is a self-judged tournament. A reinforcement learning policy, built from a state-of-the-art language model, generates competing moral arguments and then judges those arguments against each other for quality, with its own reasoning steps hidden from the judging process so the policy cannot simply reward arguments that match its existing answer. To source real dilemmas rather than invented toy cases, the team pulled conflicts from classic novels and ran each one through a logic solver to confirm it actually represented a genuine moral conflict rather than a false dichotomy.

The team found their self-judge rated argument quality in a way that lined up with human annotators, and that the trained policy began adopting the style of language that earned it the highest reward. Tested against three external benchmarks for moral reasoning, the policy became markedly more decisive and stopped flipping its answer depending on the order options were presented in. But the paper is explicit that two known failure modes, sycophancy toward a suggested framing and sensitivity to how a question is reworded, remained largely unaddressed. Two follow-up ablation tests, run with synthetic dilemmas and with the model’s reasoning left unmasked during judging, reproduced the ordering-consistency gains but showed a weaker shift in language style and, notably, higher sycophancy than the main setup.

The numbers

• The paper reports 20 pages, 10 figures and 9 tables backing its findings.
• Three external benchmarks, MoralChoice, ELEPHANT and MORABLES, were used to test the trained policy.
• Two ablation studies were run, one with synthetic dilemmas and one with unmasked thinking during judging.

The authors’ own framing is blunt: consistency is not validity. A policy can be optimized to stop contradicting itself across rephrasings of the same question without becoming any more trustworthy in what it actually decides, a distinction that matters wherever a model’s steadiness is mistaken for its being correct.