Paper Tests Microsoft Phi Model Against Human Cooperation Data

Researchers Raina Gao and Tristan Raffo posted a paper on October 21, 2026 asking a narrower question than most prior work on language models and game theory: not whether a model can play the Prisoner’s Dilemma, but whether its pattern of cooperation over many rounds looks anything like a real human’s. Earlier studies had tested LLMs against each other or against hand-picked baselines, but had skipped at least one of three things this paper adds together: a memory component so an agent can recall its partner’s past moves, an accessible open-source model rather than a closed API, and a direct comparison against recorded human cooperation data.

The team ran five open-source models, TII Falcon 7B Instruct, Google Gemma 7B, Meta Llama 3.1 8B Instruct, Microsoft Phi Mini MoE Instruct and Qwen 2.5 7B Instruct, through a standardized prompting setup across two trial types. In the fixed-partner setting, both agents kept memory of the whole interaction. In the shuffled-partner setting, a memory-enabled agent was told its opponent would change each round and was paired against a memoryless agent. Cooperation was tracked over 100 rounds, measured in 10-round increments, and checked against human baseline data from a prior published study.

The result across both setups was the same direction of error: every model’s cooperation pattern diverged from the human one, and the gap widened sharply once partners were shuffled. Human cooperation in that setting averaged 28.5%, while several of the five models cooperated more than 80% of the time, a large overshoot rather than a close match. Qwen 2.5 7B Instruct behaved almost deterministically, settling into one pattern and staying there, while Phi Mini MoE Instruct showed more variability and, for short stretches, came closer to the human rate than the other four models did.

The numbers

• 28.5%: the average human cooperation rate in the shuffled-partner trials, the benchmark the models were measured against.
• 80%: the cooperation rate several of the five models exceeded in that same shuffled setting, well above the human figure.
• 100 rounds: the length of each trial, tracked in 10-round increments to see how cooperation moved over time rather than only at the end.

The authors frame the result as evidence that open-source models have promise as stand-ins for behavioral research subjects, while being explicit that none of the five tested are yet reliable substitutes for actual human participants in a cooperation study.