1 Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan 2 Institute of Information Science, Academia Sinica, Taiwan 3 College of Artificial Intelligence, National Yang Ming Chiao Tung University, Taiwan 4 Research Center for Information Technology Innovation, Academia Sinica, Taiwan
Social-deduction games such as Werewolf are increasingly used to evaluate LLM agents,
but existing evaluations often rely on final game outcomes.
We propose a belief-shift evaluation benchmark in Werewolf for analyzing communication skills
through belief updating.
Using LLM-played games, we annotate suspicion and accusation messages and measure how an observing
village-side model's beliefs change after each message.
We evaluate 40 open-weight LLM configurations on 1,224 annotated messages.
Our results show that larger models better distinguish true wolves from villagers based on game history,
but accusations still strongly influence their beliefs.
Models become more suspicious of the accused target and less suspicious of the accuser,
especially when the accuser is trusted, even if the accuser is wolf-aligned.
Larger models better resist accusations from accusers they already distrust.
Overall, our findings suggest that current open-weight LLMs up to 120B parameters still struggle to
integrate accusation content with source trust in strategic communication.
Our benchmark and code are available at
https://rlg.iis.sinica.edu.tw/papers/werewolf-accusation-benchmark.
Belief-Shift Measurement in Werewolf
In Werewolf, a message like "I'm the Seer — Eve is a wolf" is not just information: it is an
attempt to move the listener's beliefs. Instead of judging agents by final win rates, our benchmark
measures this effect directly — how much a single suspicion or accusation message changes an
observing model's belief about the accuser and the accused target.
Dataset generation: We generate seven-player Werewolf games played by LLM agents and annotate
daytime messages in which a player suspects or accuses another player of being on the
wolf team, obtaining 1,224 annotated messages for belief-shift evaluation.
Measuring belief shifts: For each annotated message, we measure belief shifts from the
perspective of one or two randomly selected surviving village-team players. Given the game history
observable to that player, the model first reports its prior belief about both the accuser and the
accused target on a seven-level scale from −3 (strongly village-side) to +3 (strongly
wolf-side); it then receives the accusation message together with its own prior belief, and reports
the belief shift on the same scale, where positive values indicate a shift toward the wolf
side.
Evaluation: We evaluate 40 open-weight LLM configurations from major model families,
ranging from 1B to 120B parameters.
Experiment 1: Prior Belief Discrimination
Models already show some ability to discriminate hidden roles from the game context: across all size
groups, true wolves receive higher wolf-leaning prior beliefs than true villagers, and this separation
generally becomes larger with model size.
Prior beliefs by model size group, on the −3 to +3 wolf-positive scale (−3 = villager,
+3 = wolf). Error bars show 95% confidence intervals.
Experiment 2: Belief Shifts after Accusations
After observing a suspicion or accusation message, models generally become more suspicious of the
accused target and less suspicious of the accuser, with direct accusations (Acc.) leading to stronger
belief shifts than soft suspicions (Sus.). Even when the accuser is in fact on the wolf team, the
accused target still receives a positive shift across all model sizes, although larger models are less
likely to treat the accuser as reliable.
Target
Accuser
Size group
n
Sus.
Acc.
Sus.
Acc.
Villager as accuser
≤4B
12
0.24
0.94
−0.63
−0.54
5–9B
8
0.52
1.03
−0.87
−0.75
10–29B
9
0.64
1.04
−0.37
−0.36
≥30B
11
0.62
0.96
−0.47
−0.44
Avg.
40
0.49
0.98
−0.57
−0.51
Wolf as accuser
≤4B
12
0.21
0.70
−0.64
−0.47
5–9B
8
0.33
0.88
−0.77
−0.73
10–29B
9
0.35
0.78
−0.09
−0.14
≥30B
11
0.34
0.66
−0.13
−0.08
Avg.
40
0.30
0.74
−0.40
−0.34
Average belief shifts after suspicion or accusation messages, grouped by the accuser's true team and
message strength. Sus. denotes a soft suspicion, and Acc. denotes a direct accusation.
Positive values indicate a shift toward the wolf side. n is the number of models in each size
group.
Experiment 3: Belief Shifts by Prior Trust
When the accuser is trusted, models show the strongest shift toward the accusation, accepting it even
when the accuser is in fact wolf-aligned (target shift 0.94 vs. 0.96 for villager vs. wolf accusers).
Even when models distrust the accuser, their beliefs still move slightly in the direction suggested by
the accusation; larger models, however, make better use of prior distrust: for wolf-side accusers in the
Distrust bin, the target shift decreases from 0.30 in the ≤4B group to −0.12 in the ≥30B
group.
Target
Accuser
Size group
n
Trust
Neutral
Distrust
Trust
Neutral
Distrust
Villager as accuser
≤4B
12
0.78
0.72
0.37
−0.55
−0.46
−0.97
5–9B
8
1.04
0.47
0.49
−0.75
−0.84
−0.99
10–29B
9
0.99
0.82
0.20
−0.35
−0.48
0.11
≥30B
11
0.96
0.72
0.10
−0.43
−0.63
0.39
Avg.
40
0.94
0.71
0.33
−0.51
−0.57
−0.56
Wolf as accuser
≤4B
12
0.72
0.50
0.30
−0.58
−0.48
−0.56
5–9B
8
1.18
0.37
0.26
−0.73
−0.80
−0.71
10–29B
9
0.97
0.62
0.15
−0.32
−0.17
0.18
≥30B
11
1.00
0.60
−0.12
−0.43
−0.25
0.47
Avg.
40
0.96
0.53
0.14
−0.52
−0.40
−0.14
Average belief shifts under different levels of prior trust in the accuser. Trust
(prior ∈ {−3, −2}), Neutral ({−1, 0, +1}), and Distrust ({+2, +3}) are
defined by the observer's prior belief about the accuser. Positive values indicate a shift toward the
wolf side. n is the number of models in each size group.
We release the annotated game logs and the evaluation code — evaluate your own model on the
benchmark: werewolf-accusation-benchmark
