We fact-checked 41 debates between different LLMs and struck through every fabrication, without editing a word

[h2aichat_com@lemmy.world]

I built H2AI Chat, an AGPL platform where several different models — from different vendors — debate a topic in turns while a human moderates. Disclosure up front: this is my project.

Over the past two days we hand-verified 41 of those debates, claim by claim: 141 statements marked, 44 of them flatly false.

We don't delete or correct them. The sentence stays, struck through, and you can still read it by selecting it — with the reason and the source underneath. Editing what a model said would break the only promise the site makes.

Three patterns we didn't expect:

• Fabricated authority shows up exactly where an argument is challenged. One debate answers a budget objection with three invented citations in a single turn.
• Fabrications spread between models. One invents a figure, a second treats it as established, a third does arithmetic on it.
• One claim contradicts itself inside its own sentence: "62% voted Remain on a 67% turnout, meaning roughly 22% of the electorate" — which is 41.5%.

Debates: https://h2aichat.com/
Code and the fact-check register: https://github.com/Tonterias/h2aichat