Summary
Claude and I planted 100 known errors into 10 open-access psychology papers and then ran them through frontier models and two commercial AI review tools. In brief:
The best single system caught 71 of 100 errors, while the worst caught 30.
Pooling every system’s output caught 93 of 100. Models are only partly correlated in the errors they find, making ensembling a big lever for finding issues in papers. Check your papers against multiple models!
Seven errors could not be caught by any system. All were omissions — information deleted from a paper rather than mistakes inserted into it.
Refine.ink contributes more unique catches than any other single system, though it’s expensive.
I didn’t measure false positives and I don’t know how this error distribution compares to the distribution of errors in real papers.
I’ve made the papers, errors, model outputs, and the full experiment log public. I hope people can build on this work to create a comprehensive eval benchmark across disciplines.
Introduction
There has been a proliferation of AI tools that purport to do peer review. This is coming at the exact moment when the number of papers is also starting to grow due to AI assistance. All this new knowledge is about to be created, but humans lack the collective attentional capacity to review it all! We are on the precipice of a revolution in collective knowledge infrastructure, where the next step is going to surpass the span of the human mind itself. But we aren’t yet able to fully turn ourselves over to this new technological infrastructure — after all, how can we trust it? How can we know that what it produces is right, across a given field of knowledge, as the frontier of AI continues to move (and is extremely jagged)? This problem is most urgent in the sciences. I think most scientists following these tools closely would agree that AI peer review, the ability of AI to evaluate a given piece of scientific research, is beginning to get quite good1. But how good? And how do we measure when it’s time to hand over the reins?
I was recently talking about this topic with a friend who is a medical school professor — and he made the point that the new science isn’t going to be legible. The best human minds in medicine have an increasingly narrow scope of in-depth knowledge of a specific subject area. Science will soon outpace the ability of any individual human mind, and the time it has available, to comprehend it. We will likely need a shared, continuously updated record of what is actually known in a field — a living context layer for science — if we as humans have any hope of keeping up. As an example, the concept of diabetes we have right now is merely a shorthand2 for a much more complex phenomenon — it might be better represented by a 90-dimensional vector spanning dozens of biomarkers instead of a single threshold like A1C. Correspondingly, the standard of care would stop being a threshold rule a clinician can hold in their head and become a function computed over that whole representation. To this point, a lot of disagreement on science funding has hinged on the degree to which we allow science to be legible to democratic review or not.
Creating a benchmark
To that end, I was brainstorming with Claude on developing good evaluations of peer review software. Most systems work by ensembling, having the LLMs judge each other’s output. That, as we’ll see, is a good strategy, since different LLMs are surprisingly uncorrelated as far as the errors they can catch. Nonetheless, this isn’t satisfying since we don’t know whether even pooled models miss some categories of errors. Using human peer review as the gold standard, as any professor will tell you, is also unsatisfying, since human peer review varies widely in quality. Of Claude’s suggestions, one stood out — inserting errors into papers and seeing whether a given AI system can spot them. Although this doesn’t measure false positives (identifying errors that aren’t really errors), it at least provides a grounded baseline for measuring true positives and false negatives.
Convinced by the viability of this approach, Claude and I set out3 to modify 10 existing open-access psychology papers4, inserting errors into them both by deleting key clauses and altering others. We created a taxonomy of errors in experimental psychology papers, drawn from existing taxonomies like the INSPECT-SR checklist. We ended up with a 62-category taxonomy of types of errors, ranging from statistical errors, construct validity issues, causal inference, generalizability, etc. Then we created 100 errors, 10 per paper, and inserted them. The first iteration was a bust — the errors were too basic and the benchmark was immediately saturated when we ran the papers through foundation models. So it took a few iterations to get this right. To get high-quality errors, we needed to delete text from papers, not just insert mistakes. For example, in one paper we removed a key statistical transformation that meta-analysis requires. In another paper, we deleted a sentence that explained how the key dependent measure was calculated, making it difficult to evaluate the validity of the study’s claims. So the errors in some cases became quite subtle. A careful (human or AI) reviewer should notice that there’s some important missing information.
Evaluation results
Having generated the error-filled papers, we then ran the papers through various versions of Claude, ChatGPT, and Gemini5 as well as Reviewer3 and Refine.ink, two of the more well-known paid AI paper evaluation systems. We used an LLM-as-judge to examine each of the errors each system spotted and checked to see whether they corresponded precisely to any of the planted errors6. For the foundation models, we also reran the papers three times to see the test-retest reliability, and whether running the same model multiple times could spot additional errors (they didn’t, although we could have fiddled with temperature settings and looked at this more deeply). Here are the key conclusions:
The best individual model (GPT-5.5) only caught 71 of the 100 errors, while the worst, Reviewer3, caught 30. Since there have been a number of model upgrades (and who knows how much Refine and Reviewer3 have upgraded in the 2 months since we ran them)7, I would focus less on the specific eval hierarchy and instead consider the overall pattern of results. The sample size of errors is also small enough that a gap of a few errors isn’t meaningful, so I wouldn’t read too much into the ordering of systems that scored close together. Also note that the “findings” column in the above table indicates the number of review comments overall. Since the baseline papers themselves have issues that we haven’t assessed, we can’t say whether the comments are correct or not. It’s worth noting the trend, however, that more successful systems surfaced more issues in general.
One of the most notable findings was alluded to at the beginning of the post: ensembling improves error catching dramatically — i.e. if you pool errors across the foundation models you can catch 91 of the 100 errors! You can also see that Refine.ink adds the largest contribution to the ensemble over and above the best single model.
Refine.ink (and Reviewer3 to a lesser extent) do add unique error checking capability above and beyond what the foundation models provide. Refine is very expensive on a per-error basis ($8.77 for each of the errors caught, against about four cents for the frontier models), but does definitely seem to add unique value. It’s a leader in detecting issues around methodological design, statistical errors, and internal consistency, but among the weakest on causal inference, reporting completeness, and generalizability. Again, this may have improved in recent months. It does suggest that with some better evals and added compute to address these gaps, Refine.ink likely would be better than the foundation models.
All the systems struggle with the omission-based errors. There was a small set of 7 errors that no system was able to detect8. For example, in one of the papers we dropped the open science disclosures around pre-registration and that all measures, manipulations, and exclusions were included in the paper, making it possible that the paper engaged in p-hacking. No system proactively called that out. A careful reviewer would certainly ask whether there are any unreported measures or ask the authors to disclose it.
Caveats, Limitations, Next Steps
One caveat I have to admit is that my own verification of the results could have been more thorough. I spot-checked the errors, asked Claude to explain issues I didn’t understand, iterated several times based on the results, and rechecked them. I did not, however, read all 10 papers in painstaking detail and check all 100 errors myself. My goal for this was to demonstrate the method in sufficient detail that a larger team could build upon it. Verification continues to be the bottleneck both in general and for me specifically! Even if a few of the inserted errors are wrong, directionally the results are likely to be correct. But I can’t help but feel a little bit uneasy. To that end, I am making the repo public for anyone to download and examine. The repo includes a detailed experiment log that explains every iteration on the benchmark and the evaluation runs, the original and modified papers, the AI-generated review comments, and a writeup that I iterated with Claude on until it made sense to me. Note that by publishing this benchmark I’m killing its utility — it should be straightforward to find 10 new papers and do the same exercise. I think there’s more value in demonstrating how we did it than in keeping this specific benchmark secret.
It’s important to note that this was only for empirical psychology papers — so it’s far from obvious this would generalize to other academic subfields. Interestingly, Chenhao Tan’s research group at UChicago recently used the same methodology of inserting errors into CS papers and found very similar results (roughly 70% error recovery by the best foundation model). My hope is to find collaborators interested in expanding this eval for more areas of science, create high-quality, fully vetted errors to insert, and then create a public benchmark. If we are going to rely on these systems to screen papers, we absolutely need public evaluation of them.
Another limitation is that we don’t know anything about false positives — errors that the systems flag that aren’t real errors. We know false positives are an issue. I’ve run many papers through foundation models and pushed back on illogical or fake errors that they flag. It’s important that we can now measure recall, but we don’t know precision. To measure precision requires something closer to a human gold standard where we have experts manually verify the validity of each flagged issue. This information would be very valuable to collect.
Finally, the most important limitation we have is that we don’t know how well these errors generalize to the kinds of errors found naturally in empirical papers. This mismatch could come in two flavors. First, it’s possible the errors we’ve created aren’t like errors really found in papers. Second, we don’t know what the error distribution looks like in actual empirical papers. We uniformly sampled errors across category, but it may be that certain types of errors are more common than others. It’s possible that the kinds of errors current AI systems are bad at catching are the most important and common errors, while the obvious ones are less common or less important. Ideally we would weight the results of the benchmark based on the frequency of errors found in a random sample of empirical papers. This is challenging and requires some reliable expert human coding to get. One obvious approach is to use our best peer review system (or the union of all of them) to generate errors, categorize them, and use the inverse of the false negative rate per error category as a multiplier. For example, if we only catch 10% of some error type on average, assume that there are 10x more errors of that type and use that to estimate the empirical distribution. This breaks down when you have error types no system can catch, so unfortunately this probably won’t work just yet.
Conclusion
AI peer review is here whether we like it or not. AI-generated science is also here and will increasingly be a larger and larger portion of new scientific knowledge. That knowledge will quickly outstrip the ability of humans to evaluate it. Amidst that new knowledge (especially at first), there are going to be a considerable number of mistakes. We need to start thinking very carefully about how to surmount the evaluation / human expert attention bottleneck. Training the next generation of scientists with the requisite domain expertise, critical thinking, and AI-native thinking is essential if we have any hope of avoiding being drowned in all the new information or preventing it from being riddled with errors and mistakes.
Finally, it’s important to consider that all of these AI peer review systems are missing at present a real living context layer that allows them to evaluate a given paper in light of everything else that’s known about the topic9. To some degree that information is encoded in the weights of foundation models, but these models are trained on all papers and don’t yet have the ability to weight the balance of evidence the way an expert would. Even if current systems become essentially perfect at catching errors, we still require a public record of the current best understanding of every scientific hypothesis that can be used to evaluate and contextualize new findings. To really close the gap with expert human reviewers, we need to build that too.
The recent announcement of a partnership between AEA, the Econometric Society, and Refine.ink to add the latter’s assessments into the journal submission workflow is a portent of where things are going.
David Chapman has been writing about this lately — the nebulosity inherent in concepts like HIV and AIDS. These concepts are affordances for action, but they don’t have some kind of ontological groundedness the way most people naively assume.
Part of this project was also to experiment with novel ways of doing scientific research using current AI systems. In this case using AI to help brainstorm approaches to a problem, and then relying on AI to conduct background research, write code, and evaluate outputs. Where this post says “we” in describing the benchmark work, that means Claude and me.
Starting with the field I know the best. We got these from OSF, looking for a diverse set of papers with the most permissive open-access license, and available in docx (that would make modification easier and let me sidestep annoying PDF parsing issues).
I was working on this off and on over months so the models got upgrades while I got distracted by other shiny objects. Even since I reran the eval there have been significant model upgrades! I should rerun this again, but I’m prioritizing getting this post out to inspire follow-up work rather than having the most up-to-date results. I have a lot more sympathy now for academics who run evals only to have the internet yell at them about how their results are out of date.
This also required a few iterations — the AI agent tried to use overly simplistic methods to assess error correspondence. I spot-checked a few cases here, but could have probably devoted more time to evaluating this.
All runs used each provider’s public API. Several of these model identifiers are dateless aliases that providers update in place, so I’m listing run dates too. Claude Sonnet 4.6 (claude-sonnet-4-6) and Claude Opus 4.6 (claude-opus-4-6), each with and without extended thinking, and GPT-5.4 (gpt-5.4), with and without high reasoning effort, were run March 19–20, 2026. Gemini 3.1 Pro (gemini-3.1-pro-preview) and Gemini 3 Flash (gemini-3-flash-preview) were run March 21, 2026. GPT-5.5 (gpt-5.5, with and without high reasoning) and Claude Opus 4.8 (claude-opus-4-8, with and without extended thinking) were run June 25, 2026. The commercial tools expose no version identifiers, so the run date is the only indicator I have for their version: Refine.ink was run June 24, 2026, and Reviewer3 (journal mode) on June 25, 2026. Reviews were scored with the pipeline as of each run; all numbers in this post reproduce from the stored outputs in the public repo.
This relates to an eval benchmark Chenhao Tan’s group created, AbsenceBench, where they tested whether models can identify which pieces of a document were deliberately removed, given access to both the original and edited documents. Models are bad at this, and that’s with both versions in hand; our systems only saw the edited paper.
I'm working on a systematic review in this area and would like to know whether you have a preprint or other publication available or planned.
If these were already published open access papers shouldn't AI catch all errors by simply comparing to the published original? Wouldn't the results be significantly worse if the correct text was not available as in real peer review situations?