Post-publication peer-review should focus on highly influential articles

Authors: Ian Hussey & Jamie Cummins

tl;dr: 9.2% of all citations go to just 0.32% of psychology articles. To have the most impact, post-publication peer-review should focus on these influential articles.

In their recent piece in the Chronicle of Higher Education “Social Science Is Broken. Here’s How to Fix It”, Gelman & King (2025) argued that science needs “a post-publication, citation-triggered review process,” where highly cited and therefore influential articles are automatically given post-publication peer review at a certain citation threshold.

We agree in many respects, and are already doing something like this. Our bug bounty program for science, ERЯOR, is built on similar principles. We believe that:

1. Science would benefit from multiple parallel systems of verification in addition to pre-publication peer review, such as a more formalized post-publication peer review process.
2. Given finite time and resources, re-examining the most influential articles has the greatest potential to uncover important errors that might exist.

In ERЯOR, both peer reviewers and original authors are paid for the their time, with additional bonus payments made to either or both parties contingent on any errors detected. This can help provide a much-needed incentive for doing this work. You can find a description of the process and the bounties here.

Skew in citations

Citations are extremely skewed: the vast majority of published articles are cited very few times, with a tiny minority being cited a lot (Radicchi et al., 2008). What about in psychology specifically? Using the March 2025 CrossRef database, Jamie Cummins and I extracted all publications in all psychology journals published from 2015 to 2025 and their citation counts.

• The modal number of citations of psychology articles is zero.
• Receiving just a single citation puts an article in the 38th percentile.
• The median is 4. If your article has 5 citations, it is cited more often than the majority of articles, congratulations!

The top 1% of articles attract a very disproportionate number of citations. So much so that I had to make a second subplot for them to make the plot make sense:

Highly cited articles

As Gelman and King (2025) argue, this means that revisiting a tiny subset of all published articles to check them for errors could have an outsized impact on our collective knowledge.

Gelman and King (2025) estimated that, in political science, 5% of all published articles account for 28% of all citations. They suggested that a 250 citations threshold might be used as the criterion to trigger post-publication review. We at ERЯOR have been using a slightly different criterion that tries to take the recency of the publication into account: ≥30 citations per year since publication. We had been applying this and searching for qualifying papers somewhat unsystematically until now, but Gelman & King’s (2025) article prompted us to formalize this criterion and search strategy.

Of the 645,164 publications (9.2%) in psychology journals in the last decade in the CrossRef database, just 2,092 articles (0.32%) received at least 30 citations per year since publication.

Of course, others might choose different criteria, but the underlying point remains the same: reevaluating a tiny subset of all published articles has great potential for identifying the most influential errors and therefore improving the strength of the scientific knowledge base.

Systematizing recruitment to ERЯOR

Future invitations to participate in the ERЯOR bug bounty program will be randomly drawn from the list of DOIs we created using this criterion, which includes the top .32% of articles by citations.

Doing so will increase the generalizability of ERROR’s findings, although other sources of selection bias are likely to still be present. These sources of selection bias are problematic for a secondary goal of ERROR: to estimate the prevalence of errors among highly cited articles. Most notably, because ERЯOR requires that the original authors give their consent for the article to be reviewed, this introduces a self-selection bias. Nonetheless, an estimate of the prevalence of errors in highly cited psychology articles may still be possible using methods such as Multilevel-regression and Post-stratification (see Alsalti et al., 2023).

Data and code

All data and code to reproduce this search and analysis is available on the ERЯOR GitHub repository.

Appendix: Table of percentiles of citations

Some readers may prefer to see the precise number of citations that generated the plot above.