Does meta-research cite retracted work?
Both reviewers returned unsound: three fatal findings and eleven major. Full review at reviews/P02.
The most concrete finding is verified and correct. The temporal classification is wrong. probes.py compares calendar years, so every same-year match falls into pre-retraction by default. Recounting: 83 strictly post, 66 strictly pre, and 41 same-year that the probe reported as pre-retraction and which are actually undetermined. Section 5 of the document below presents 107 pre-retraction as a finding that justifies the whole design, and 41 of those 107 are an artifact of comparing years. Retraction Watch carries full dates; PubMed often carries only a year, so the honest fix produces three classes and not two.
The reviewers also found, independently of each other, that the comparison arm does not exist: the document promises articles matched on journal and year, and what it actually specifies is whatever screening excludes, with no matching ratio, no replacement rule and no weighting. And that section 7’s feasibility numbers were computed over both caches without applying this study’s own eligibility criteria, so they describe a different population from the one the protocol defines.
Left exactly as submitted. A successor will be a new document.
Probes PA to PC ran on 2026-09-16 against full text already cached by P01 and P01v2, so they cost nothing to NCBI. Section 7 is written from studies/P02-citing-retracted/out/probes.json.
The screening gate is shared with P01v2: both studies draw their comparison arm from the same frame and neither runs until screening agreement is measured against the author’s labels.
No exposure-by-outcome model has been fitted. Awaiting review by two models; the second is quota-blocked until 2026-09-20.
1. The question
Citing a retracted paper as though it still stands is a documented failure of the literature, and meta-research is the field that documents it. Retracted randomized trials have been shown to go on being cited uncorrected in systematic reviews and clinical guidelines (PMID 35779825), and guidelines for low back pain have been shown to cite retracted and predatory work (PMID 40156953).
Nobody has turned that instrument on meta-research itself. A field with domain expertise in this exact failure, failing at the same rate as everyone else, would say something about how far knowing a problem gets you toward avoiding it.
2. Hypothesis
Meta-research articles cite retracted work, after retraction, at a lower rate than a matched comparison of articles in the same journals and years.
Primary null. The rate ratio is 1.
What would falsify it. A rate ratio at or above 1. The direction is stated because the alternative is the interesting one.
Prior expectation. Nothing in the corpus estimates this, so the design has no anchor and does not pretend to one.
3. Design
Cross-sectional study of reference lists. The unit of analysis is one reference; the unit of modeling is one article, with its matchable reference count as the offset.
4. Sample
The same frame as P01v2: the qualified tier B blocks with every tier A PMID removed, screened to meta-research, restricted to PMC full text. Reusing it rather than building a third population means one screening effort, one independence argument, and one set of exclusions.
Exposed arm: screened meta-research articles. Comparison arm: articles from the same journals and years that screening excluded, which is to say articles that are not meta-research. This is a better comparison than a fresh PubMed sample because it is drawn from the same query frame and differs on exactly the axis of interest.
Both arms require a reference list in PMC. Probe PA found 4948 of the cached articles carry one.
5. Measurement
Retraction Watch is retrieved as an open CSV through Crossref Labs. Probe PB found 60972 distinct DOIs whose RetractionNature is Retraction; expressions of concern and corrections are excluded because they are not retractions and counting them would inflate every arm.
A cited DOI counts as a citation of retracted work when it appears in OriginalPaperDOI.
The classification that decides the result
A reference is only a failure if the retraction preceded the citing article. Every match is classified:
| class | meaning | probe count |
|---|---|---|
| post-retraction | the retraction predates the citing article | 83 |
| pre-retraction | the citing article came first | 107 |
| undated | no usable retraction date | 0 |
The pre-retraction class is the majority. 107 of 190 matches are citations of work that had not yet been retracted, and a study that counted those as failures would report more than twice the true rate. The probe is what establishes that; it was not assumed.
A third class, an article citing a retracted paper deliberately as an example, is separated by the retract topic label and by the citation context from the surrounding sentence, and every article carrying that label is additionally reported separately.
6. Outcome
Count of post-retraction citations per article, with the article’s DOI-matchable reference count as the offset.
7. Feasibility, computed
Every number here is emitted by out/probes.json.
| quantity | value |
|---|---|
| articles with a reference list in cache | 4948 |
| total references | 268290 |
| references carrying a DOI | 222730 |
| DOI coverage | 83.0 percent |
| median references per article | 46 |
| citations of retracted work, any timing | 190 |
| articles citing at least one | 152 |
| post-retraction citations | 83 |
| articles with at least one post-retraction citation | 70 |
DOI coverage is 83.0 percent, not the 62 percent the original one-page proposal estimated from three articles. The earlier figure came from a sample too small to estimate anything and is superseded.
70 articles carry the outcome. That is enough to fit a count model and not enough to slice finely; the design commits to one primary comparison and no subgroup analyses.
A defect the probe caught
A first version computed DOI coverage as distinct DOI-like strings per article divided by the reference count and reported 1.41. A rate above 1 is impossible and is the only reason it was noticed: the plain-text fallback regex matches several times inside one reference, so the numerator was not a subset of the denominator. Coverage is now counted per reference.
8. Analysis
In R.
glm.nb(n_post_retraction ~ arm + year_c + offset(log(n_matchable_refs)), data = d)Negative binomial rather than Poisson, with the dispersion parameter reported rather than assumed. Preregistered sensitivity analyses:
- Excluding every article carrying the
retracttopic label from both arms. - Poisson with robust standard errors, to show the distributional choice is not carrying the result.
- Restricted to articles published from 2015, when Retraction Watch coverage is densest.
- Counting expressions of concern alongside retractions.
9. Threats, and what happened to each
| threat | what was done | removed or disclosed |
|---|---|---|
| Pre-retraction citations counted as failures | Classified by date; the probe shows they are the majority | Removed |
| Papers about retraction cite retracted work deliberately | Separated by topic label and reported separately | Disclosed |
| DOI coverage may differ between arms | Measured per arm before the model is fitted; the offset is biased if it differs | Disclosed |
| Retraction Watch is incomplete | A missing retraction biases both arms toward the null equally | Disclosed |
| Retraction dates are the database’s, not always the publisher’s | A late date misclassifies post as pre, in both arms | Disclosed |
| Comparison arm is defined by screening exclusion | Depends entirely on the screening gate; the study cannot run before it | Disclosed |
Reference lists in <back> only |
Articles whose references PMC does not structure are absent, not zero | Disclosed |
10. Probes
| probe | computes | goes to | what would change the design | status |
|---|---|---|---|---|
| PA | reference counts and DOI coverage | 7 | coverage below about 50 percent makes the offset unreliable | done |
| PB | raw Retraction Watch matches | 7 | fewer than about 30 matches makes the study unpowered | done |
| PC | the post and pre retraction split | 5, 7 | a post-retraction count under about 20 ends the study | done |
| PD | DOI coverage separately per arm, after screening | 9 | a differential in coverage biases the offset and needs adjustment | outstanding, needs screening |
| PE | screening agreement, shared with P01v2 probe P6 | 4 | below kappa 0.75 triggers the hand-screening fallback | outstanding, needs the author |
11. Deviations
Recorded in a DEVIATIONS section on the result page: what changed, when, why, and the estimate before and after where computable.
12. What this cannot show
- It is not causal. Meta-research authors may differ from other authors in ways unrelated to their expertise in retraction.
- A citation is not an endorsement. The design counts post-retraction citations and does not read what each one says about the work it cites. A citation flagging a paper as retracted is counted as a failure unless the
retractlabel catches it. - Retraction Watch is the ground truth and it is not. Coverage is best for recent, English-language, indexed work.