The package asserts a retraction from an exact identifier (DOI or PMID) with high confidence. For a reference that carries no identifier it falls back to title matching, and two thresholds govern that fallback:
- the fuzzy threshold
(
getOption("retraction.fuzzy_threshold"), default 0.90): a title similarity below this is not even surfaced as a possible match; - the
title_exactgate (similarity 0.985 plus an exact publication year and a matching first author): only a match clearing this gate is asserted as retracted; everything else is reported as “possible” for the user to verify.
This article reports a calibration of those two numbers against a
labeled corpus, so they are evidence-based rather than guessed. The
labeled corpus ships with the package in
inst/extdata/calibration_corpus.csv, and the two scripts
that build and analyze it (calibration_corpus.R,
calibration_analysis.R) live in the data-raw/
directory of the source
repository, so the study is reproducible from a repository
checkout.
The labeled corpus
599 references in three groups:
- 200 exact-title retracted — records sampled from the Retraction Watch corpus, cited with their exact title (as a well-formatted bibliography would).
- 200 perturbed retracted — the same records with a realistic citation variation (about 15% of title words dropped, lower-cased), to probe how the thresholds behave on imperfect titles.
- 199 clean — non-retracted articles sampled from OpenAlex.
Every reference is matched by title, year, and author only (the DOI is withheld), which is exactly the hard case the thresholds govern.
Result 1: the assertion gate never false-accuses
At the title_exact gate (0.985 + year + first
author):
| Metric | Value |
|---|---|
| Precision | 1.000 |
| Recall | 0.532 |
| Clean references false-flagged | 0 / 199 (0.000) |
Flag rate by group: exact-title retracted 1.000, perturbed retracted 0.065, clean 0.000.
Reading: no clean reference was ever asserted as retracted (zero false accusations), and every exact-title retracted reference was recovered. The perturbed titles almost never clear the gate (0.065) — by design they fall to “possible” rather than being asserted, which is the conservative behavior we want. The overall recall of 0.532 is dominated by the perturbed group; on citations that reproduce the title faithfully, recall is 1.0.
Result 2: 0.90 is the empirical sweet spot for surfacing
Sweeping the fuzzy threshold and measuring how often a retracted reference is surfaced (as flagged or possible) versus how often a clean reference is wrongly surfaced:
| Threshold | Retracted surfaced | Clean surfaced | Precision |
|---|---|---|---|
| 0.84 | 0.980 | 0.774 | 0.718 |
| 0.86 | 0.958 | 0.437 | 0.815 |
| 0.88 | 0.945 | 0.035 | 0.982 |
| 0.90 | 0.885 | 0.000 | 1.000 |
| 0.92 | 0.790 | 0.000 | 1.000 |
| 0.94 | 0.688 | 0.000 | 1.000 |
0.90 is the lowest threshold at which no clean reference is surfaced while still recovering 88.5% of retracted references. Below 0.88 the clean-surfacing rate climbs steeply (43.7% at 0.86), which would bury real hits in false positives. Above 0.90 precision stays perfect but recall falls with no benefit.
Conclusion
The defaults are validated by this corpus:
-
0.985
title_exactgate: perfect precision, zero false accusations — the right posture for a tool that could otherwise mislabel a clean paper. - 0.90 fuzzy threshold: the operating point where clean false-positives reach zero while retaining high recall.
These are heuristics, not calibrated probabilities, and the corpus is modest (599 references); the numbers should be revisited as the corpus grows and across non-English titles. But they show the current thresholds are conservative in the direction that matters: the package prefers “possible, please verify” over a false assertion.
Reproducing
The two scripts below are in the data-raw/ directory of
the source repository
(they are not part of the installed package). Run them from a repository
checkout:
retraction_sync() # local corpus
source("data-raw/calibration_corpus.R") # rebuild the labeled set (online)
source("data-raw/calibration_analysis.R") # recompute the tables above