Review of P03: blocked
Review of Is bad news cited more?.
A reviewer produced a fatal finding backed by a verbatim quote and no reviewer rebutted it. Under the rule, evidence outranks votes, so this outcome does not depend on the tally.
Adjudicated by build/audit/adjudicate.py at rule version 1. Coverage is complete: chatgpt, grok returned a verdict out of a roster of chatgpt, grok.
1 fatal, 17 major, 9 minor. 18 of the severe findings carry a verbatim quote.
Votes
| reviewer | vote | confidence |
|---|---|---|
chatgpt |
unsound | 0.99 |
grok |
unsound | 0.88 |
What each reviewer said
chatgpt
Read-only checks reproduced the citation summaries for the 1,790 stored rows, but uncovered duplicate article identities, version mismatches, and a false recent-publication percentage. The classifier gate can pass despite complete failure to identify adequate conclusions. The proposed model also omits the promised journal adjustment, and the falsification rule misinterprets confidence intervals. These are repairable protocol defects. Classification and detectable-effect estimates remain unavailable, as disclosed. Literature searches found related precedents, but no verified study testing this exact within-meta-research contrast.
grok
Citation counts in section 5 match data/lit/openalex-records-summary.json and studies/_shared/openalex_probe.py: median 8, mean 36.29, 90th percentile 75, maximum 2937, 284 zeros, 1790 of 1805 PMIDs. The frozen snapshot exists. Direction has not been crossed with citations, so this is not a look at the hypothesis of the P01v2 kind. The kappa stop is written before any labels. Those parts hold. The primary analysis does not. Section 2’s single incidence rate ratio cannot be read from a four-level direction variable whose mixed and not-applicable levels have no stated fate in the glm.nb. topic_block and journal_size are not defined, and 1009 of 1830 labelled records carry two or more topic labels. The 76 percent figure used to keep 2025 and 2026 papers is the 2020-onward share (1375 of 1805 PMIDs), not the 2025-2026 share (484 of 1805, 26.8 percent). Of 284 zeros, 221 are from 2025-2026, so most zeros are papers with almost no time at risk, not papers nobody used. The citation tail that motivates quantile regression includes phrase-matched methods reviews, science-of-science surveys, and a clinical meta-analysis, not only verdicts that a practice is inadequate. I did not find a published study that already runs this exact contrast on meta-research articles. Duyx et al. 2017 (PMID 28603008) is the citation-bias review this question belongs to and is not cited. Serra-Garcia and Gneezy 2021 (PMID 34020944) is related and studies a different population.
Fatal findings
F1. The primary null is a single incidence rate ratio for conclusion direction. The exposure instrument in section 6 has fou
Reviewer grok · Section 2. Hypothesis
The primary null is a single incidence rate ratio for conclusion direction. The exposure instrument in section 6 has four labels (inadequate, adequate, mixed, not applicable) and section 7 enters direction as one term in glm.nb. A four-level factor in that model yields three coefficients, not one IRR. Mixed and not applicable have no stated coding, reference level, or exclusion rule. Not applicable is not a spare bin: the unscreened tier A query admits methods papers, science-of-science reviews, and clinical articles that only use a query phrase, so a large share of the 1790 rows may be not applicable. Until those rows are dropped or separately modeled, the coefficient named in section 2 does not exist, and probe PC cannot compute a detectable rate ratio for the contrast the hypothesis actually states.
The incidence rate ratio for conclusion direction is 1.
What would fix it. Define the primary estimand as the negative-binomial IRR for inadequate versus adequate. Exclude mixed and not applicable from that model and report their counts. Name the reference level. If mixed is retained, specify the dummy coding and which coefficient is the primary test. Recompute PC on the inadequate-versus-adequate split only, after the 300 labels exist.
Major findings
M1. Overall four-category kappa does not validate the primary inadequate-versus-adequate comparison. Computed counterexample
Reviewer chatgpt · Section 6. Exposure, and the gate
Overall four-category kappa does not validate the primary inadequate-versus-adequate comparison. Computed counterexample: among 300 reference labels, let 210 be inadequate, 30 adequate, 30 mixed, and 30 not applicable. A classifier that labels every adequate article inadequate and classifies everything else correctly achieves kappa 0.762, exceeding the gate despite zero sensitivity for adequate conclusions. A single overall misclassification rate also cannot characterize these asymmetric errors.
If neither model reaches kappa 0.70 against the author’s labels, the study stops and the negative methodological result is published.
What would fix it. Require prespecified class-specific sensitivity and precision for adequate and inadequate conclusions, with uncertainty and sufficient validation examples in both classes. Publish the complete confusion matrix, account for stratified sampling, and propagate category-specific classification uncertainty into the analysis.
M2. The reference standard lacks an operational coding rubric. No rule distinguishes negative background rhetoric from the f
Reviewer chatgpt · Section 6. Exposure, and the gate
The reference standard lacks an operational coding rubric. No rule distinguishes negative background rhetoric from the focal conclusion, improvement from adequate absolute practice, or multiple conflicting practice assessments from one overall verdict. Author-model agreement on the same abstract measures agreement with one person’s interpretation, not whether the instrument recovers the article’s conclusion. Blinding to citation counts is also unspecified.
The exposure is the direction of the article’s own conclusion, and no rule extracts it from an abstract.
What would fix it. Freeze a coding manual with decision rules and examples before labeling. Blind reference coding to citation outcomes, independently double-code and adjudicate a validation subset, and check abstract labels against full text. Alternatively, explicitly restrict the exposure and claims to evaluative framing in abstracts. Freeze model versions, prompts, and the primary-classifier selection rule before validation.
M3. The retrieval script counts OpenAlex rows as resolved articles without enforcing one article per PMID. The frozen file c
Reviewer chatgpt · Section 4. Sample; 5. Outcome
The retrieval script counts OpenAlex rows as resolved articles without enforcing one article per PMID. The frozen file contains 1,790 work IDs but only 1,789 distinct PMIDs, leaving 16 unmatched PMIDs and a unique-PMID match rate of 99.1%. PMID 42018420 appears as W4417119991 and W7155183362 with the same DOI but citation counts 3 and 0. Identity problems extend beyond duplication: PMID 33315211 is a correction whose corpus DOI differs from the original-article DOI returned by OpenAlex; PMIDs 41992899 and 41932597 resolve to preprint metadata. The reported distribution therefore describes returned database objects before article/version reconciliation.
The tier A corpus, all 1805 records carrying a PMID. 1790 resolve in OpenAlex.
What would fix it. Create and validate an explicit PMID-to-work linkage table. Resolve duplicates, corrections, and preprint/journal versions under a prespecified article-unit policy; investigate citation consolidation without blindly summing potentially overlapping citations. Recompute sample counts and outcome summaries after reconciliation.
M4. The nominal sample cannot all enter the proposed abstract classifier. Of 1,805 PMID records, 112 lack abstracts; 108 of
Reviewer chatgpt · Section 4. Sample; 6. Exposure, and the gate
The nominal sample cannot all enter the proposed abstract classifier. Of 1,805 PMID records, 112 lack abstracts; 108 of the 1,789 uniquely matched records lack abstracts, leaving 1,681 matched articles with available abstracts before other exclusions. The corpus is also unscreened: its publication types include comments, editorials, and 17 published errata. No rule distinguishes missing exposure information from genuinely not-applicable conclusions or specifies which publication types belong in the comparison. These decisions affect both eligibility and citation distributions.
A stratified random sample of 300 abstracts is classified by the author into inadequate, adequate, mixed, or not applicable. This is the reference standard.
What would fix it. Prespecify eligible article types and handling of absent or uninformative abstracts before classification. Retrieve full text where appropriate; otherwise record missing exposure separately from not applicable. Publish an eligibility flow and base power calculations on the resulting analyzable comparison groups.
M5. The stated 76% is contradicted by the frozen corpus. In data/lit/records.jsonl, 254 PMID records date from 2025 and 230
Reviewer chatgpt · Section 4. Sample
The stated 76% is contradicted by the frozen corpus. In data/lit/records.jsonl, 254 PMID records date from 2025 and 230 from 2026: 484/1,805, or 26.8%. Using OpenAlex publication years also gives approximately 27%, not 76%. This false number supplies the stated justification for retaining immature citation histories.
Articles from 2025 and 2026 are retained and age is modeled, not trimmed; excluding recent work would drop 76 percent of the corpus.
What would fix it. Replace 76% with a reproducibly computed proportion and identify the denominator and publication-date source. Reassess the citation-window design using the correct age distribution; retaining recent articles remains possible, but this justification does not support it.
M6. An interval containing 1 neither falsifies the null IRR = 1 nor establishes absence of a positive association. An interv
Reviewer chatgpt · Section 2. Hypothesis; 9. Probes
An interval containing 1 neither falsifies the null IRR = 1 nor establishes absence of a positive association. An interval such as 0.5–3 contains 1 while remaining compatible with a threefold citation advantage. Because detectable-effect calculations remain outstanding, the proposed rule could declare the substantive hypothesis falsified precisely when the analysis is uninformative.
Primary null. The incidence rate ratio for conclusion direction is 1.
What would falsify it. An interval containing 1.
What would fix it. Specify confidence level and distinguish rejection, non-rejection, and evidence excluding a meaningful citation advantage. Define a smallest meaningful effect and an appropriate interval or equivalence criterion. Calculate precision or power using eligible group sizes, dispersion, covariate overlap, and classification error before interpreting a null result.
M7. The hypothesis promises adjustment for journal, but the formula adjusts for journal_size. Journals of equal size can dif
Reviewer chatgpt · Section 2. Hypothesis; 7. Analysis
The hypothesis promises adjustment for journal, but the formula adjusts for journal_size. Journals of equal size can differ substantially in visibility, readership, editorial selection, and citation behavior. A size covariate neither supplies journal effects nor accounts for within-journal dependence. Its definition and measurement date are also absent.
glm.nb(cited_by_count ~ direction + poly(years_since_pub, 2) + topic_block + journal_size, data = d)
What would fix it. Specify an actual journal adjustment, such as justified journal effects with suitable uncertainty estimation, and assess within-journal exposure overlap. If only journal size is intended, define its construction and revise the hypothesis and interpretation to acknowledge that journal identity remains uncontrolled.
M8. Topic adjustment is neither operationalized nor checked for support. Among the 1,789 uniquely matched records, the exist
Reviewer chatgpt · Section 7. Analysis; 8. Threats, and what happened to each
Topic adjustment is neither operationalized nor checked for support. Among the 1,789 uniquely matched records, the existing label file gives 797 no substantive topic label beyond the generic metares label, while 380 have multiple substantive labels. P03 never defines how these become topic_block. Moreover, where topic determines direction, the adjusted contrast requires extrapolation or is unidentified within that topic. Presenting adjusted and unadjusted estimates does not resolve this problem.
Some topics almost never produce an adequate verdict, so adjusting for topic may absorb the effect. Reported adjusted and unadjusted, with the difference discussed rather than resolved
What would fix it. Define and validate topic coding independently of conclusion direction, including unlabeled and multilabel records. Prespecify direction-by-topic overlap checks and handling of unsupported strata. State whether the target is a within-topic association or a broader descriptive difference, and restrict interpretation accordingly.
M9. The protocol keeps 2025 and 2026 articles because excluding them would drop 76 percent of the corpus. That number is fal
Reviewer grok · Section 4. Sample
The protocol keeps 2025 and 2026 articles because excluding them would drop 76 percent of the corpus. That number is false for the window named in the sentence. Among the 1805 PMID records, 484 are from 2025-2026 (26.8 percent). The 76 percent figure is the 2020-onward share (1375 of 1805; 1396 of 1830 in MAP.md). The same error appears in the OpenAlex matches: 482 of 1790 are from 2025-2026 (26.9 percent), 1369 of 1790 from 2020 onward (76.5 percent). The false percentage is what justifies keeping 223 matched 2026 papers whose median citation count is 0.
Articles from 2025 and 2026 are retained and age is modeled, not trimmed; excluding recent work would drop 76 percent of the corpus.
What would fix it. Replace 76 percent with the computed 2025-2026 share (about 27 percent). If the scientific reason to keep brand-new papers is not that share, say the real reason. If 2026 papers are kept, treat their zeros as time-at-risk, not as unused papers, and recompute the section 5 distribution with and without them.
M10. Zeros are retained because a zero means nobody used the paper. On the frozen OpenAlex snapshot that sentence is mostly f
Reviewer grok · Section 5. Outcome
Zeros are retained because a zero means nobody used the paper. On the frozen OpenAlex snapshot that sentence is mostly false. 161 of 223 matched 2026 papers have zero citations, and 60 of 259 matched 2025 papers do. That is 221 of the 284 zeros (77.8 percent). For those rows a zero is the expected count after a few months of indexing, not evidence of non-use. Dropping zeros would still condition on the outcome for older papers; keeping 2026 zeros folds a pile of structural zeros into the negative-binomial dispersion. The justification in the threats table (Removed) therefore overstates what was fixed.
284 articles have zero citations. They are retained; a zero is informative about a paper nobody used and dropping them would condition on the outcome.
What would fix it. Split zeros by years since publication. Keep old zeros as informative. For papers from 2025-2026, either exclude them from the primary count model, use an explicit time offset or hurdle on papers with at least one full year of follow-up, or model time to first citation. Do not describe 2026 zeros as papers nobody used.
M11. Section 5 computes the citation distribution on all 1790 matches and then says the incentive argument lives in the tail,
Reviewer grok · Section 5. Outcome
Section 5 computes the citation distribution on all 1790 matches and then says the incentive argument lives in the tail, which is why quantile regression is fitted. That tail is not a tail of verdicts about research practice. The maximum of 2937 is Deeks et al. 2003 (PMID 14499048), an HTA methods review that matched because the abstract contains meta-epidemiological techniques. Also in the top 15: Fortunato et al. 2018 Science of science (phrase match); Roberts et al. 2020 on racial inequality in psychology (mentions metascience); Simons et al. 2017 COG (keyword meta-science); Huang et al. 2020 on gender and scientific careers (science of science); Fazel et al. 2012 on violence in psychosis (PMID 23418482), a clinical systematic review whose abstract says a novel meta-epidemiological approach was used to group similar risk. Wood 2008 and Smaldino 2016 are on-target and also sit in that tail. The mixture was visible in the frozen file when the tail analysis was written, and before any not-applicable label exists to remove the off-target rows.
The incentive argument lives in the tail and a mean effect would not find it.
What would fix it. Do not choose quantile regression from the unscreened citation distribution. After the 300 labels, recompute the citation distribution inside inadequate, adequate, mixed, and not applicable. If the upper tail is mostly not applicable, drop the claim that the incentive lives in that tail, and make the primary estimand a contrast among papers that actually issue a verdict.
M12. The same paragraph that specifies glm.nb as the primary model says a mean effect would not find the incentive. Negative
Reviewer grok · Section 7. Analysis
The same paragraph that specifies glm.nb as the primary model says a mean effect would not find the incentive. Negative binomial regression with a log link is a model for the conditional mean. With right-skewed citations the mean is already tail-sensitive (mean 36.29 versus median 8). If the scientific claim is a shift only in the upper quantiles, the primary estimand is the 90th-percentile contrast, not the NB IRR. As written, a null on the primary model could be read as no incentive while the text already says that model is the wrong place to look.
The incentive argument lives in the tail and a mean effect would not find it.
What would fix it. Pick one primary estimand and stick to it. If the hypothesis is a mean count ratio after age, journal, and topic adjustment, drop the sentence that a mean effect would not find it. If the hypothesis is a tail incentive, make the 90th-percentile quantile regression primary and treat glm.nb as secondary.
M13. The hypothesis says the comparison is adjusted for journal. The model adjusts for journal_size, which is never defined.
Reviewer grok · Section 7. Analysis
The hypothesis says the comparison is adjusted for journal. The model adjusts for journal_size, which is never defined. OpenAlex probe fields do not include a venue-size or impact-factor series, so journal_size is not a computed variable in the frozen snapshot. If it means count of corpus papers per journal, 428 of 639 journals in the PMID sample are singletons and the covariate is mostly 1. If it means journal identity, 639 levels on 1790 rows is not the formula as written. topic_block is likewise undefined. Labels in records-labelled.jsonl are multi-label: 1009 of 1830 records carry two or more topics, 11 carry none. A single factor cannot be formed without a rule that is not in the protocol. The fitted coefficient on direction is therefore not the adjusted IRR described in section 2.
glm.nb(cited_by_count ~ direction + poly(years_since_pub, 2) + topic_block + journal_size,
What would fix it. Define journal_size as a named, dated variable (for example OpenAlex source citedness, or 2025 Scopus CiteScore) retrieved into the snapshot. If the adjustment is journal identity, say so and use a random intercept or a collapsed size band; do not claim journal adjustment for a size proxy. Define topic_block as a mutually exclusive rule (primary label, or a pre-specified hierarchy) and report how many records are multi-label under that rule. Fit the unadjusted model as already promised in section 8.
M14. What would falsify the hypothesis is an interval containing 1. No interval, alpha, or smallest interesting rate ratio is
Reviewer grok · Section 2. Hypothesis
What would falsify the hypothesis is an interval containing 1. No interval, alpha, or smallest interesting rate ratio is named. Probe PC, which would say what rate ratio the realized split can detect, is outstanding and depends on the exposure split that does not yet exist. A wide interval containing 1 is non-rejection, not falsification of a directional claim that inadequate papers accrue more citations. The interesting alternative is IRR greater than 1; an interval such as 0.9 to 2.5 contains 1 and still includes a large incentive. This is the same class of null overclaim P01v2 made with a computed power grid; here the grid has not even been computed.
An interval containing 1.
What would fix it. Name a 95 percent CI as the interval. State a smallest IRR that would still matter. Falsify the directional claim only if the CI lies entirely below 1, or run an equivalence test against the margin PC says is detectable. Do not treat a CI that contains 1 as a negative methodological result until PC is in the protocol.
M15. The protocol frames the question as publication bias moved to the citation stage and does not cite the citation-bias lit
Reviewer grok · Section 1. The question
The protocol frames the question as publication bias moved to the citation stage and does not cite the citation-bias literature. Duyx, Urlings, Swaen, Bouter, and Zeegers, Journal of Clinical Epidemiology 2017, PMID 28603008, reviewed 52 studies and found statistically significant articles cited 1.6 times as often (95 percent CI 1.3 to 1.8) and articles whose authors concluded support for their hypothesis cited 2.7 times as often (CI 2.0 to 3.7). Jannot, Agoritsas, Gayet-Ageron, and Perneger, Journal of Clinical Epidemiology 2013, PMID 23257169, found significant studies cited about twice as often, with much of the association running through journal impact factor. That literature says positive and hypothesis-supporting results are cited more, not less. In meta-research, concluding that a practice is inadequate is often the result the authors sought, so inadequate may be the field’s positive finding. If so, P03 is a population-specific test of ordinary citation bias, not an inverted publication-bias mechanism. Serra-Garcia and Gneezy, Science Advances 2021, PMID 34020944, found nonreplicable original papers cited more than replicable ones; that is a different population (original findings, not meta-research conclusions) and is also uncited. I did not find a study that already classifies meta-research conclusions as inadequate versus adequate and models OpenAlex or Web of Science counts on that split.
This is publication bias asked about the field that studies publication bias, and asked at the citation stage rather than at acceptance.
What would fix it. Cite Duyx 2017 and Jannot 2013 as citation bias, not publication bias at the citation stage. State whether inadequate is coded as bad news or as support for a problem-finding hypothesis. Distinguish Serra-Garcia and Gneezy 2021 as related work on a different unit. Keep the claim of novelty only for the meta-research-conclusion contrast, which I could not find already done.
Minor findings
M1. years_since_pub is undefined. Both frozen inputs retain publication year rather than a complete publication date, and Pu
Reviewer chatgpt · Section 7. Analysis
years_since_pub is undefined. Both frozen inputs retain publication year rather than a complete publication date, and PubMed-derived versus OpenAlex years disagree for 60 matched rows. A quadratic cannot repair inconsistent date definitions or recover within-year citation opportunity, which matters especially for recent articles.
Age enters as a quadratic because citation accrual is not linear in time and the corpus spans 1965 to 2026.
What would fix it. Specify the publication event and date source, reconcile discrepancies, and freeze sufficient date precision to calculate elapsed time. Add a prespecified age-model or mature-publication-window sensitivity analysis.
M2. The claim that a mean effect would miss a tail effect is mathematically false. Increasing citations only among the highe
Reviewer chatgpt · Section 5. Outcome; 7. Analysis
The claim that a mean effect would miss a tail effect is mathematically false. Increasing citations only among the highest-cited 5% raises the mean while potentially leaving both the 75th and 90th percentiles unchanged. The proposed quantile regressions estimate particular conditional quantiles, not citation concentration or rewards accruing to the extreme tail.
The incentive argument lives in the tail and a mean effect would not find it.
What would fix it. Define the intended tail estimand. Retain quantile regression if those conditional percentiles answer the question, but remove the false claim about means. If extreme citation concentration is the target, prespecify a measure that actually captures it.
M3. An OpenAlex resolution percentage is not a representativeness test. The repository’s search protocol selects articles us
Reviewer chatgpt · Section 4. Sample; 9. Probes
An OpenAlex resolution percentage is not a representativeness test. The repository’s search protocol selects articles using self-identifying meta-research vocabulary in PubMed/PMC and explicitly excludes much older work lacking that vocabulary. Even complete linkage would not establish coverage of the field. Conversely, less than 90% linkage need not bias estimates if missingness is unrelated to the relevant variables.
a match rate below about 90 percent makes the sample unrepresentative
What would fix it. Restrict the target population and claims to the defined retrieval frame unless broader coverage is assessed. Replace the blanket 90% interpretation with a prespecified audit of unmatched records and plausible selection mechanisms.
M4. Zero indexed citations does not mean nobody used a paper. It can reflect limited follow-up, incomplete reference extract
Reviewer chatgpt · Section 5. Outcome
Zero indexed citations does not mean nobody used a paper. It can reflect limited follow-up, incomplete reference extraction or matching, database coverage, or use without citation. OpenAlex documents these citation-linkage limitations. OpenAlex citation documentation. Retaining zeros is appropriate; the interpretation assigned to them is not.
284 articles have zero citations. They are retained; a zero is informative about a paper nobody used and dropping them would condition on the outcome.
What would fix it. Describe these as articles with no citations recorded by OpenAlex at the snapshot date. Remove the claim about nobody using them and distinguish recorded citations from research use.
M5. The status explicitly says Not registered, while Section 7 calls analyses Preregistered. A dated proposal and an actual
Reviewer chatgpt · Section Status callout; 7. Analysis
The status explicitly says Not registered, while Section 7 calls analyses Preregistered. A dated proposal and an actual preregistration are different provenance claims. The available history supports prior specification of the kappa threshold, but does not establish registration.
Preregistered alongside: quantile regression at the 75th and 90th percentiles,
What would fix it. Use planned or prespecified until registration exists. Then provide its identifier, timestamp, frozen version, and a clear account of the citation data inspected before registration.
M6. The promised self-citation secondary analysis cannot be reproduced from the frozen file alone. The retrieval SELECT reta
Reviewer chatgpt · Section 8. Threats, and what happened to each
The promised self-citation secondary analysis cannot be reproduced from the frozen file alone. The retrieval SELECT retains target-article authorships and aggregate citation counts, but not the identities and authorships of citing works. Fetching those later would use a different citation graph from the primary snapshot.
Self-citation and field size | Not in the primary model; OpenAlex exposes the data and it is a stated secondary | Disclosed |
What would fix it. Define self-citation and archive the citing-work identities and authorships needed for the same citation snapshot, or explicitly designate this as a separate later-snapshot analysis.
M7. The framing omits directly relevant precedents. Dubben and Beck-Bornholdt already investigated publication bias in studi
Reviewer chatgpt · Section 1. The question
The framing omits directly relevant precedents. Dubben and Beck-Bornholdt already investigated publication bias in studies of publication bias in 2005. Duyx et al. reviewed citation bias, including associations with authors’ stated conclusions, in 2017. Neither establishes that this exact within-meta-research comparison has been done, so they do not refute its specific novelty; they do require a narrower account of its contribution. Dubben and Beck-Bornholdt; Duyx et al..
This is publication bias asked about the field that studies publication bias, and asked at the citation stage rather than at acceptance.
What would fix it. Cite these precedents and explain the specific extension to evaluative conclusion direction within meta-research. Distinguish citation bias from publication bias, and negative evaluations of practice from statistically negative or hypothesis-inconsistent results.
M8. The stated OpenAlex n is 1790 works from 1805 PMIDs. The jsonl has 1790 unique OpenAlex IDs but only 1789 unique PMIDs.
Reviewer grok · Section 4. Sample
The stated OpenAlex n is 1790 works from 1805 PMIDs. The jsonl has 1790 unique OpenAlex IDs but only 1789 unique PMIDs. PMID 42018420 is two works with the same DOI, citation counts 3 and 0. The match rate is therefore 1789 unique PMIDs of 1805 (99.1 percent), and the 284 zeros include that extra zero row.
1790 of 1805 tier A PMIDs resolved in OpenAlex, a match rate of 99.2 percent.
What would fix it. Deduplicate on PMID or DOI before computing match rate and citation summaries. Report 1789 matched PMIDs and recompute the table.
M9. Section 5 says quantile regression is fitted on the upper deciles. Section 7 specifies the 75th and 90th percentiles. Th
Reviewer grok · Section 5. Outcome
Section 5 says quantile regression is fitted on the upper deciles. Section 7 specifies the 75th and 90th percentiles. The 75th percentile is not an upper decile.
a quantile regression on the upper deciles is fitted alongside
What would fix it. Name the quantiles once. If 75th and 90th, do not call them the upper deciles.
The raw verdicts
data/audit/P03/ holds each reviewer’s verdict as returned, the prompt both were given, and adjudicated.json. Nothing on this page is summarized from anything the repository does not also carry in full.