Review of P01v2: blocked
Review of Advocacy and practice in meta-research: protocol.
A reviewer produced a fatal finding backed by a verbatim quote and no reviewer rebutted it. Under the rule, evidence outranks votes, so this outcome does not depend on the tally.
Adjudicated by build/audit/adjudicate.py at rule version 1. Coverage is complete: chatgpt, grok returned a verdict out of a roster of chatgpt, grok.
3 fatal, 19 major, 2 minor. 22 of the severe findings carry a verbatim quote.
Votes
| reviewer | vote | confidence |
|---|---|---|
chatgpt |
unsound | 0.98 |
grok |
unsound | 0.88 |
What each reviewer said
chatgpt
Pinned outcome instruments demonstrably misclassify real disclosures, and validation cannot detect important extraction failures. Power and null-result interpretations also exceed what calculations establish. Frame counts and all 15 power-grid cells reproduced; no residual tier A overlap was found by PMID, DOI, or PMCID. Screening yields, human agreement, and power for the specified mixed model remain unverified. Python extractor functions were executed; R execution was blocked by read-only temporary-file restrictions, so grading was checked through source inspection and a Python translation reproducing all six P4 counts. Literature searches did not establish an exact prior study duplicating this comparison.
grok
PMID exclusion of the 205 tier A records is real. I found no leftover overlap by PMID, PMCID, DOI, or title, so the article set is not R01’s. Section 4.1 block counts match the 2026-09-16 PubMed search log, and the section 7 n’s match probes.json. The extractor is imported rather than copied. Those parts hold. The confirmatory claim does not. P3 and P4 already fetched and graded the confirmatory full-text frame by arm (95/520 versus 132/3435; crude odds ratio 5.59), and section 7 publishes both rates. That is an exposure-by-outcome look, even without glmer. Section 7 then overstates what a null can mean: the grid does not detect 1.5 in the worst cell, power is an unclustered two-proportion test rather than the mixed model, and it assumes the same screening yield in both arms. The unlabeled P6 titles are mostly clinical reviews in the comparison arm and database or resource papers in the exposed arm, so yield 0.5 is not a floor and yield is not common across arms. The written grading rule does not match grade.R, P7 is not stratified by grade, screening is unpinned, and section 13 points at R01’s analyzer, which is a different model. I looked for a published empirical test of this same topic-versus-own-statement contrast and did not find one. Cristea, Naudet, and Caquelin 2022 (PMID 35623542) is a commentary that meta-researchers should evaluate their own sharing; it is not this analysis.
Fatal findings
F1. Outcome ascertainment confounds disclosure with publisher XML placement. Executing the pinned extractor on PMC5808091, a
Reviewer chatgpt · Section 6.1 Extraction; 9. Threats, and what happened to each
. Moving the identical sections into
That code searches
<front>and<back>only, never<body>
What would fix it. Identify articles’ own availability statements across JATS locations while distinguishing disclosures from methods and results discussing availability. Validate extraction against complete articles across both arms and publisher formats, including records currently classified ‘none’, then freeze the corrected instrument.
F2. The document presents itself as a confirmatory protocol whose instruments were fixed before data were retrieved and whos
Reviewer grok · Section Status: protocol
The document presents itself as a confirmatory protocol whose instruments were fixed before data were retrieved and whose hypothesis has not been tested on this sample. Probes P3 and P4 already fetched PMC full text for the confirmatory frame, graded every full-text record under the pinned rule, and wrote per-arm grade counts into probes.json and p4-grades.csv. Section 7 then publishes both rates. The unscreened 2x2 is 95/520 actionable in the exposed arm versus 132/3435 in the comparison arm (odds ratio 5.59). No glmer is required for that to be a look at the hypothesis. Section 10 describes P4 as the comparison arm only; the code and the protocol text both report the exposed arm as well. The author who will adjudicate screening disagreements has seen the direction and size of the association. A confirmatory protocol cannot be written after that look.
No outcome has been analyzed under this protocol. The probes measured the frame and the instruments; they did not test the hypothesis, and no exposure-by-outcome model has been fitted on this sample.
What would fix it. Drop the confirmatory label. State that P3/P4 were an outcome-by-exposure pilot on the unscreened frame. Freeze any screening or analysis choice that could still be influenced by those rates. Do not claim that instruments were fixed before data retrieval. Register the revised protocol before P6/P7 labeling and before screening the frame.
F3. Section 1 says this is a confirmatory test with every instrument fixed before any data are retrieved. The confirmatory s
Reviewer grok · Section 1. Why this design exists
Section 1 says this is a confirmatory test with every instrument fixed before any data are retrieved. The confirmatory sample has already been retrieved, extracted, and graded. The instruments that were frozen are the R01 extractor and grading rule; the data of this study were not unseen when this page was written. The same commit that pins the code also ran the probes.
This specifies a confirmatory test of the same hypothesis, on a sample R01 did not use, with every instrument fixed before any data are retrieved.
What would fix it. Replace the sentence with an accurate timeline: R01 instruments frozen, then this frame harvested and graded for feasibility, then this protocol written from those numbers. That is a protocol after a pilot, not a confirmatory test with unseen data.
Major findings
M1. Pinned code does not implement the stated locator rule. The extractor’s _text() retains visible text but discards xlink:
Reviewer chatgpt · Section 6.1 Extraction; 6.2 Grading
Pinned code does not implement the stated locator rule. The extractor’s _text() retains visible text but discards xlink:href targets. For PMC10202448, it retains ‘GitHub repository’ while losing the repository URL; REPO matches github.com rather than the bare repository name, producing ‘declarative’ under the pinned rules. Separately, grade_one(s) receives no article DOI or canonical URL, so it cannot exclude the article’s own identifiers. Its locator filter removes only an incomplete DOI-URL prefix, not matching article identifiers.
a named repository, or a DOI or URL that is not the article’s own
What would fix it. Preserve hyperlink targets and relevant data citations; pass normalized article identifiers into grading; distinguish article links from data/code locators. Verify these boundaries with real disclosures and negative examples before repinning.
M3. P7 validates grading conditional on successful extraction, not ascertainment of the primary outcome. The supplied p7-gra
Reviewer chatgpt · Section 6.3 Validation of the grading rule; 10. The probes that must run before this is a protocol
P7 validates grading conditional on successful extraction, not ascertainment of the primary outcome. The supplied p7-grading-sample.csv contains 150 nonempty statements; the grading translation classifies 133 as declarative and 17 as actionable, with zero ‘none’ records. A person seeing only extracted text cannot discover omitted body statements or discarded hyperlink targets. Consequently, even perfect kappa could coexist with systematic, publisher-dependent false negatives in the analysis population.
150 statements, stratified by rule grade, are graded by the author, blind to the article’s exposure and to the rule’s grade.
What would fix it. Add an end-to-end validation sample of complete articles, including extraction negatives and each extraction route. Have humans independently identify and grade disclosures. Report actionable-outcome sensitivity and specificity by arm, alongside agreement, and make those results govern instrument acceptance.
M4. The proposed hand-graded co-primary analysis changes the population and estimand. P7 excludes every article without an e
Reviewer chatgpt · Section 6.3 Validation of the grading rule
The proposed hand-graded co-primary analysis changes the population and estimand. P7 excludes every article without an extracted statement, although those articles belong in the primary outcome’s negative category. It also samples by rule grade. An unweighted analysis of these records does not reproduce the association among all eligible articles. The actual file contains only 19 exposed articles and 17 rule-positive outcomes before eligibility screening, so it cannot inherit section 7’s power claims.
the primary analysis is repeated on the hand-graded subsample alone as a co-primary, with both reported together regardless of whether they agree.
What would fix it. Use a representative, adequately sized hand-assessed article sample that includes extraction negatives. If sampling is stratified, specify nonzero inclusion probabilities and an appropriate analysis. Define the co-primary decision rule and calculate its power separately.
M5. Fully open-access publication does not imply complete PMC deposition. NCBI distinguishes full participation, NIH-portfol
Reviewer chatgpt · Section 8. Analysis, sensitivity analysis 6; 9. Threats, and what happened to each
Fully open-access publication does not imply complete PMC deposition. NCBI distinguishes full participation, NIH-portfolio participation, and selective deposit; even full participation starts at a specified publication date. Thus the proposed restriction does not establish equal or complete ascertainment. See PMC participation agreements. Moreover, higher marginal PMC coverage in the exposed arm does not identify the direction of selection bias in the exposure-outcome association, contrary to section 9’s assertion that it favors the hypothesis.
Within a fully open-access journal every article is in PMC, so the differential selection that P2 exposed cannot operate.
What would fix it. Base this sensitivity on verified journal-year PMC coverage and usable XML availability, rather than open-access status alone. Remove the guarantee that selection cannot operate and the unsupported claim about its direction; state the restricted target population explicitly.
M6. Computed grid values do not establish the advertised power. probes.R solves an independent two-proportion normal approxi
Reviewer chatgpt · Section 7. Sample size and power
Computed grid values do not establish the advertised power. probes.R solves an independent two-proportion normal approximation, whereas the primary analysis estimates a journal-conditional odds ratio with year adjustment. It does not model journal clustering, exposure concentration, or screening losses separately by arm. Even its own approximation contradicts detection across OR 1.5–3: the worst cell requires OR 2.18, and OR 1.5 has approximately 30.5% power at baseline 0.038 and yield 0.5. Comparing R01’s 0.205 with this different frame’s 0.0384 also does not establish a lower bound on the screened baseline.
The design detects effects across that entire band
What would fix it. Calculate power for the planned mixed model using journal composition and separately justified screening yields for each arm. Include uncertainty in the screened baseline and validation fallbacks. Select a smallest effect of interest and report scenarios that fail to achieve its target power.
M7. A confidence interval containing 1 neither falsifies a positive association nor establishes absence. The same interval c
Reviewer chatgpt · Section 2. Objective and hypothesis; 7. Sample size and power
A confidence interval containing 1 neither falsifies a positive association nor establishes absence. The same interval can include substantial positive effects. Even a genuinely 80%-powered test fails to reject its null in about 20% of repetitions at the planning alternative. Power calculations do not turn nonsignificance into an equivalence test. The proposed rule could therefore label an inconclusive result as evidence against the hypothesis.
An adjusted odds ratio whose 95 percent confidence interval includes 1, in a sample powered as in section 7.
What would fix it. Prespecify a scientifically meaningful effect threshold and interpret the confidence interval against that threshold. Use an appropriate equivalence or effect-exclusion criterion for absence claims; classify sufficiently wide intervals as inconclusive.
M8. Outcome analysis has already occurred: P4 tabulates actionable outcomes by exposure, reporting 95/520 versus 132/3435, w
Reviewer chatgpt · Section Status callout; 1. Why this design exists; 7. Sample size and power; 8. Analysis
Outcome analysis has already occurred: P4 tabulates actionable outcomes by exposure, reporting 95/520 versus 132/3435, which reveals a crude odds ratio of approximately 5.59. Absence of a fitted regression does not make these outcomes unseen. Section 8 also calls analyses preregistered while the status says ‘Not registered’. The earlier e725b073034c4a777acbd1c10a375291be65e314 version already specified the primary model, so this is not evidence that the primary hypothesis was invented afterward; it is evidence that the account of prior outcome access is inaccurate.
No outcome has been analyzed under this protocol.
What would fix it. Provide an accurate timeline of specification, retrieval, arm-specific outcome summaries, and subsequent decisions. Distinguish pre-probe specifications from post-probe additions. Any decisions revised using these outcomes require an untouched evaluation sample for a prospective confirmatory claim, or explicit exploratory labeling.
M9. The claimed executable analysis does not implement P01v2. The pinned R01 analyze.R hardcodes R01 input paths, derives ex
Reviewer chatgpt · Section 13. Availability; 8. Analysis
The claimed executable analysis does not implement P01v2. The pinned R01 analyze.R hardcodes R01 input paths, derives exposure from topic_datashare rather than the new frame’s arm definition, retains article-type adjustment, and lacks P01v2 screening/validation branches and sensitivities 3, 5, and 6. The confirmatory directory contains probe code, not an analysis driver implementing the stated protocol. Running the identified analysis unchanged would analyze the wrong sample under the wrong specification.
The analysis code that will run is already written and public; it is the code that produced R01, and section 6 pins it.
What would fix it. Provide and pin a P01v2 analysis driver implementing the declared sample, exposure, validation contingencies, primary and secondary models, and all six sensitivities. Include reproducible sample-generation procedures and a documented command that consumes the confirmatory inputs.
M10. The protocol says a 95 percent CI that includes 1, in a sample powered as in section 7, can be read as evidence of absen
Reviewer grok · Section 2. Objective and hypothesis
The protocol says a 95 percent CI that includes 1, in a sample powered as in section 7, can be read as evidence of absence. Power to detect an odds ratio of 2.18 at 80 percent is not power to rule out 1.5, and it is not an equivalence test. There is no smallest effect size of interest and no TOST or CI-exclusion rule against a margin. Failure to reject OR=1 remains absence of evidence for anything smaller than the detectable value. The mixed model in section 8 is also less powered than the two-proportion calculation used in section 7, because data-availability statements cluster by journal.
Section 7 fixes the power in advance so that a null can be read as evidence of absence rather than absence of evidence, which was not true of R01.
What would fix it. State a smallest odds ratio that would still matter (for example 1.5). Falsify only if the 95 percent CI lies entirely below that margin, or run a preregistered equivalence test. Recompute power for glmer with (1|journal) under a plausible ICC. Delete the claim that a null here is evidence of absence.
M11. Three power claims do not survive the protocol’s own numbers and files. First, section 2 says 1.5 to 3 would be consiste
Reviewer grok · Section 7. Sample size and power
Three power claims do not survive the protocol’s own numbers and files. First, section 2 says 1.5 to 3 would be consistent with R01 and section 7 says the design detects that entire band. On the protocol’s own grid, at yield 0.5 and baseline 0.038 the detectable OR is 2.18, so 1.5 is not detectable in the cell treated as worst case. Second, the grid never goes below yield 0.5, but the unlabeled P6 title list is not a 50 percent meta-research sample. The comparison titles are mostly clinical systematic reviews and meta-analyses of patient outcomes (the report block’s PRISMA/CONSORT phrases and the qualifier ‘we searched PubMed’ retrieve ordinary reviews). The exposed titles include database papers, resource descriptions, and clinical studies that mention data sharing (Scientific Data alone is 17 exposed and 1 comparison in the full-text frame). A yield of 0.15 at baseline 0.038 gives a detectable OR of about 3.49, which trips the protocol’s own P5 rule that a detectable effect above about 2.5 means the frame must be widened. Third, the unscreened 18.3 percent vs 3.8 percent contrast is not a lower bound on the screened contrast: screening should remove data journals from the exposed arm (lowering the exposed rate) as well as clinical reviews from the comparison arm (raising the comparison rate). The power calculation never allows those rates to move in opposite directions.
The design detects effects across that entire band, which is the property R01 lacked: its floor was roughly 3.7, so it could only ever have seen an effect as large as the one it reported.
What would fix it. Treat the P6 file as a yield probe before claiming power. Estimate yield (and yield by arm) from the 200 titles, or from a hand-screened subset, and rebuild the grid down to that yield. If the worst-case detectable OR exceeds 2.5, widen the frame or drop the claims that the design detects 1.5 and that a null rules something out. Separate the unscreened mixture rates from the screened target population.
M12. Section 9 asserts that differential PMC coverage favors the hypothesis. Section 12 says the direction of bias on the bet
Reviewer grok · Section 9. Threats, and what happened to each
Section 9 asserts that differential PMC coverage favors the hypothesis. Section 12 says the direction of bias on the between-arm comparison is not obvious and is not claimed toward the null. Both cannot stand. If section 9 is right, the primary glmer on the PMC full-text set is biased in the direction of a positive finding, and the analysis that was added to block that path is only sensitivity 6. If section 12 is right, section 9’s directional claim should be withdrawn. The 20-point PMC gap is large enough that this is not a side issue.
Arms are differentially selected into the analysis set and the direction favors the hypothesis.
What would fix it. Pick one account of the bias and make the analysis that removes differential PMC selection the primary (for example, journals in which every sampled article is in PMC, with that rule defined before seeing outcomes). Keep the current model as a sensitivity. Align sections 9 and 12.
M13. Section 12 calls the qualified frame a high-precision slice. The qualifier includes ‘we searched PubMed’, ‘publications’
Reviewer grok · Section 12. What this design still cannot do
Section 12 calls the qualified frame a high-precision slice. The qualifier includes ‘we searched PubMed’, ‘publications’, and ‘PRISMA statement’ via the report block, which is how ordinary clinical reviews enter. That is a precision failure, not only the recall failure section 12 describes (papers that say ‘we examined 300 trials’ drop out). The P6 titles are the evidence. High precision is the claim that screening was supposed to earn, not a property of the harvested frame.
The sample is a high-precision slice of the field, not the field.
What would fix it. Describe the qualified frame as high-recall for papers that mention literature-search boilerplate and low-precision for meta-research. Put the precision claim only after screening, and only if P6 kappa and yield support it.
M14. The written rule grades a DOI or URL as actionable only when it is not the article’s own DOI. grade_one() takes only the
Reviewer grok · Section 6.2 Grading
The written rule grades a DOI or URL as actionable only when it is not the article’s own DOI. grade_one() takes only the statement string. It never receives the article DOI. The filter loc[!str_detect(loc, ‘doi.org/10.[0-9]{4,9}/$’)] drops only a DOI URL with an empty path, not this article’s DOI and not the journal site. The comment in grade.R claims the journal site is excluded; that is not implemented either. On the confirmatory frame, 38 of 227 actionable grades are URL/DOI without a named repository (12 exposed, 26 comparison). Those are exactly the grades this clause is supposed to police.
A DOI or URL present that is not the article’s own DOI: actionable.
What would fix it. Pass each article’s DOI (and host) into the grader and drop locators that match them, or change test 4 to match the code. Pin the revised function and re-run P4/P7.
M15. The screening instrument is not pinned. Extract.py is pinned to a commit hash; screening is ’two models from different p
Reviewer grok · Section 4.3 Screening
The screening instrument is not pinned. Extract.py is pinned to a commit hash; screening is ‘two models from different providers’ with no model identifiers, no prompt, no temperature, and no code path. Later in section 10 the protocol allows P6 to use a single model while a reviewer is quota-blocked. Dual-review screening then cannot run, and single-model screening is not one of the stated fallbacks (the stated fallback is author screening of a subsample). The author is also not blind to exposure or to the P4 association when adjudicating disagreements.
Screening is on title and abstract, by two models from different providers, each given the same written criterion and blind to the other.
What would fix it. Pin model names, the full screening prompt, and decoding settings in the repository at a commit hash, as was done for extract.py. If only one model is available, use the written fallback (author-screened subsample, power recomputed) rather than an unlisted single-model screen. Keep the author’s adjudication blind to arm and to P4 rates.
M16. P7 is described as 150 statements stratified by rule grade. The file studies/P01v2-confirmatory/out/p7-grading-sample.cs
Reviewer grok · Section 6.3 Validation of the grading rule
P7 is described as 150 statements stratified by rule grade. The file studies/P01v2-confirmatory/out/p7-grading-sample.csv has 150 non-empty statements, 0 empties, and under the pinned rule 17 actionable and 133 declarative. Equal stratification on three grades would be about 50 each, including none. Equal stratification on the two non-none grades would be about 75/75. 17/133 matches a simple random sample of extracted statements (frame: 227 actionable of 1688 statements, 13.4 percent; P7: 11.3 percent). The most common analysis grade, none, is unvalidated. Kappa on 17 actionable items is a noisy gate for the 0.80 / 0.60 thresholds. No script in the repository produces this file, so the sampling is not itself pinned.
150 statements, stratified by rule grade, are graded by the author, blind to the article’s exposure and to the rule’s grade.
What would fix it. Draw 50 none, 50 declarative, and 50 actionable, or draw a prevalence-weighted sample and say so. Put the sampler in the repo. Report kappa for the primary binary (actionable vs not) on a sample that includes none. Do not treat agreement on boilerplate declarative statements as agreement on the primary outcome.
M17. Sensitivity 3 restricts to journals with at least 10 articles in each arm. In the full-text frame that is 4 journals (BM
Reviewer grok · Section 8. Analysis
Sensitivity 3 restricts to journals with at least 10 articles in each arm. In the full-text frame that is 4 journals (BMJ Open, PLOS ONE, Journal of medical Internet research, F1000Research). Scientific Data, the journal that most confounds exposure with policy (17 exposed, 1 comparison), is dropped. That is not a general within-journal estimate. Sensitivity 6 is the analysis meant to block differential PMC selection, but ‘fully open-access journal’ is undefined, and the supporting sentence is false: gold OA is not the same as PMC deposit, and PMC also holds author manuscripts from non-OA journals. OpenAlex oa_status is already in data/lit/openalex-frame-P01v2-summary.json and is not used. The multiplicity paragraph says five sensitivities; there are six. There is no P01v2 analyze.R; year_c is not defined here (R01 used year minus 2020); sensitivity 4 says authorship but R01 used a repository-name regex in the statement.
Within a fully open-access journal every article is in PMC, so the differential selection that P2 exposed cannot operate.
What would fix it. Probe how many journals pass the 10-per-arm rule and either drop sensitivity 3 or replace it with all journals that published both arms. Define sensitivity 6 with an operational OA/PMC rule (for example, journals whose articles in this frame are 100 percent PMC, or OpenAlex gold+diamond) and do not equate OA with PMC. Write the analysis script this protocol actually specifies. Count six sensitivities.
M18. P3’s stated design-change rule is a route distribution unlike tier A’s. R01 title-matching routes were about 16 percent
Reviewer grok · Section 10. The probes that must run before this is a protocol
P3’s stated design-change rule is a route distribution unlike tier A’s. R01 title-matching routes were about 16 percent of the exposed arm and 7 percent of the comparison arm. On this frame, among full-text records, title-matching routes are 5.0 percent exposed (26/520) and 9.3 percent comparison (318/3435): the imbalance reversed. The protocol never reports that comparison and does not say why it does not change the design. P4’s stated output is the comparison-arm grade distribution; the probe and section 7 both emit the exposed arm as well.
A route distribution unlike tier A’s means publishers differ between frames and journal adjustment is carrying more weight than assumed
What would fix it. Report the P3 route table against R01, and either justify keeping the design or add the journal-weight consequence P3 said would follow. Describe P4 as what it actually computed: both arms.
M19. Section 13 says the analysis code that will run is already written and is the code that produced R01. studies/P01-advoca
Reviewer grok · Section 13. Availability
Section 13 says the analysis code that will run is already written and is the code that produced R01. studies/P01-advocacy-and-practice/R/analyze.R fits art_type, does not exclude comments/editorials up front, does not implement sensitivities 3, 5, or 6, and flags the author’s papers by matching ‘choxos|sofi-mahmudi’ in the statement rather than by authorship. There is no analyze.R under studies/P01v2-confirmatory/. The primary model in section 8 is a different model.
The analysis code that will run is already written and public; it is the code that produced R01, and section 6 pins it.
What would fix it. Write and pin the analysis script that implements section 8, including the six sensitivities and an operational rule for the author’s papers. Do not point at R01’s analyzer as the code that will run.
Minor findings
M1. The multiplicity paragraph refers to five sensitivity analyses. Section 8 lists six. Sensitivity 6 is the one added for
Reviewer grok · Section 8. Analysis
The multiplicity paragraph refers to five sensitivity analyses. Section 8 lists six. Sensitivity 6 is the one added for the main selection threat, so undercounting it is not cosmetic.
The secondary and the five sensitivity analyses are not corrected for multiplicity and are not used to claim significance; they exist to show whether the primary estimate is fragile.
What would fix it. Say six, or state that sensitivity 6 is exempt from the ‘not used to claim significance’ rule if it is to be interpreted as the deconfounded estimate.
M2. The realized-counts row labeled ‘frame, after removing 205 tier A overlaps from 7617 records’ gives 704 and 6526. Those
Reviewer grok · Section 7. Sample size and power
The realized-counts row labeled ‘frame, after removing 205 tier A overlaps from 7617 records’ gives 704 and 6526. Those are eligible_exposed and eligible_comparison after the further drop of 90 publication-type and 92 language records (7230 eligible). The label omits those exclusions. The numbers match probes.json; the row title does not.
frame, after removing 205 tier A overlaps from 7617 records
What would fix it. Label the row as after overlap, publication-type, and language exclusions.
The raw verdicts
data/audit/P01v2/ holds each reviewer’s verdict as returned, the prompt both were given, and adjudicated.json. Nothing on this page is summarized from anything the repository does not also carry in full.