The agenda
Studies proposed against the map, published before they are run
Each proposal below is a protocol. It states a hypothesis, what would falsify it, the unit of analysis, the data source with a call that proved the source reachable, and the analysis in R, and it is published in that state before any analysis is run.
The order matters and it is deliberate. A protocol written after the result is known is not a protocol, and a project that criticizes selective reporting cannot itself decide what its question was once it has seen the answer. Every proposal therefore carries a date and a status, and a result page will link back to the protocol as it stood.
Status values
The distinction between a design and a protocol is not cosmetic. It comes from the sibling project’s design standard and it is the rule this project works to: the difference between a design and a protocol is a set of computed numbers. A document that asserts its sample size, its power or its feasibility rather than measuring them is a design, whatever it calls itself.
Every document from P10 onward is written against this project’s own design standard, twelve requirements each traced to the review finding that produced it.
| status | meaning |
|---|---|
design |
the question, hypothesis, threats and probes are written. Feasibility is not yet measured |
protocol-pending-validation |
probes computed and written in. One or more validation gates need human labels before it runs |
protocol-reduced |
probes computed, and they came back small enough that a pre-written rule cut the analysis down |
blocked-by-review |
two reviewers found an evidenced fatal flaw that stands unrebutted. Left as submitted |
agenda |
a real gap that public data cannot answer, recorded rather than skipped |
reported |
result published with the protocol and, when they exist, the reviews |
Nothing here is registered. Registration needs the probes complete, the validation gates passed, and two reviews attached. No document has all three.
Where each one stands
The table at the top of this page is the only status table, and it is read from each document’s front matter when the site is built. An earlier version of this page also kept a hand-written table; it drifted from the documents twice, once miscounting them and once still listing blocked protocols as ready, so it was removed rather than maintained.
Review
Review is by two models from different providers, each given the document and instructed to refute it, with output validated against a schema and adjudicated by the fixed rule in build/audit/adjudicate.py. That rule downgrades an unsubstantiated severe finding rather than counting it, lets a lone reviewer holding a verbatim quote block a document two others called sound, and publishes a contested document as contested rather than picking a winner. Twelve golden cases pin it. No document is reviewed by one model alone.
Five documents have been reviewed and all five were blocked: P01v2, P02, P03, P05 and P06, each on evidenced fatal findings. The findings for P01v2 and P02 were checked against primary source and all five reviews are published whole at the reviews page. The P03, P05 and P06 findings have not yet been acted on; each of those documents carries a banner saying so above its original text.
P10 to P29 have not been reviewed. They were written after those five reviews, against a standard distilled from them, and their review is scheduled for later. The designs are reviewed when their probes have run, because a reviewer asked to assess feasibility numbers that do not exist yet is being asked to review a wish.
How these were chosen
The selection rule was to turn the field’s own instrument on the field. Meta-research measures whether others share data, cite retracted work, follow checklists, replicate, and report inconvenient findings. Those questions had not been asked about meta-research itself, and most are answerable from data already harvested or openly available.
The rule also kept out the obvious alternative. A study measuring transparency, or FAIRness, in one more specialty is a rate in a new population, and rates in new populations are what this corpus already holds in quantity. Every document here has an exposure, an outcome and a direction, and every one can come out the other way.
The second round, P10 to P29, kept that rule where it applied and widened it where the author’s own tools made a sharper question possible: rtransparency and rfair are validated instruments, and several designs exist because a validated instrument turns a vague question into a measurable one. Candidates were dropped when a relevance-ordered search found the question already answered; the table below lists them.
P04 and P30 are not runnable and say so. P09 is the most important question of the set and is the least far along, which is a fact about difficulty rather than about priority.
Candidates dropped for lack of novelty
A dropped idea and the reason it was dropped are part of the trail. Each reason below is a record in data/lit/novelty-log.json.
| candidate | why it was dropped | log key |
|---|---|---|
| Journal transparency policy against the transparency of the meta-research it publishes | An individual participant data meta-analysis of predictors of data and code sharing covers it (PMID 37433624) | earlier round |
| Articles that share data are corrected more often | Tested at policy level and titled with its answer: “No evidence that mandatory open data policies increase error correction” (PMID 36109655) | dropped-open-data-corrections |
| Articles that share data or code contain fewer numerical errors | The data-sharing and reporting-error relation is established in psychology, including a replication (PMID 37053137); the numeric checks it relies on also fit biomedical reporting poorly | dropped-sharing-numeric-consistency |
| Retrospectively registered trials report larger effects | Three corpus studies test it (PMIDs 27377062, 41921890, 40334717) | dropped-registration-effect-size |
| Preprint versions of retracted articles go unflagged | Surveyed for COVID-19 (PMID 37035368), and inside the author’s own retraction-tracking work | dropped-preprints-of-retracted |
| Novelty claims and promotional language earn citations | “Scientific publications that use promotional language in the abstract receive more citations” (PMID 40764759) | dropped-hype-citation |
| Whether reviews using language models report their validation | The literature is too crowded to claim novelty from a top-ten search | dropped-llm-screening-validation |