The chronology of the evidence
What the dates say that a count of findings cannot
Every finding in this catalog carries the publication year of the paper that made it, so a problem’s evidence can be read as a sequence rather than a pile. That answers three questions a pooled count cannot: has later work answered what an earlier paper left open, is a problem being closed in pieces by different groups, and is a problem simply being restated across the years with nothing happening in between.
195 problems carry at least one dated finding, drawn from a full-text reading of 687 papers published between 2010 and 2026.
Did the later work answer the earlier work?
27 problems had progress as their newest evidence and every assertion of openness older. Publication order alone proves nothing there: two papers can straddle a date and be about different questions, and a later paper can close a corner of a problem whose earlier statement was about something else. Each was put to an independent reviewer with the older and newer claims side by side, and without the classification, so it could not agree with the mechanism by reading its label.
None was fully answered.
| The later work | Problems |
|---|---|
| answers what the earlier work left open | 0 |
| answers a component; a named part survives | 22 |
| is about a different question | 3 |
| does not answer it | 2 |
| could not be judged from the claims | 0 |
The five in the bottom two rows are where the dates mislead, and each is recorded on its own page alongside the reviewer’s reasoning.
Reasserted after progress
74 problems have partial progress on the record and a later paper still calling them open. These are the entries most likely to be reported as settled by someone reading only the newest method paper, and the ones whose prior work should cite both sides.
| Problem | Registry says | Papers | Span | Chronology |
|---|---|---|---|---|
| IDN-01 Transitivity and conditional constancy cannot be tested in-sample | Confirmed open | 103 | 2011 to 2026 | 2011 open · 2012 open · 2013 open · 2014 part · 2014 open · 2015 open · 2016 open · 2017 part · 2017 open · 2018 open · 2019 open · 2020 part · 2020 open · … |
| HET-03 Single shared heterogeneity variance across classes and populations | Partially addressed | 91 | 2010 to 2026 | 2010 open · 2011 open · 2012 open · 2013 part · 2013 open · 2014 open · 2015 part · 2015 open · 2016 part · 2016 open · 2017 part · 2017 open · 2018 part · … |
| CMP-03 Strict component additivity may be clinically false | Partially addressed | 58 | 2018 to 2026 | 2018 open · 2019 open · 2020 open · 2021 part · 2021 open · 2022 open · 2023 open · 2024 part · 2024 open · 2025 open · 2026 open |
| EVB-02 Evidence-set assembly can favor the sponsor before modeling begins | Overstated | 52 | 2010 to 2026 | 2010 open · 2011 open · 2012 open · 2013 part · 2013 open · 2014 part · 2014 open · 2015 part · 2015 open · 2016 open · 2017 open · 2018 open · 2019 open · … |
| MOD-03 Ecological versus within-study effect modification | Unverifiable | 50 | 2011 to 2026 | 2011 open · 2012 open · 2013 open · 2014 part · 2014 open · 2015 open · 2016 open · 2017 part · 2017 open · 2018 part · 2018 open · 2020 open · 2021 part · … |
| CMU-02 Priors, not data, can drive posteriors in weakly identified models | Partially addressed | 40 | 2012 to 2026 | 2012 open · 2013 part · 2013 open · 2014 open · 2016 part · 2016 open · 2017 open · 2018 part · 2018 open · 2019 open · 2020 open · 2021 open · 2022 open · … |
| CMP-06 Measurement error and component misclassification are ignored | Confirmed open | 32 | 2018 to 2026 | 2018 part · 2018 open · 2019 open · 2020 open · 2021 open · 2022 open · 2023 open · 2024 open · 2025 open · 2026 open |
| CMP-04 Additivity model selection can be unstable | Confirmed open | 30 | 2016 to 2026 | 2016 open · 2018 open · 2019 open · 2020 open · 2021 part · 2021 open · 2022 part · 2022 open · 2023 open · 2024 open · 2025 open · 2026 open |
| COV-01 Deciding which covariates are true effect modifiers | Confirmed open | 29 | 2012 to 2025 | 2012 open · 2013 open · 2014 part · 2014 open · 2016 open · 2017 open · 2018 open · 2019 open · 2020 part · 2020 open · 2021 part · 2021 open · 2023 open · … |
| HET-01 Design and conduct differences survive baseline covariate balancing | Overstated | 29 | 2015 to 2026 | 2015 open · 2016 part · 2017 open · 2018 open · 2019 part · 2019 open · 2020 open · 2021 open · 2022 open · 2023 open · 2024 open · 2025 open · 2026 open |
| DEC-04 No PAIC-specific reporting standard or credibility appraisal tool | Partially addressed | 26 | 2011 to 2025 | 2011 open · 2013 part · 2013 open · 2014 open · 2015 open · 2016 part · 2016 open · 2020 part · 2020 open · 2021 open · 2022 open · 2023 part · 2023 open · … |
| DIA-07 Scenario grids omit network topology and information placement | Confirmed open | 25 | 2011 to 2025 | 2011 open · 2012 open · 2013 open · 2014 part · 2015 part · 2015 open · 2016 part · 2017 open · 2018 open · 2019 open · 2020 open · 2021 part · 2021 open · … |
| ADJ-12 Routine NMA output omits graph diagnostics and value of information | Overstated | 24 | 2013 to 2024 | 2013 part · 2014 part · 2014 open · 2015 part · 2016 part · 2017 part · 2017 open · 2018 part · 2018 open · 2019 part · 2019 open · 2020 part · 2022 part · … |
| DIS-07 External-control bridges become cross-design causal synthesis | Partially addressed | 22 | 2011 to 2026 | 2011 open · 2015 part · 2015 open · 2016 open · 2018 part · 2018 open · 2021 open · 2022 open · 2023 part · 2023 open · 2024 open · 2025 open · 2026 open |
| OUT-13 Reconstruction uncertainty from digitized Kaplan-Meier curves | Partially addressed | 21 | 2011 to 2026 | 2011 open · 2012 open · 2014 part · 2014 open · 2015 open · 2016 open · 2017 open · 2020 open · 2022 open · 2023 open · 2024 open · 2025 open · 2026 open |
| EVB-01 Selective IPD availability threatens transport across the network | Confirmed open | 21 | 2012 to 2026 | 2012 open · 2013 open · 2015 open · 2016 open · 2020 part · 2020 open · 2021 part · 2021 open · 2022 part · 2022 open · 2023 open · 2024 open · 2025 open · 2026 open |
| OUT-04 Different instruments and standardized mean difference transport | Partially addressed | 20 | 2012 to 2026 | 2012 open · 2015 open · 2016 part · 2016 open · 2017 open · 2018 open · 2019 open · 2020 part · 2020 open · 2022 open · 2023 part · 2024 open · 2025 part · … |
| MIS-04 Model and bridge selection uncertainty never enters the interval | Confirmed open | 19 | 2013 to 2026 | 2013 open · 2015 open · 2016 open · 2018 open · 2019 open · 2020 open · 2021 open · 2022 open · 2023 open · 2024 open · 2025 part · 2025 open · 2026 open |
| EST-01 Target population is unnamed and set by data availability | Overstated | 19 | 2014 to 2026 | 2014 open · 2015 part · 2017 open · 2018 open · 2019 open · 2020 part · 2022 part · 2022 open · 2023 open · 2024 open · 2025 open · 2026 open |
| IDN-05 The shared effect modifier check has low and uncalibrated power | Overstated | 16 | 2013 to 2026 | 2013 open · 2014 open · 2015 open · 2016 open · 2017 open · 2018 part · 2021 open · 2022 open · 2024 open · 2025 open · 2026 open |
| COV-07 Unknown target joint covariate distribution | Partially addressed | 15 | 2015 to 2026 | 2015 open · 2020 part · 2020 open · 2021 part · 2021 open · 2022 open · 2023 part · 2024 part · 2024 open · 2025 open · 2026 open |
| IDN-06 ML-NMR interactions can rest solely on aggregate-data variation | Unverifiable | 15 | 2013 to 2024 | 2013 open · 2014 open · 2015 part · 2016 part · 2017 open · 2018 part · 2020 open · 2021 part · 2021 open · 2022 open · 2023 open · 2024 open |
| EST-10 Bucher ITC and aggregate-data NMA name no target population | Unverifiable | 13 | 2011 to 2025 | 2011 open · 2012 open · 2015 open · 2017 open · 2018 open · 2019 part · 2020 open · 2021 open · 2022 part · 2025 open |
| MIS-01 Missing aggregate summaries and unavailable IPD have no remedy | Unverifiable | 13 | 2014 to 2026 | 2014 part · 2018 part · 2018 open · 2020 part · 2020 open · 2022 open · 2024 open · 2025 part · 2025 open · 2026 open |
| IDN-11 Trial-engagement effects are mistaken for prognostic imbalance | Partially addressed | 13 | 2015 to 2026 | 2015 open · 2017 open · 2018 open · 2019 open · 2023 open · 2024 part · 2024 open · 2025 open · 2026 open |
| QBA-16 Probabilistic QBA needs elicited and validated bias distributions | Confirmed open | 12 | 2012 to 2026 | 2012 part · 2013 part · 2014 open · 2018 open · 2019 open · 2020 open · 2021 open · 2022 open · 2023 open · 2025 open · 2026 open |
| CMP-22 IPD availability may be informative rather than random | Confirmed open | 12 | 2012 to 2025 | 2012 open · 2016 open · 2018 part · 2019 part · 2020 open · 2021 open · 2022 part · 2022 open · 2025 open |
| CMU-04 Computational diagnostics and Monte Carlo error go unreported | Partially addressed | 12 | 2013 to 2025 | 2013 open · 2014 open · 2017 open · 2020 open · 2021 open · 2023 part · 2025 open |
| OUT-19 No multivariate population-adjusted model for correlated outcomes | Overstated | 11 | 2014 to 2025 | 2014 part · 2014 open · 2016 open · 2017 open · 2018 open · 2019 open · 2020 open · 2025 open |
| MOD-01 Conventional STC reports a conditional effect at target means | Partially addressed | 11 | 2018 to 2026 | 2018 open · 2020 open · 2021 open · 2022 part · 2024 part · 2025 part · 2025 open · 2026 open |
Restated across the years, with nothing in between
20 problems were called open by at least three papers spanning at least eight years, and no paper in the corpus reports any progress on them at all. Independent restatement over a long period, by authors who mostly do not cite each other, is the strongest evidence this reading can give that a gap is real rather than an artifact of one author’s framing.
| Problem | Registry says | Papers | Span |
|---|---|---|---|
| DIS-01 Cross-gap contrasts are absent from the randomized likelihood | Confirmed open | 26 | 2013 to 2026 |
| QBA-12 Treatment version, care context and subsequent-therapy drift | Confirmed open | 23 | 2015 to 2026 |
| EVB-05 No reporting standard covers what population adjustment needs | Unverifiable | 21 | 2012 to 2026 |
| CMP-07 Causal meaning of a component effect is ambiguous | Unverifiable | 20 | 2018 to 2026 |
| DIS-03 Baseline risk is not a neutral anchor for disconnected evidence | Not supported | 14 | 2015 to 2026 |
| IDN-13 Time, geography and setting are confounded in non-crossed evidence | Confirmed open | 12 | 2011 to 2026 |
| MOD-04 No post-selection inference for model choice in population adjustment | Confirmed open | 11 | 2011 to 2025 |
| DIA-05 No shared benchmark platform with common scenarios, seeds and truth | Confirmed open | 10 | 2011 to 2026 |
| IDN-04 Recorded-covariate adjustment cannot remove unmeasured modifier bias | Confirmed open | 10 | 2012 to 2026 |
| DIS-02 No data-driven criterion can rank candidate bridge families | Overstated | 10 | 2014 to 2025 |
| DIA-01 No diagnostic can falsify unmeasured effect modification in PAIC | Overstated | 10 | 2015 to 2025 |
| DIS-11 Predictive criteria and population comparison cannot validate a bridge | Partially addressed | 9 | 2015 to 2026 |
| DIS-04 Dose-response bridges assume a curve that may not transport | Confirmed open | 8 | 2014 to 2026 |
| DIS-05 Class-effect bridges extrapolate to treatments with no randomized link | Confirmed open | 7 | 2016 to 2025 |
| OUT-15 Treatment switching and subsequent therapies in survival PAIC | Confirmed open | 7 | 2017 to 2026 |
| OUT-12 Survival extrapolation is unidentified beyond the observed follow-up | Not supported | 6 | 2011 to 2025 |
| SFW-15 netmeta and gemtc carry no target-population representation | Confirmed open | 5 | 2014 to 2024 |
| DEC-10 Trimming redefines the target population without naming it | Unverifiable | 5 | 2018 to 2026 |
| DIA-03 PAIC diagnostics are not evaluated as classifiers of realized error | Overstated | 3 | 2013 to 2025 |
| QBA-19 Latent-U Bayesian QBA hides nonidentification behind a proper prior | Unverifiable | 3 | 2012 to 2021 |
The same gap, named again and again
Papers name the gaps they leave behind, in their own words. Across the corpus the same gap gets named repeatedly by authors who never cite each other, and that repetition is invisible while the gaps sit as free text under the paper that wrote them. Clustering the text does not find it, because a recurrence is a paraphrase rather than a near duplicate. So all 3619 future-research gaps were read and attached to a theme instead, and the chronology falls out.
139 themes were named by three or more papers across eight or more years; 78 of those have no paper in the corpus reporting any progress at all.
The most-restated of them are not registered problems yet. They are candidates from this reading, and they need the same scrutiny as any other entry before they become one.
| Theme | Papers | Named in | Progress reported |
|---|---|---|---|
| IDN-01 Transitivity and conditional constancy cannot be tested in-sample | 60 | 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023… | 2014, 2017, 2020, 2025 |
| MIS-04 Model and bridge selection uncertainty never enters the interval | 55 | 2012, 2013, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024… | 2025 |
| EVB-02 Evidence-set assembly can favor the sponsor before modeling begins | 51 | 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2022, 2023… | 2013, 2014, 2015, 2022, 2023, 2025 |
| CMP-03 Strict component additivity may be clinically false | 47 | 2013, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | 2021, 2024 |
| EVB-03 Trials combined under one outcome label are not one estimand | 37 | 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025… | 2013, 2014, 2018, 2019, 2020, 2026 |
| MIS-01 Missing aggregate summaries and unavailable IPD have no remedy | 37 | 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | 2014, 2018, 2020, 2025 |
| IDN-15 Global inconsistency diagnostics lack calibrated operating characteristics | 36 | 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022… | none |
| HET-03 Single shared heterogeneity variance across classes and populations | 34 | 2010, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022… | 2013, 2015, 2016, 2017, 2018, 2021, 2022 |
| DEC-18 NMA rank output lacks a decision-theoretic reconciliation with net benefit | 34 | 2011, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023… | none |
| CMP-05 No versioned intervention ontology maps onto CNMA component matrices | 34 | 2013, 2015, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | 2026 |
| EVB-05 No reporting standard covers what population adjustment needs | 33 | 2011, 2012, 2013, 2014, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023… | none |
| MOD-03 Ecological versus within-study effect modification | 33 | 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020, 2021, 2022, 2023… | 2014, 2017, 2018, 2021, 2022, 2023 |
| DIA-20 Transitivity dissimilarity screens have no calibrated threshold | 29 | 2011, 2013, 2014, 2015, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025… | none |
| DIA-07 Scenario grids omit network topology and information placement | 28 | 2012, 2013, 2014, 2016, 2017, 2019, 2020, 2021, 2022, 2023, 2024, 2025 | 2014, 2015, 2016, 2021, 2022, 2023 |
| CMP-34 No sensitivity or reporting standard for discretionary component coding | 28 | 2018, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | none |
| IDN-13 Time, geography and setting are confounded in non-crossed evidence | 25 | 2011, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025… | none |
| EVB-08 Network small-study-effects diagnostics lack an assumption-light default | 24 | 2011, 2012, 2013, 2014, 2015, 2017, 2018, 2019, 2020, 2021, 2022, 2023… | none |
| QBA-16 Probabilistic QBA needs elicited and validated bias distributions | 24 | 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2024… | 2012, 2013 |
| DIA-26 No validated diagnostic localizes inconsistency to a design | 23 | 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2022, 2023, 2024, 2025… | none |
| DEC-23 The treatment hierarchy is published without a certainty rating | 23 | 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | none |
| DEC-30 ROB-MEN reliability is unmeasured and CINeMA agreement collapses on large networks | 22 | 2014, 2016, 2017, 2019, 2020, 2021, 2022, 2023, 2024, 2025 | none |
| CMP-31 No practical-identification diagnostic for component effects | 22 | 2018, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | none |
| CMP-29 No component-level transitivity and coherence assessment | 21 | 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026 | none |
| COV-01 Deciding which covariates are true effect modifiers | 20 | 2013, 2018, 2019, 2021, 2022, 2023, 2024, 2025 | 2014, 2020, 2021 |
| CMU-02 Priors, not data, can drive posteriors in weakly identified models | 19 | 2012, 2016, 2017, 2018, 2021, 2023, 2024, 2025, 2026 | 2013, 2016, 2018 |
| OUT-04 Different instruments and standardized mean difference transport | 19 | 2012, 2015, 2016, 2017, 2019, 2020, 2021, 2022, 2024, 2025 | 2016, 2020, 2023, 2025 |
| DEC-04 No PAIC-specific reporting standard or credibility appraisal tool | 17 | 2011, 2013, 2014, 2016, 2017, 2020, 2021, 2023, 2024, 2026 | 2013, 2016, 2020, 2023 |
| OUT-02 Rare events, separation, and zero-event studies in weighted synthesis | 17 | 2012, 2014, 2016, 2018, 2019, 2020, 2021, 2022, 2025 | 2019, 2022 |
| DEC-13 Published PAIC analyses cannot be reconstructed from the record | 17 | 2014, 2016, 2017, 2018, 2019, 2022, 2023, 2024, 2025, 2026 | 2013, 2026 |
| OUT-19 No multivariate population-adjusted model for correlated outcomes | 17 | 2011, 2014, 2016, 2017, 2018, 2019, 2020, 2022, 2023 | 2014 |
| DIA-05 No shared benchmark platform with common scenarios, seeds and truth | 16 | 2011, 2012, 2015, 2016, 2017, 2021, 2022, 2023, 2024, 2025, 2026 | none |
| MOD-09 No criterion says when ML-NMR should be preferred to MAIC or STC | 16 | 2011, 2015, 2016, 2017, 2019, 2021, 2022, 2023, 2024, 2025, 2026 | 2020, 2023, 2024, 2026 |
| DIA-09 Narrow outcome families and hazard-ratio-only survival scoring | 16 | 2013, 2014, 2017, 2019, 2020, 2021, 2024, 2025, 2026 | none |
| DEC-27 No rule says which of three conflicting estimates to use | 15 | 2012, 2013, 2015, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2026 | none |
| DEC-29 Ranking metrics reward imprecision and no correction exists | 15 | 2013, 2014, 2016, 2018, 2019, 2020, 2021, 2022, 2025, 2026 | none |
| CMP-32 No sample-size or allocation method for component NMA | 15 | 2015, 2018, 2021, 2022, 2023, 2024, 2025, 2026 | none |
| HET-01 Design and conduct differences survive baseline covariate balancing | 15 | 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2023, 2024, 2025 | 2016, 2019 |
| DEC-25 No agreed rule turns an NMA posterior into a recommendation | 15 | 2015, 2016, 2017, 2019, 2020, 2021, 2022, 2024, 2025, 2026 | none |
| EVB-04 No decision-value method chooses which IPD or bridging trial to get | 15 | 2013, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 | 2016, 2022, 2023 |
| MOD-19 No valid selection criterion for cut points and powers in time-varying survival NMA | 14 | 2011, 2017, 2019, 2020, 2021, 2022, 2024, 2025, 2026 | none |
Findings by publication year
What the corpus says about these problems, by the year it was said.
| Year | Confirms open | Partly addresses | Resolves | Contradicts |
|---|---|---|---|---|
| 2010 | 4 | 0 | 0 | 0 |
| 2011 | 22 | 1 | 0 | 0 |
| 2012 | 40 | 5 | 0 | 0 |
| 2013 | 36 | 23 | 0 | 0 |
| 2014 | 59 | 26 | 0 | 0 |
| 2015 | 61 | 11 | 0 | 0 |
| 2016 | 79 | 19 | 0 | 0 |
| 2017 | 92 | 23 | 0 | 0 |
| 2018 | 114 | 37 | 0 | 0 |
| 2019 | 81 | 17 | 0 | 0 |
| 2020 | 119 | 22 | 0 | 0 |
| 2021 | 124 | 30 | 0 | 0 |
| 2022 | 175 | 31 | 0 | 0 |
| 2023 | 108 | 20 | 0 | 0 |
| 2024 | 168 | 14 | 0 | 1 |
| 2025 | 158 | 25 | 0 | 0 |
| 2026 | 135 | 29 | 0 | 0 |