Constancy assumptions are never tested across calendar time
Statement
Older-to-newer temporal holdout, fitting component and effect-modifier models on earlier trials and predicting a later one, is not performed in this literature, so there is no empirical estimate of how far constancy survives calendar-time change. What such a holdout tests is narrower than the source claims: it assesses joint out-of-time predictive transportability rather than equality of specific component or effect-modifier coefficients, and it requires the newer trial’s contrast to lie in the estimable row space of the training design with adequate covariate overlap. Prediction error against calendar gap also combines sampling error, ordinary between-study heterogeneity, design and outcome-definition changes and any genuine drift, so it does not by itself identify a decay rate or justify a universal trial-age cutoff. The claim that no published study has done this is an absence claim the literature auditor could not verify.
Why it is open
Constancy of component effects and effect modification is assumed across studies conducted decades apart, and calendar time bundles changes in standard of care, diagnostic criteria, outcome ascertainment and treatment versions, any of which can break it. Chronology is always recorded, so the holdout is uniquely feasible, yet it is not performed and no estimate of context-specific drift exists. Separating drift from sampling error, ordinary heterogeneity and design change needs repeated holdouts or a hierarchical temporal model rather than a single fit-and-predict exercise, which is more work than the design first appears to require.
What has been tried
No prior work is identified for this problem in the reviewed sources.
Probable solution or research direction
Run repeated older-to-newer holdouts on networks where the newer contrast is estimable from the earlier design and the estimands align, and report prediction error with its sources separated, using a hierarchical temporal model that distinguishes sampling error and ordinary between-study heterogeneity from systematic temporal drift. Report the results as context-specific drift estimates rather than a universal decay rate, and use them to inform drift priors in subnetwork-specific and hierarchical component models. Treat any inclusion-window recommendation as conditional on indication and outcome, since holdout prediction alone cannot justify a general trial-age cutoff.
Our study
Question. In repeated older-to-newer holdouts on real networks, does standardized prediction error grow with the calendar gap to the training evidence beyond what sampling error and ordinary heterogeneity explain, and does the raw error slope mistake those components for drift?
Design. Five aggregate networks with publication years from multinma, netmeta and metadat; random-effects NMA fitted to every trial published at least 0, 3, 6, 10 or 15 years before each held-out trial; slope of the standardized squared error on gap against each network’s own simulated no-drift null. 5 networks, 400 null and 200 drift replicates each.
The protocol is registered before any result is seen: read it.
Verification
Verdict. Partially addressed. Auditors agreed the problem is open and no counterevidence surfaced. Reopened by the full-text reading: 3 findings from 3 papers survived both the reader and an independent second reviewer, so the verdict moves from confirmed-open to partially-addressed.
The source said. “Temporal validation, fitting component and effect-modifier models on older trials and predicting a newer trial to test explicitly whether those coefficients survive calendar-time change, has not been carried out, so there is no empirical estimate of how fast constancy decays.”
What was wrong with it. The source presented temporal holdout as a direct test of coefficient constancy that would yield a decay rate and justify inclusion windows; GPT-5.6 Sol narrowed it to a test of joint out-of-time predictive transportability whose error confounds sampling variation, heterogeneity and design change, so isolating drift needs repeated holdouts or a hierarchical temporal model. The literature auditor recorded the absence claim as unverified rather than confirmed, since no source asserts the absence.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- confirmed-open -> partially-addressed 3 surviving findings from: Analysis of adaptive platform trials using a network approac; Estimating relative efficacy in acute postoperative pain: ne; From investigational product to active reference: evolution . New citations: 10.1080/20016689.2019.1603538, 10.1097/j.pain.0000000000001322, 10.1177/17407745221112001
- Literature and prior-art check
- open / supported-with-caveat. The mechanism claim is well supported by the wider meta-epidemiological literature, which documents that treatment effect estimates drift with calendar time: recursive cumulative meta-analyses show effects evolving over successive trials, and a meta-epidemiological survey of hundreds of meta-analyses reports that older randomized trials are associated with larger intervention effects. The logical claim, that fitting on older trials and predicting a newer one directly tests constancy of component and effect-modifier coefficients, is true by construction. The claim I cannot confirm is the absence one: ‘No published study tests whether component and effect-modifier constancy survives calendar-time change’ cannot be verified by failing to find one, and I found no source asserting the absence, so I mark that sub-claim unverified rather than confirmed. I retrieved the meta-epidemiological items at title and summary level only, which is why my confidence is at the abstract-reading level.
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C3 is technically plausible: calendar time can proxy changes in standard care, populations, measurement, and treatment versions. C1, however, overstates what temporal holdout prediction identifies. It tests joint out-of-time predictive transportability, not equality of specific component or effect-modifier coefficients. Prediction requires the newer trial’s component and interaction contrast to lie in the older training design’s estimable row space and requires adequate covariate overlap. Failure can arise from baseline-risk drift, outcome-definition changes, heterogeneity, or other misspecification; successful prediction does not prove coefficient constancy. Prediction error versus calendar gap also does not by itself identify a decay rate because it combines sampling error, ordinary heterogeneity, design changes, and potential temporal drift. Estimating drift requires repeated holdouts or a hierarchical temporal model that separates these sources. It cannot generally justify a universal trial-age cutoff. C2 is an uncited literature-absence assertion outside this audit. The work is a useful roadmap item, but not a uniquely decisive validation or an automatically High-severity omission.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
9 papers in the reviewed corpus bear on this problem, published 2014 to 2025. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2014 | confirms it is open | Placebo for depression: we need to improve the quality of scientific information but also reject too simplistic approaches or ideological nihilism |
| 2017 | confirms it is open | Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens Using an Information Theoretic Approach |
| 2018 | partly addresses it | Estimating relative efficacy in acute postoperative pain: network meta-analysis is consistent with indirect comparison to placebo alone |
| 2019 | partly addresses it | From investigational product to active reference: evolution of oral sumatriptan efficacy versus placebo for the treatment of acute migraine episodes and potential impact in comparative analyses |
| 2020 | confirms it is open | Assessing the Quality and Coherence of Network Meta-Analyses of Biologics in Plaque Psoriasis: What Does All This Evidence Synthesis Tell Us? |
| 2020 | confirms it is open | Visualizing the evolution of evidence: Cumulative network meta-analyses of new generation antidepressants in the last 40 years |
| 2022 | partly addresses it | Analysis of adaptive platform trials using a network approach |
| 2023 | confirms it is open | Immunonutrition in patients who underwent major abdominal surgery: A comprehensive systematic review and component network metanalysis using GRADE and CINeMA approaches |
| 2025 | confirms it is open | Disease-free survival as surrogate for overall survival in esophageal cancer: An individual patient data meta-analysis of neoadjuvant chemotherapy and chemoradiotherapy |
Source
Derived from C:L2111-2113 of the reviewed corpus.