Time, geography and setting are confounded in non-crossed evidence
Statement
When calendar period, geography and care setting are each constant within each of two non-crossed studies, all three are affine functions of the binary study indicator, so only one composite between-study difference is identified and a failed transport check cannot be attributed to a dimension. This is a property of non-crossed evidence rather than of indirect comparison as a whole: once a network carries crossed structure, network meta-regression and multilevel network meta-regression can carry region, period or setting as study-level covariates, which multinma supports. Separating the dimensions therefore requires crossed variation, external information or explicit identifying restrictions, and no estimator can do it by computation alone.
Why it is open
Assembling evidence with crossed structure, such as multi-regional or multi-period trials and registries spanning more than one care setting, is rarely done for an indirect comparison, so the standard two-study case carries the collinearity by construction. Without separation a failed transport check yields no targeted remedy, and a bias parameter elicited for undifferentiated study differences mixes mechanisms whose plausible magnitudes and directions differ, which degrades both elicitation and sensitivity analysis. The remedy is evidence design, measurement and explicit restriction rather than a better estimator, and tooling reflects that: in cpaic, an unreleased development package on GitHub that is not on CRAN, study intercepts and generic covariates are available but no calendar-time, geography or setting specific transport diagnostics are exported.
What has been tried
| Work | What it contributes |
|---|---|
| multinma R package | supports network meta-regression on study-level terms, including baseline risk, so region, period or setting can be carried as covariates once the network has enough crossed structure |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 | exposes study intercepts and generic covariates but exports no calendar-time, geography or care-setting specific transport diagnostics |
Probable solution or research direction
Design evidence sets with separately indexed transport dimensions, using multi-regional and multi-period trials plus real-world data to estimate dimension-specific drift, and report dimension-specific transport diagnostics so that failures are attributable. Where crossed structure is unavailable, state the identifying restriction that assigns the composite difference to a dimension instead of leaving it implicit, and elicit separate bias parameters for period, region and setting rather than one undifferentiated study effect. Since study-level regression already exists in released software, the work is assembling the crossed evidence and the diagnostics on top of it, not building a new estimator.
Verification
Verdict. Confirmed open. Auditors agreed the problem is open and no counterevidence surfaced.
The source titled this. “Calendar time, geography and setting are not separately identified” The title states the non-identification as a general fact, but both auditors narrowed it to non-crossed evidence, since crossed networks can carry these as study-level covariates.
The source said. “Transportability should be tested separately across calendar time, geography and care setting rather than conflated into a single study or source indicator, but standard practice indexes everything by a study identifier so these dimensions cannot be separated.”
What was wrong with it. The source stated the non-identification as a fact about ITC and PAIC practice in general. Both auditors narrowed it to non-crossed evidence in which each dimension is constant within a study, and the literature auditor noted that regression on study-level terms such as region or period is already available in multinma, so the missing qualifier belongs in the statement.
- Literature and prior-art check
- open / supported-with-caveat. I found no work resolving this and no work asserting the absence, so I am judging the claims on their merits. The load-bearing claim, that in a two-study indirect comparison calendar time, geography and care setting are not separately identified, is a straightforward collinearity fact: with two studies the study indicator is binary and every study-level covariate is a deterministic function of it, so no contrast can attribute a discrepancy to a dimension. That is defensible without citation. The caveat is scope. The title generalizes to ITC/PAIC as a whole, but the identification failure is specific to two-study (and more generally to non-crossed) evidence structures; network meta-regression and ML-NMR can and do carry study-level covariates such as region or period once the network has enough crossed structure, and multinma supports regression on study-level terms including baseline risk. The entry’s own why_open concedes this (‘separating them requires evidence with crossed structure’), so the fix is to move that qualifier into the title and statement rather than to change the vote. The remaining claims are descriptive of practice and I found nothing contradicting them.
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C1 and C4 are sound when all three dimensions are relevant to the target decision. C3 is a rank statement only under the missing qualifier that each dimension is constant within each of two studies: then every dimension is an affine function of the binary study indicator, so the design identifies at most one composite between-study contrast, not three dimension-specific effects. A two-study label alone does not imply failure if multiregional, multiperiod, site-level, or registry data provide crossed variation. C2 is a broad practice claim, not a mathematical necessity. At cpaic@9d150e9, study intercepts and generic covariates are available but no separate calendar-time, geography, or setting transport diagnostics are exported. No estimator can solve perfect collinearity without external restrictions; the open task is evidence design, measurement, and sensitivity analysis rather than computation. The asserted high severity is context-dependent.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
12 papers in the reviewed corpus bear on this problem, published 2011 to 2026. Every paper found here asserts the problem is open; none reports progress.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2011 | confirms it is open | Quantifying nonsteroidal anti-inflammatory drug (NSAID) risk: can combining the evidence with the use of a network meta-analysis provide a solution? |
| 2012 | confirms it is open | Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes |
| 2015 | confirms it is open | A primer on network meta-analysis with emphasis on mental health |
| 2015 | confirms it is open | Addressing risk of bias in trials of cognitive behavioral therapy |
| 2017 | confirms it is open | Can Matching-Adjusted Indirect Comparison Methods Mitigate Placebo Response Differences Among Patient Populations in Adjunctive Trials of Brivaracetam and Levetiracetam? |
| 2018 | confirms it is open | Estimating relative efficacy in acute postoperative pain: network meta-analysis is consistent with indirect comparison to placebo alone |
| 2021 | confirms it is open | Is the answer to clinical questions provided by Bayesian network meta-analysis? Regarding the optimal duration of DAPT and the best subsequent SAPT |
| 2022 | confirms it is open | Analysis of adaptive platform trials using a network approach |
| 2023 | confirms it is open | Impact of pharmacological interventions on intrapulmonary shunt during one-lung ventilation in adult thoracic surgery: a systematic review and component network meta-analysis |
| 2023 | confirms it is open | Placebo effects on all-cause mortality of patients with COVID-19 in randomized controlled trials of interleukin 6 antagonists: A systematic review and network meta-analysis |
| 2024 | confirms it is open | Assessing the relative efficacy of components of opioid-free anaesthesia in adult surgical patients: protocol for a systematic review and component network meta-analysis |
| 2026 | confirms it is open | Considerations for assessing the feasibility of network meta-analysis of seasonal vaccines |
Source
Derived from C:L226, C:L404 of the reviewed corpus.