Model and bridge selection uncertainty never enters the interval

Missing data & uncertainty propagation
High priority
Partially addressed
When the same data select a PAIC model or bridge and ordinary conditional intervals are then reported, nominal coverage is generally not guaranteed, although the distortion can run in either direction and a prespecified or externally chosen model does not create the problem at all. Model averaging as the response is established in evidence synthesis generally (Rover, Wandel and Friede 2019; Bartos et al., arXiv:2110.01076) but has not been applied to PAIC, and predictive stacking needs a held-out unit aligned with the prediction target and dependence structure, so patient, study, and contrast level prediction are different estimands to choose between rather than an ambiguity that blocks the method. For bridges the difficulty is narrower than the source claimed: likelihood-based weights can be formed when candidate bridges imply different observed-data likelihoods or predictions, and are unavailable only for observationally equivalent identifying assumptions, which need prior elicitation or sensitivity analysis instead. No PAIC software feeds model or bridge averaging into the reported effect interval; in cpaic, an unreleased development package on GitHub that is not on CRAN, loo() is observation-level and warns that it does not validate cross-gap prediction, while prior_sensitivity() and bridge_fragility() remain diagnostics. The same failure occurs one step earlier, at the treatment partition. Whether doses, formulations, schedules, devices, class members or control conditions become one node or two is settled before any model is fitted, on clinical judgment or on a deviance comparison among candidate groupings, and the alternatives that fit almost as well are neither reported nor propagated into the interval or the ranking. Model fit cannot arbitrate that choice, because candidate partitions are not nested in a way that makes deviance comparable across different data configurations and because splitting can disconnect the network, which is a structural rather than a statistical objection. The same defect appears wherever a study’s contribution is reconstructed from an assumed nuisance parameter rather than reported: most published crossover trials give no precision estimate that respects the paired design, so a meta-analyst must impute a within-participant correlation, use first-period data only, or exclude the trial, and whichever is chosen is then conditioned on as known. Across 83 crossover trials assembled for one network, 19 (23 percent) reported a paired precision estimate and 50 (60 percent) supplied a point estimate and precision usable without further assumptions, falling to 26 (31 percent) once a washout period was also required (doi:10.1371/journal.pone.0133023), so both the eligible set and the imputed correlation move the contributed weights, and no study has reported how far pooled effects or rankings shift between the resulting eligible sets.

MIS-04 Partially addressed High priority Promising

ImportantA study has been run against this problem

For choices among MAIC and STC specifications the candidates are close: choosing the most precise method covered 0.938 to 0.956, like a prespecified one. Choosing the most favorable estimate shifted it by up to 0.26 on the log odds ratio scale while its interval still covered 0.927 or more, because the favored method often had a wide interval. Averaging with between-model variance over-covered (up to 0.991) because the candidates share data; that difference makes the refuting sentence fail as registered. Read the study below.

Statement

When the same data select a PAIC model or bridge and ordinary conditional intervals are then reported, nominal coverage is generally not guaranteed, although the distortion can run in either direction and a prespecified or externally chosen model does not create the problem at all. Model averaging as the response is established in evidence synthesis generally (Rover, Wandel and Friede 2019; Bartos et al., arXiv:2110.01076) but has not been applied to PAIC, and predictive stacking needs a held-out unit aligned with the prediction target and dependence structure, so patient, study, and contrast level prediction are different estimands to choose between rather than an ambiguity that blocks the method. For bridges the difficulty is narrower than the source claimed: likelihood-based weights can be formed when candidate bridges imply different observed-data likelihoods or predictions, and are unavailable only for observationally equivalent identifying assumptions, which need prior elicitation or sensitivity analysis instead. No PAIC software feeds model or bridge averaging into the reported effect interval; in cpaic, an unreleased development package on GitHub that is not on CRAN, loo() is observation-level and warns that it does not validate cross-gap prediction, while prior_sensitivity() and bridge_fragility() remain diagnostics. The same failure occurs one step earlier, at the treatment partition. Whether doses, formulations, schedules, devices, class members or control conditions become one node or two is settled before any model is fitted, on clinical judgment or on a deviance comparison among candidate groupings, and the alternatives that fit almost as well are neither reported nor propagated into the interval or the ranking. Model fit cannot arbitrate that choice, because candidate partitions are not nested in a way that makes deviance comparable across different data configurations and because splitting can disconnect the network, which is a structural rather than a statistical objection. The same defect appears wherever a study’s contribution is reconstructed from an assumed nuisance parameter rather than reported: most published crossover trials give no precision estimate that respects the paired design, so a meta-analyst must impute a within-participant correlation, use first-period data only, or exclude the trial, and whichever is chosen is then conditioned on as known. Across 83 crossover trials assembled for one network, 19 (23 percent) reported a paired precision estimate and 50 (60 percent) supplied a point estimate and precision usable without further assumptions, falling to 26 (31 percent) once a washout period was also required (doi:10.1371/journal.pone.0133023), so both the eligible set and the imputed correlation move the contributed weights, and no study has reported how far pooled effects or rankings shift between the resulting eligible sets.

Why it is open

Selection happens on the same data used for estimation, so the reported interval is conditional on a choice it does not price. The standard remedy needs a predictive unit at which the stacking weights are meaningful, and in a network the choice among patient, study, and contrast has to be made deliberately as a statement of the prediction target rather than inherited from a default. Projection predictive selection controls predictive loss and does not by itself deliver valid post-selection intervals, and robust Bayes covers only the model and prior classes actually represented, so neither is a finished answer and neither has been tried in PAIC. For bridges that are observationally equivalent no fit statistic can separate them, and the only honest input is a prior on assumption plausibility, which nobody has elicited.

What has been tried

Work What it contributes
Rover, Wandel and Friede 2019, Statistics in Medicine (locator not supplied by the auditor) heavy-tailed mixture priors for effect estimation in meta-analysis, which effectively result in a model-averaging technique robust to prior-data conflict; established for meta-analysis, not applied to population-adjusted comparisons
Bartos et al., Bayesian Model-Averaged Meta-Analysis in Medicine Bayesian model averaging over meta-analytic models so that model uncertainty enters the reported estimate; again for meta-analysis rather than for population-adjusted or network settings
cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 provides prior_sensitivity() and bridge_fragility() diagnostics and an observation-level loo() that warns it does not validate cross-gap prediction; no model or bridge averaging enters the reported effect interval
Efthimiou et al. 2016, Research Synthesis Methods (GetReal review) States that no recipe exists for deciding whether similar-but-not-identical interventions become one node or two, that the choice is left to clinical judgment plus goodness-of-fit comparison between candidate structures, and that as included studies adopt increasingly different node definitions transitivity becomes less plausible and inconsistency or heterogeneity rises.
Welton et al. 2015, Medical Decision Making Compares five treatment-definition models for one dataset and finds the deviance information criterion spread narrow, 168.6 to 180.5, while the resulting decision changes; resolves the choice by expert judgment and states that the approach can help determine whether grouping is reasonable only with clinical input. The closest thing to a fit-based arbitration of node granularity, and it shows fit does not arbitrate.
Mills et al. 2013, BMJ Survey of the effects of excluding treatments from network meta-analyses, reporting node-level effect changes up to 3.64-fold and reordering of the top three treatments in half of 18 networks. Explicitly declines to question the clustering of regimens into nodes by the original analysts and names that clustering as an additional, unquantified source of variability.
Ishaq et al. 2023, European Spine Journal Delphi consensus study built specifically to group conservative neck-pain treatments into network nodes, which records that no generally recommended procedure exists for grouping similar-enough treatments into nodes and that several different methods could therefore be followed.
Zhang et al. 2026, Frontiers in Medicine Prespecifies three node layers, 17, 5 and 4 nodes, for a single sedation dataset and reports that the numerically best treatment in the finest layer was a sparse variant supported only by indirect evidence, with interventions at different conceptual levels risking inflated treatment hierarchies.
Ades et al. 2024, Research Synthesis Methods Reviews twenty years of network meta-analysis controversies and identifies lumping over doses and co-treatments as an avoidable source of quantitative heterogeneity, so the concern is not confined to one clinical area.
Li, Yu, Hawkins and Dickersin 2015, Design, Analysis, and Reporting of Crossover Trials for Inclusion in a Meta-Analysis, PLOS ONE 10:e0133023 Measures the exposure across 83 crossover trials eligible for one network: 19 (23 percent) report a precision estimate that accounts for the pairing, 50 (60 percent) supply a point estimate and precision usable without further assumptions, 26 (31 percent) if a washout period is also required, and 16 (19 percent) report first-period results separately, so most crossover edges in such a network rest on an assumed within-participant correlation.
Elbourne, Altman, Higgins, Curtin, Worthington and Vail 2002, Meta-analyses involving cross-over trials: methodological issues, International Journal of Epidemiology 31:140-149 The standard reference for including crossover trials in a synthesis, setting out the options of imputing a within-participant correlation, using first-period data only, or excluding the trial, and treating the imputed correlation as a fixed input rather than as an estimated quantity carrying uncertainty.
Curtin, Elbourne and Altman 2002, Meta-analysis combining parallel and cross-over clinical trials. II: Binary outcomes, Statistics in Medicine 21:2145-2159 Derives the reconstruction of a crossover trial’s contrast and standard error from reported summaries under an assumed correlation, the calculation on which the assumption is operationalized; the resulting standard error is then conditioned on as known.
Curtin 2017, Meta-analysis combining parallel and crossover trials using generalised estimating equation method, Research Synthesis Methods 8 Combines aggregated results from crossover and parallel trials through a generalized estimating equation with a marginal estimation approach and can adjust for a carryover effect; it is a fixed-effects pairwise method, states that it is limited by the level of detail crossover reports usually give, and has not been carried into a network or evaluated for what it does to rankings.

Probable solution or research direction

The machinery exists outside PAIC, so the task is to port and test rather than to invent. Apply Bayesian model averaging as practiced in meta-analysis to population-adjusted networks, and state which predictive unit the stacking weights target, since that choice defines the estimand. Test whether projection predictive selection and robust Bayesian analysis over model and prior classes actually restore coverage here rather than assuming they do. For bridges, build likelihood-based weights where candidates imply different observed-data predictions, reserve elicited priors over assumption plausibility for the observationally equivalent case, and feed existing diagnostics such as bridge fragility into the reported interval instead of reporting them beside it.

Our study

Choosing among population-adjustment models on the same data

The question put to it. When the adjustment model is chosen from the analyzed data, does the chosen model’s interval keep its coverage, and what does averaging over candidates change?

What it found. For choices among MAIC and STC specifications the candidates are close: choosing the most precise method covered 0.938 to 0.956, like a prespecified one. Choosing the most favorable estimate shifted it by up to 0.26 on the log odds ratio scale while its interval still covered 0.927 or more, because the favored method often had a wide interval. Averaging with between-model variance over-covered (up to 0.991) because the candidates share data; that difference makes the refuting sentence fail as registered.

Results.

  • Registered verdict FAILS, through the averaged interval’s coverage differing from the prespecified one’s by more than 0.03.
  • Selection by smallest SE: coverage 0.938 to 0.956, bias as the prespecified linear STC.
  • Selection by most favorable estimate: bias -0.046 to -0.259, coverage 0.927 to 0.960.
  • Equal-weight average with added between-model variance: coverage 0.949 to 0.991, the widest intervals.

Design.

Estimand
Marginal log odds ratio, B versus A, in the target (DIA-08)
Methods compared
prespecified linear STC, prespecified MAIC on means, selection by smallest SE, selection by most favorable estimate, equal-weight model averaging
Data-generating mechanism
Re-scoring of DIA-08’s stored replicates (16 scenarios x 2000) under prespecified, precision-selected, result-selected and averaged use of four adjustment methods.
Replicates
32,000 stored replicates from DIA-08 re-scored

What this does not answer. Outcome and weighting models only; treatment partitions and bridges not studied; equal weights. Peer review has not been done.

Peer review. Not yet done. The result stands on the study alone until it is.

Read it: PDF · Markdown · OpenDocument · Code and data: studies/MIS-04-selection-over-models-bridges-and-partitions · Protocol: registered before the run

Verification

Verdict. Partially addressed. Auditors agreed the problem is open and no counterevidence surfaced. Reopened by the full-text reading: 1 finding from 1 paper survived both the reader and an independent second reviewer, so the verdict moves from confirmed-open to partially-addressed, on a single finding.

The source said. “Results are reported from one selected model or one selected bridge as if the selection step had not happened, so the interval omits the variability that the selection itself contributes.”

What was wrong with it. The source said reporting a single selected model understates uncertainty and that likelihood-based weights for bridge selection are not available at all. The code auditor narrowed both: post-selection distortion can run in either direction and does not arise for prespecified or externally chosen models, and likelihood-based weights do exist when candidate bridges imply different observed-data likelihoods or predictions, failing only for observationally equivalent assumptions; the literature auditor added that the entry carried no prior work despite an established model-averaging literature in evidence synthesis (Rover, Wandel and Friede 2019; Bartos et al., arXiv:2110.01076), and found no projection-predictive or robust-Bayes application to PAIC.

Revised by the full-text reading. Applied full-text reading.

merged-in
absorbed a proposal the reading raised separately: No criterion or uncertainty propagation for treatment-node granularity 6 citation(s) added. The drafter disagreed with placing it here: Four of the five readers had this accepted as a distinct problem and only one was judged a merge, and the two objects are not the same. MIS-04 concerns selection among statistical models and bridges fitted to a fixed data configuration, whereas the node partition determines which contrasts exist, which studies are usable and whether the network is connected at all, so candidate partitions cannot be compared by deviance or by any within-model criterion and averaging over them is not defined. The merged material also carries two demands MIS-04 does not make: a criterion stating when two arms may share a node, and a diagnostic for the equality that a merge asserts. It is recorded as a merge as directed.
merged-in
absorbed a proposal the reading raised separately: Crossover-trial reconstruction uncertainty is not propagated in NMA 4 citation(s) added. The drafter disagreed with placing it here: The merge is workable but the host is arguably wrong. MIS-04 is about selection among competing models and bridges using the analyzed data, whereas an imputed within-participant correlation is a fixed nuisance input used to reconstruct one study’s precision, which is the species of defect DEC-11 already names in its title as reconstruction error and OUT-13 treats for a different data source, digitized Kaplan-Meier curves. Recommend routing the addition to MIS-04 as the reviewer decided but cross-linking DEC-11 and OUT-13, since a reader looking for crossover reconstruction will look for it under reconstruction rather than under model selection.
Verdict changed
confirmed-open -> partially-addressed 1 surviving finding from: Simplifying fractional polynomials in Bayesian network meta-. New citations: 10.57264/cer-2025-0126
Literature and prior-art check
open / supported-with-caveat. The entry carries no prior_work at all, which understates a mature adjacent literature. Model averaging as a response to selection uncertainty in evidence synthesis is established: Rover, Wandel & Friede (2019), Stat Med, describe ‘heavy-tailed mixture priors for effect estimation in meta-analysis, which effectively results in a model-averaging technique… robust in the sense that a potential prior-data conflict… is explicitly anticipated’; also Bartos et al., ‘Bayesian Model-Averaged Meta-Analysis in Medicine’ (arXiv 2110.01076). So ‘Reporting a single selected model understates uncertainty’ is supported but generic, not a PAIC finding. I could not retrieve any source stating the second claim, that model averaging in network settings requires choosing a valid predictive unit for stacking weights; it is defensible on general grounds about leave-one-out stacking in hierarchical data, but I found nobody writing it for NMA, so treat it as the authors’ own argument rather than a sourced one. I found no projection-predictive or robust-Bayes application to PAIC. The bridge half is the genuinely novel and correct part: candidate bridges are competing identifying assumptions rather than nested models, so likelihood-based weights do not exist and averaging would need a prior on assumption plausibility. I found no work doing that.
GPT-5.6 Sol, technical and source-code lens
open / overstated. C1 is too categorical. If the same data select a model and ordinary conditional intervals are reported, nominal coverage is generally not guaranteed, but distortion may be in either direction; externally chosen or prespecified models do not create this post-selection problem. C2 is correct for predictive stacking: the held-out unit and dependence structure must match the intended prediction task. It is not a requirement of every form of model averaging, and patient-, contrast-, or study-level prediction are different estimands rather than an irresolvable ambiguity. C3 is plausible as a research direction, but projection predictive selection controls predictive loss rather than automatically delivering valid post-selection intervals, and robust Bayes addresses only the model or prior classes actually represented. Likelihood-based bridge weights are not universally unavailable: they can be formed when bridge models imply different observed-data likelihoods or predictions. They cannot learn the plausibility of observationally equivalent identifying assumptions, for which prior elicitation or sensitivity analysis is appropriate. At cpaic 9d150e9, loo() is explicitly observation-level and warns that it does not validate cross-gap prediction; prior_sensitivity() and bridge_fragility() provide diagnostics, but no model or bridge averaging enters the reported effect interval. The interval problem therefore remains open, but it is decision-critical mainly after material data-adaptive selection or when conclusions are sensitive to the bridge.
Decision path
R5:default-open → R9:substantiated-single-overstatement:codex

How the evidence falls in time

19 papers in the reviewed corpus bear on this problem, published 2013 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.

This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.

Year Says Paper
2013 confirms it is open Evidence synthesis for decision making 3: heterogeneity - subgroups, meta-regression, bias, and bias-adjustment
2013 confirms it is open Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials
2013 confirms it is open Graphical tools for network meta-analysis in STATA
2015 confirms it is open Accounting for Heterogeneity in Relative Treatment Effects for Use in Cost-Effectiveness Models and Value-of-Information Analyses
2015 confirms it is open Network meta-analysis combining individual patient and aggregate data from a mixture of study designs with an application to pulmonary arterial hypertension
2016 confirms it is open Assumptions of Mixed Treatment Comparisons in Health Technology Assessments - Challenges and Possible Steps for Practical Application
2018 confirms it is open Assessing the consistency assumptions underlying network meta-regression using aggregate data
2019 confirms it is open Creating historical controls using data from a previous line of treatment - two non-standard approaches
2019 confirms it is open A comparative review of network meta-analysis models in longitudinal randomized controlled trial
2020 confirms it is open Bayesian splines versus fractional polynomials in network meta-analysis
2021 confirms it is open Penalized Poisson model for network meta-analysis of individual patient time-to-event data
2022 confirms it is open A dose-effect network meta-analysis model with application in antidepressants using restricted cubic splines
2022 confirms it is open Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: Application to a melanoma network
2023 confirms it is open Model selection for component network meta-analysis in connected and disconnected networks: a simulation study
2024 confirms it is open Generalized Fused Lasso for Treatment Pooling in Network Meta-Analysis
2025 partly addresses it Simplifying fractional polynomials in Bayesian network meta-analysis via variable powers
2025 confirms it is open Methods for information-sharing in network meta-analysis: Implications for inference and policy
2025 confirms it is open NMAsurv: An R Shiny application for network meta-analysis based on survival data
2026 confirms it is open Efficacy of selpercatinib as a first-line treatment for RET-fusion positive non-small-cell lung cancer: a novel two-stage Bayesian network meta-analysis

Source

Derived from C:L214-215, C:L252 of the reviewed corpus.