Priors, not data, can drive posteriors in weakly identified models
At the thresholds the literature suggests, they miss most of it. Pairing each warning with whether that replicate’s own interval missed, a composite of four diagnostics has sensitivity 0.37 at a false-alarm rate of 0.16, falling to 0.13 outside two engineered geometries; the one sensitive rule fires on half the sound analyses too. Used as continuous scores the same statistics reach AUC 0.51 to 0.78, so this is a failure of thresholds rather than of the statistics. What no threshold repairs is that these diagnostics measure the prior’s influence while the harm comes from the prior’s location: with disconnected evidence and a tight prior four standard deviations from the truth, going from 100 to 400 per arm drops the warning rate from 0.98 to 0.01 while coverage stays near zero. Read the study below.
Statement
In population-adjusted network and bridge models a proper prior with a finite, positive marginal likelihood yields a proper posterior even along directions the likelihood cannot identify, so a narrow credible interval can reflect the prior rather than the evidence, and a clean R-hat does not rule that out. The diagnostics that would expose it already exist as general Bayesian workflow tools: power-scaling sensitivity (Kallioinen et al., arXiv:2107.14054, with the priorsense package on CRAN) and multinma’s plot_prior_posterior(), shipped since version 0.1.0. What is missing is their application here: the literature and recency auditors found no power-scaling or per-parameter contraction analysis of PAIC bridge or disconnected-network models, and no released ITC package computes a contraction measure or benchmarks a decision contrast against a prior-only fit.
Why it is open
The posterior blends prior and likelihood and nothing in a printed interval separates them, so a well-behaved posterior for a cross-gap contrast or an aggregate-only interaction is formatted exactly like an evidence-based estimate. Convergence diagnostics interrogate the sampler rather than the likelihood’s contribution, though the exemption is not total: Kallioinen et al. note that a case with both an uninformative prior and a noninformative likelihood will usually announce itself through fitting problems. The gap is wiring and practice rather than method: no released ITC package computes power scaling or prior-to-posterior contraction, so running either on a bridge model is bespoke work, and because prior dominance grows with scenario severity a single check on one benign dataset says nothing about the cases that matter.
What has been tried
| Work | What it contributes |
|---|---|
| Kallioinen, Paananen, Burkner and Vehtari 2024, power-scaling sensitivity for priors and likelihoods, Statistics and Computing (arXiv:2107.14054) | defines power-scaling sensitivity: prior sensitivity with little or no likelihood sensitivity is the signature of a likelihood that is noninformative relative to the prior; notes that a jointly uninformative prior and likelihood usually shows up as fitting problems instead |
| priorsense R package | implements power-scaling prior and likelihood sensitivity diagnostics for fitted Bayesian models; named by the recency auditor as available but not wired into ITC packages |
| multinma R package, plot_prior_posterior() since version 0.1.0 | plots prior against posterior for model parameters, a partial prior-versus-posterior facility that stops short of a numerical contraction measure or power scaling |
| cpaic, unreleased development package on GitHub (not on CRAN), commits 7563a64 and a99b718 | returns NA instead of a finite summary for contrasts the component design does not identify, and adds prior predictive checks plus refits under tighter and looser prior scales as an empirical identification diagnostic; it implements no formal power scaling, no numerical contraction measure, and no prior-only benchmark fit |
Probable solution or research direction
Apply the existing tools rather than build new ones. Run power-scaling sensitivity and per-parameter prior-to-posterior contraction on PAIC bridge, component and disconnected-network models, benchmark each decision contrast against a prior-only fit, and report contraction for the contrasts that carry the decision and not only for model parameters, since contraction is parameterization-dependent. Wire these into released packages so they stop being bespoke scripts, and evaluate them conditional on scenario severity rather than at one benign scenario.
Our study
Do prior-sensitivity diagnostics catch the analyses that are actually wrong?
The question put to it. In a population-adjusted network a proper prior yields a proper posterior even where the likelihood identifies nothing, so a narrow interval can reflect the prior rather than the evidence. The diagnostics that would expose this exist. What are their operating characteristics: do they flag the analyses whose intervals actually miss the truth, and how often do they cry wolf?
What it found. At the thresholds the literature suggests, they miss most of it. Pairing each warning with whether that replicate’s own interval missed, a composite of four diagnostics has sensitivity 0.37 at a false-alarm rate of 0.16, falling to 0.13 outside two engineered geometries; the one sensitive rule fires on half the sound analyses too. Used as continuous scores the same statistics reach AUC 0.51 to 0.78, so this is a failure of thresholds rather than of the statistics. What no threshold repairs is that these diagnostics measure the prior’s influence while the harm comes from the prior’s location: with disconnected evidence and a tight prior four standard deviations from the truth, going from 100 to 400 per arm drops the warning rate from 0.98 to 0.01 while coverage stays near zero.
Results.
- Prespecified verdict: diagnostics fail at the thresholds the literature suggests. Composite sensitivity 0.366 (MCSE 0.002), false alarm 0.158; 0.127 and 0.025 outside the engineered positive controls.
- Sensitivity and specificity trade off completely. Tight-and-loose prior refitting catches 0.83 of misses outside the controls and fires on 53% of covered replicates. Contraction, the prior-only benchmark and power-scaling keep false alarms low and catch almost nothing.
- The composite adds almost nothing over its best component: 0.366 against contraction’s 0.363, at a higher false-alarm rate. Requiring two of four rules to agree neither buys specificity nor recovers sensitivity.
- More data makes the warning quieter while the answer stays wrong. Disconnected evidence, tight prior, true interaction four prior SDs away: coverage near zero at both sample sizes, warning rate 0.98 at 100 per arm and 0.01 at 400. Contraction is what every diagnostic reads as reassurance.
- A first implementation measured power-scaling by posterior mean shift and found sensitivity 0.000 everywhere. That was our error: with a zero-centred prior, scaling the prior up and the likelihood down give identical posterior means, so the standard rule can never fire. Measured distributionally it works, and the verdict does not depend on the amendment.
- The structural rank screen is the only rule with no false alarms at all, and it is blind to every failure where the contrast is estimable and the answer is still wrong.
Design.
- Estimand
- Target-population marginal C versus B mean difference, and the conditional interaction coefficient
- Methods compared
- no-diagnostic baseline, structural rank screen, prior-to-posterior contraction, prior-only benchmark, power-scaling sensitivity, tight and loose refit, composite rule
- Data-generating mechanism
- A conjugate Gaussian population-adjusted network in which every piece of evidence reduces without loss to a normal observation, so the posterior and every power-scaled posterior are exact and no diagnostic is contaminated by MCMC noise. 120 scenarios crossing the evidence available about the weakly identified treatment, arm size, whether the truth agrees with the zero-centered prior, prior scale, and the target population. 2000 replicates each.
- Replicates
- 2000 per scenario, 240,000 datasets and 480,000 contrast evaluations
What this does not answer. The other half of CMU-02, wiring these diagnostics into released software, is untouched; neither multinma nor cpaic is run. The model is conjugate Gaussian with an identity link, a correctly specified linear mean and known variances, so nothing transfers directly to a non-conjugate posterior where MCMC error enters and the diagnostics behave differently. Only automated thresholds are evaluated; a plot read by an experienced analyst is a different instrument. Thresholds were taken from the source literature and not tuned. CMP-14 is touched only for the generic aggregate-only interaction, not for component models, and IDN-06 is not addressed at all because this design contains no within-versus-between discrepancy.
Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/CMU-02-prior-driven-posteriors · Protocol: registered before the run · Review: full history
What this project measured
Study studies/CMU-02-prior-driven-posteriors, run and decided against a rule registered before the run. 120 scenarios, 2000 replicates each, two contrasts; 480,000 replicate-contrasts, of which intervals missed on 56,210. Harmful scenarios: 34 of 240, and 18 of those lie outside the engineered positive controls, so the phenomenon is not only an artifact of the controls.
Conclusion: the diagnostics fail at the thresholds the literature suggests. Per replicate, pairing each warning with whether that replicate’s interval actually missed:
| rule | sensitivity | false-alarm rate |
|---|---|---|
| contraction | 0.363 | 0.141 |
| prior-only benchmark | 0.209 | 0.128 |
| power-scaling | 0.209 | 0.130 |
| tight-and-loose refit | 0.698 | 0.526 |
Excluding the engineered controls the picture is worse: contraction falls to 0.122 sensitivity, power-scaling to 0.007. The only rule with real sensitivity, the tight-and-loose refit, fires on more than half of all replicates.
The statistics are not worthless, the thresholds are. Threshold-free, contraction reaches AUC 0.711 and prior sensitivity 0.766 (0.749 and 0.781 excluding controls). The ranking information is there; the cutoffs in use do not exploit it.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “In population-adjusted network and bridge models a proper prior guarantees a proper posterior even for contrasts the likelihood cannot identify, so a narrow credible interval can reflect the prior rather than evidence. Prior predictive checks, power-scaling sensitivity, and per-parameter prior-to-posterior contraction are not routinely reported, so a reader cannot tell which quantities the data actually informed.”
What was wrong with it. The source implied the diagnostics themselves were absent. Power scaling (Kallioinen et al. 2024, arXiv:2107.14054; priorsense on CRAN) and multinma’s plot_prior_posterior() have existed for years, and the source-code auditor found that cpaic, an unreleased development package on GitHub and not on CRAN, already returns NA for structurally non-identified contrasts and ships prior predictive checks and tight/loose prior refits; the other two auditors found nothing equivalent in released ITC packages, and that disagreement stands. The propriety claim was also narrowed: a proper prior gives a proper posterior when the marginal likelihood is finite and positive, not unconditionally.
Dissent. literature dissented (open): I read the power-scaling paper in full (arXiv 2107.14054; Kallioinen, Paananen, Burkner, Vehtari, Stat Comput 2024) and the SBC test-quantities paper (arXiv 2211.02383). Kallioinen supports the diagnostic claim directly: prior sensitivity with little or no likelihood sensitivity is the signature of a likelihood that is noninformative relative to the prior. Modrak et al. independently corroborate the core worry: standard rank-based SBC ‘could never detect … when the posterior is equal to the prior’. Both are generic Bayesian-workflow tools, not ITC methods. I found no application of priorsense or power-scaling to PAIC bridge or disconnected-network models; PubMed returns nothing, and multinma’s R/ directory and NEWS contain no priorsense or power-scaling entry, so the tooling is genuinely not wired in. Caveats: (i) the absence claim cannot be verified, only unrefuted; (ii) ‘convergence diagnostics do not detect weak identification’ holds for a proper informative prior but not universally, since Kallioinen notes fully uninformative cases ‘will likely be detectable from model fitting issues’; (iii) multinma has shipped plot_prior_posterior() since v0.1.0, a partial prior-vs-posterior facility. codex dissented (overstated): C1-C3 are correct for ordinary Bayesian models with a finite, positive marginal likelihood: a proper prior can regularize a likelihood-null direction and produce finite credible intervals. A proper prior alone is not an unrestricted mathematical guarantee of posterior propriety. C4 is correct in the important one-way sense that good convergence does not establish identification, although weak identification can sometimes cause poor geometry. C5 is useful but contraction is parameterization-dependent and should also be assessed for decision contrasts. C6 measures sensitivity to power perturbations; it is not a definitive identification test. C7 is an unsubstantiated literature-practice claim and is stale as a package-capability claim because cpaic wires in related diagnostics. C8 and C9 are reasonable. At 9d150e9, cpaic returns NA for design-nonidentified contrasts, offers per-contrast tight/loose-prior refits, prior predictive fitting, and per-parameter prior/posterior plots. It does not implement formal prior or likelihood power scaling, a numerical contraction measure, or a benchmark against a separate prior-only fit for every contrast.
- Literature and prior-art check
- open / supported-with-caveat. I read the power-scaling paper in full (arXiv 2107.14054; Kallioinen, Paananen, Burkner, Vehtari, Stat Comput 2024) and the SBC test-quantities paper (arXiv 2211.02383). Kallioinen supports the diagnostic claim directly: prior sensitivity with little or no likelihood sensitivity is the signature of a likelihood that is noninformative relative to the prior. Modrak et al. independently corroborate the core worry: standard rank-based SBC ‘could never detect … when the posterior is equal to the prior’. Both are generic Bayesian-workflow tools, not ITC methods. I found no application of priorsense or power-scaling to PAIC bridge or disconnected-network models; PubMed returns nothing, and multinma’s R/ directory and NEWS contain no priorsense or power-scaling entry, so the tooling is genuinely not wired in. Caveats: (i) the absence claim cannot be verified, only unrefuted; (ii) ‘convergence diagnostics do not detect weak identification’ holds for a proper informative prior but not universally, since Kallioinen notes fully uninformative cases ‘will likely be detectable from model fitting issues’; (iii) multinma has shipped plot_prior_posterior() since v0.1.0, a partial prior-vs-posterior facility.
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1-C3 are correct for ordinary Bayesian models with a finite, positive marginal likelihood: a proper prior can regularize a likelihood-null direction and produce finite credible intervals. A proper prior alone is not an unrestricted mathematical guarantee of posterior propriety. C4 is correct in the important one-way sense that good convergence does not establish identification, although weak identification can sometimes cause poor geometry. C5 is useful but contraction is parameterization-dependent and should also be assessed for decision contrasts. C6 measures sensitivity to power perturbations; it is not a definitive identification test. C7 is an unsubstantiated literature-practice claim and is stale as a package-capability claim because cpaic wires in related diagnostics. C8 and C9 are reasonable. At 9d150e9, cpaic returns NA for design-nonidentified contrasts, offers per-contrast tight/loose-prior refits, prior predictive fitting, and per-parameter prior/posterior plots. It does not implement formal prior or likelihood power scaling, a numerical contraction measure, or a benchmark against a separate prior-only fit for every contrast. Cites choxos/cpaic@7563a6429c93f97120a9ae8107c3a62f6c9382e7: Prevents a finite posterior summary from being reported for a contrast that the component design does not identify.. Cites choxos/cpaic@a99b718a79eda5937dc0734cddfbe632022e277e: Adds prior predictive checks and refitting under tighter and looser prior scales for target contrasts..
- Grok 4.5, recency lens
- open / supported-with-caveat. The identification and presentation claims (proper priors yield finite posteriors without likelihood information; disconnected-network contrasts can be prior-driven; posteriors alone do not show estimability; R-hat does not detect weak identification) remain standard Bayesian results and were not overturned by 2024–2026 ITC literature. Power-scaling sensitivity (Kallioinen et al. 2024; R package priorsense on CRAN) and prior-to-posterior tools (including multinma plot_prior_posterior) exist as general Bayesian workflow pieces, so C5–C6 and C8 are methodologically available rather than uninvented. What the agenda correctly flags as open is routine application and package wiring for weakly identified PAIC/bridge/disconnected models, and evaluation conditional on scenario severity: searches across RSM/SiM/MDM/Value in Health, arXiv stat.ME, and maicplus/multinma/gemtc docs found no standard PAIC reporting of per-parameter contraction, power-scaling, or prior-only benchmarks for decision contrasts. Proposal of diagnostics is not solution of the ITC practice gap.
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
40 papers in the reviewed corpus bear on this problem, published 2012 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2012 | confirms it is open | Evidence Synthesis for Decision Making 2: A Generalized Linear Modeling Framework for Pairwise and Network Meta-analysis of Randomized Controlled Trials |
| 2012 | confirms it is open | Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons |
| 2013 | partly addresses it | Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials |
| 2013 | confirms it is open | Methods used to conduct and report Bayesian mixed treatment comparisons published in the medical literature: a systematic review |
| 2014 | confirms it is open | A Microsoft-Excel-based tool for running and critically appraising network meta-analyses: an overview and application of NetMetaXL |
| 2014 | confirms it is open | A design-by-treatment interaction model for network meta-analysis with random inconsistency effects |
| 2014 | confirms it is open | Network meta-analysis of multiple outcome measures accounting for borrowing of information across outcomes |
| 2016 | partly addresses it | Implementing informative priors for heterogeneity in meta-analysis using meta-regression and pseudo data |
| 2016 | confirms it is open | Meta-Analysis of a Complex Network of Non-Pharmacological Interventions: The Example of Femoral Neck Fracture |
| 2016 | confirms it is open | Two new methods to fit models for network meta-analysis with random inconsistency effects |
| 2017 | confirms it is open | Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models |
| 2017 | confirms it is open | Pharmacological interventions for alcoholic liver disease (alcohol-related liver disease): an attempted network meta-analysis |
| 2018 | partly addresses it | Graphs of study contributions and covariate distributions for network meta-regression |
| 2018 | confirms it is open | Component network meta-analysis identifies the most effective components of psychological preparation for adults undergoing surgery under general anesthesia |
| 2018 | confirms it is open | Validating Bayesian Inference Algorithms with Simulation-Based Calibration (Talts, Betancourt, Simpson, Vehtari, Gelman; arXiv:1804.06788v2) |
| 2018 | confirms it is open | A design-by-treatment interaction model for network meta-analysis and meta-regression with integrated nested Laplace approximations |
| 2018 | confirms it is open | Borrowing of strength from indirect evidence in 40 network meta-analyses |
| 2018 | confirms it is open | Incorporating external evidence on between-trial heterogeneity in network meta-analysis |
| 2019 | confirms it is open | An empirical comparison of Bayesian modelling strategies for missing binary outcome data in network meta-analysis |
| 2019 | confirms it is open | Network meta-analysis of rare events using the Mantel-Haenszel method |
| 2019 | confirms it is open | Participants’ outcomes gone missing within a network of interventions: Bayesian modeling strategies |
| 2019 | confirms it is open | Shared parameter model for competing risks and different data summaries in meta-analysis: Implications for common and rare outcomes |
| 2020 | confirms it is open | Interventions for adults with a history of complex traumatic events: the INCiTE mixed-methods systematic review |
| 2021 | confirms it is open | Supplementary appendix to Furukawa et al., Dismantling, optimising, and personalising internet cognitive behavioural therapy for depression: a systematic review and component network meta-analysis using individual participant data |
| 2021 | confirms it is open | Prior Choices of Between-Study Heterogeneity in Contemporary Bayesian Network Meta-analyses: an Empirical Study |
| 2022 | confirms it is open | Bayesian models for aggregate and individual patient data component network meta-analysis |
| 2022 | confirms it is open | Network meta-analysis of rare events using penalized likelihood regression |
| 2022 | confirms it is open | metapack: An R Package for Bayesian Meta-Analysis and Network Meta-Analysis with a Unified Formula Interface |
| 2023 | confirms it is open | Dismantling cognitive-behavioural therapy components for attention-deficit hyperactivity disorder in adolescents and adults: protocol for a network meta-analysis |
| 2023 | confirms it is open | A Bayesian (meta-)regression model for treatment effects on the risk difference scale |
| 2023 | confirms it is open | Synthesizing cross-design evidence and cross-format data using network meta-regression |
| 2023 | confirms it is open | The reporting quality and transparency of orthopaedic studies using Bayesian analysis requires improvement: A systematic review |
| 2024 | confirms it is open | Components and Delivery Formats of Cognitive Behavioral Therapy for Chronic Insomnia in Adults: A Systematic Review and Component Network Meta-Analysis |
| 2024 | confirms it is open | Network meta-analysis for an ordinal outcome when outcome categorization varies across trials |
| 2024 | confirms it is open | crossnma: An R package to synthesize cross-design evidence and cross-format data using network meta-analysis and network meta-regression |
| 2024 | confirms it is open | Twenty years of network meta-analysis: Continuing controversies and recent developments |
| 2025 | confirms it is open | Tipping point analysis in network meta-analysis |
| 2026 | confirms it is open | Creating Treatment and Component Hierarchies in Component Network Meta-Analysis |
| 2026 | confirms it is open | Anchors Away: Navigating Unanchored Indirect Comparisons with Multilevel Unanchored Meta-Regression (ML-UMR) (Chandler & Ishak; arXiv:2606.20341v1). The batch file points at media/media/Table1.pdf; the full LaTeX source mlumr.tex sits in the same arXiv package and was read. |
| 2026 | confirms it is open | Efficacy of selpercatinib as a first-line treatment for RET-fusion positive non-small-cell lung cancer: a novel two-stage Bayesian network meta-analysis |
Source
Derived from C:L214-215, C:L257, C:L1822, C:L1988, C:L2086-2089 of the reviewed corpus.