Aggregate-data-only interactions may be prior-driven
Not reliably, and the reason is structural. On an exact identity-link arm of 504 scenarios and a logit arm of 72, no threshold on contraction, on either effective-rank reading, or on the estimability screen separates scenarios whose 95% interval covers nominally from those that fail. The summaries are functions of Fisher information and the prior, so within one design they are exactly constant while coverage ranges from 0 to 0.96. They partly catch the failure they were designed for: contraction flags 59% of failures caused by a tight prior. They catch none of the 142 failures caused by a biased route, where the ecological route carries between-study confounding or the additivity route a synergy, because confounding moves the posterior mean and not its information. A clean contraction says the likelihood spoke, not that it was right. Every result is exploratory. Read the study below.
Statement
Interaction parameters informed only by aggregate data can be regularized into finite posterior intervals by the prior, so a proper credible interval is not evidence that the likelihood contributed anything. Implemented component PAIC already ships partial mitigation: plot_prior_posterior() compares each prior with its posterior, prior_sensitivity() quantifies how far target contrasts move when priors are tightened or loosened, and estimable_effects_at() separates no first-order information from IPD-supported and aggregate-only information. What is missing is a default numerical summary: prior-to-posterior contraction per interaction parameter and an effective likelihood rank, reported without the analyst having to ask.
Why it is open
A proper prior yields a proper posterior under ordinary integrability conditions, so the usual output gives no signal that the likelihood contributed nothing, and the parameters where this bites hardest are exactly the weakly identified interaction terms that a bridged component model depends on. The existing diagnostics are visual or require a deliberate refit, so an analyst who does not run them sees a clean interval with no warning attached. Prior-to-posterior contraction and effective likelihood rank are the summaries that would make this visible by default, and neither is computed.
What has been tried
| Work | What it contributes |
|---|---|
| cpaic 0.1.0 (development version, choxos/cpaic@9d150e9) | unreleased development R package on GitHub, not on CRAN; records prior specifications, provides plot_prior_posterior(), prior_sensitivity(), and estimable_effects_at(), documents that weakly identified interaction parameters are regularized by priors, and reports no numerical prior-to-posterior contraction or effective likelihood rank |
Probable solution or research direction
Promote the existing prior-versus-posterior comparison and prior-sensitivity refit from optional diagnostics to default reported output, and add numerical prior-to-posterior contraction for every interaction parameter plus an effective likelihood rank for the fitted model. Attach to each reported interaction the label of whether it is identified by IPD, by aggregate arms, or by neither, which the estimability screen already computes. Calibrate these summaries specifically to interaction parameters in a bridged component model, since the generic Bayesian workflow versions are already standard.
Our study
Prior-to-posterior contraction and effective likelihood rank measure information, not bias
The question put to it. CMP-14 asks for two default summaries, prior-to-posterior contraction per interaction and an effective likelihood rank, so an analyst sees when an aggregate-only interaction is held up by its prior. Would those summaries tell the analyst whether the reported interval covers?
What it found. Not reliably, and the reason is structural. On an exact identity-link arm of 504 scenarios and a logit arm of 72, no threshold on contraction, on either effective-rank reading, or on the estimability screen separates scenarios whose 95% interval covers nominally from those that fail. The summaries are functions of Fisher information and the prior, so within one design they are exactly constant while coverage ranges from 0 to 0.96. They partly catch the failure they were designed for: contraction flags 59% of failures caused by a tight prior. They catch none of the 142 failures caused by a biased route, where the ecological route carries between-study confounding or the additivity route a synergy, because confounding moves the posterior mean and not its information. A clean contraction says the likelihood spoke, not that it was right. Every result is exploratory.
Results.
- All four registered statistics (contraction, target-level likelihood ratio, whole-model effective rank, estimability screen) take values shared by failing and nominal scenarios on both arms, so no threshold on any of them separates the classes. On E1 there are 251 failing and 169 nominal scenarios; on E2, 41 and 12.
- On the identity link with known residual variance the posterior covariance does not involve the outcomes. Within one information state, spread, network size and prior, contraction, target ratio and effective rank are exactly constant across discordance and synergy levels while coverage ranges over 0.956.
- Decomposed by cause (post hoc), 109 E1 failures come from a prior of SD 0.1 pulling a true interaction of 0.4 toward zero and 142 from a biased route. Contraction flags 58.7% of the first and 0% of the second; target ratio 41.3% and 0%; effective rank 56.0% and 0%.
- Scenario pairs matched on spread, size and prior whose contraction differs by less than 0.02 differ in coverage by up to 0.951 on E1.
- On the logit arm the rules score far better on average (contraction Youden 0.805 against 0.220 on E1) because 34 of its 41 failures sit at the tight prior and it has only 12 nominal scenarios. The overlap persists.
- Three aggregate routes identify an interaction on the logit link, through differences in covariate means, covariate variances or baseline risk; only the first works on the identity link. None is randomized.
Design.
- Estimand
- The within-trial interaction of the target component, true value 0.4; coverage of its nominal 95% interval
- Methods compared
- prior-to-posterior contraction, target-level likelihood to prior precision ratio, whole-model effective likelihood rank, estimability screen, source survival (post hoc candidate)
- Data-generating mechanism
- Exact computation on a four-component network, one scalar covariate, twelve arms in every state. The target interaction is identified by its own IPD trial, by a combination arm under additivity, by the between-study contrast in aggregate covariate means, by differing covariate variances (logit only), or not at all. E1: identity link, known variance, 504 scenarios crossing state, covariate spread, network size, interaction prior SD, discordance and synergy. E2: logit link, asymptotic, 72 scenarios.
- Replicates
- Exact and asymptotic computation: 504 E1 scenarios and 72 E2 scenarios, no Monte Carlo replicates
What this does not answer. Nothing here is confirmatory: E1 was computed before its protocol existed, E2’s separation rules were rebuilt after its output was read, and the cause decomposition was added for the manuscript. E1 is exact for a design realized at its quadrature nodes and E2 is asymptotic, so no posterior was sampled and neither is an average over realized designs. The exact invariance of the summaries holds for known-variance Gaussian models; with estimated variances or random effects they would move with a departure and whether that movement could ever warn is untested. One target component, one covariate, one network geometry, a normal within-study covariate law on the logit arm, and departures acting on the mean only. The protocol went through thirteen rounds of critique; the manuscript has not been peer reviewed.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/CMP-14-aggregate-only-interactions · Protocol: registered before the run
What this project measured
Study studies/CMP-14-aggregate-only-interactions, decided by R/11-decision.R from the run’s own result files. 504 scenarios on the exact arm, 72 on the nonlinear arm.
Conclusion: no diagnostic in the panel separates failing from nominal scenarios, on either arm.
| arm | failing | nominal | neither | total |
|---|---|---|---|---|
| E1, exact | 251 | 169 | 84 | 504 |
| E2, nonlinear | 41 | 12 | 19 | 72 |
A scenario fails if coverage is below 0.90 and is nominal if coverage is within 0.01 of 0.95; gross overcoverage sits in neither class, being a real defect of an interval but not the defect this study measures.
All seven statistics overlap on both arms: contraction, target_ratio, eff_rank, prec_within, prec_between, surv_between and surv_sd each take at least one value shared by a failing and a nominal scenario. An overlap is an existence claim no weighting can move, so no threshold on any of them separates the two classes whatever mixture of scenarios a practitioner faces. All six state-separation rules likewise return no separation.
Standing: exploratory on both arms, and that is not a hedge. E1 ran before the protocol existed and every E2 rule was rebuilt after E2’s output had been read, so neither arm is confirmatory and the nonlinear arm is not the safer of the two. The result is a measurement whose grid and outcome definitions were chosen with earlier numbers already seen; a confirmatory version needs a fresh study with these outcomes registered in advance.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Interaction parameters informed only by aggregate data can be regularized into finite posterior intervals by the prior rather than by the likelihood. A finite credible interval may therefore reflect the prior rather than information in the data.”
What was wrong with it. The source said no diagnostic in standard output separates prior-driven from likelihood-driven interaction estimates. GPT-5.6 Sol found three at choxos/cpaic@9d150e9 (plot_prior_posterior(), prior_sensitivity(), estimable_effects_at()), so the gap narrows to their being optional rather than default and to the absence of numerical contraction and likelihood-rank summaries; it also corrected the wording, since a prior regularizes a posterior but never makes the likelihood identify a parameter.
- Literature and prior-art check
- open / supported-with-caveat. The statistical content is uncontroversial Bayesian bookkeeping: a proper prior always yields a proper posterior, so a finite credible interval is not evidence of likelihood information. I found no paper offering a routine prior-versus-likelihood diagnostic for effect-modifier interactions in ML-NMR or CNMA, and no source asserting that absence, so the ‘no diagnostic in standard output’ claims are unverifiable rather than established. Two caveats for the editor. The generic remedies named (prior-to-posterior contraction, prior predictive checks, prior sensitivity) are standard Bayesian workflow rather than a research gap, so the novel part is specifically calibrating them to interaction parameters in a bridged component model. And the claim about what cpaic’s documentation recognises cannot be checked: cpaic is not on CRAN and I found no public repository. Confidence is capped because I read no text bearing directly on this entry.
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / supported-with-caveat. C2-C4 are supported: a proper posterior interval can be dominated by prior regularization, cpaic has no numerical prior-to-posterior contraction table or effective-likelihood-rank report, and its documentation explicitly recognizes prior regularization of weakly identified Gamma. C1 should say prior-regularized or prior-driven posterior rather than likelihood identification: a prior does not make the likelihood identify a parameter. Proper-prior/proper-posterior language also requires ordinary posterior-integrability conditions rather than holding as an unrestricted theorem. The report understates existing mitigation: plot_prior_posterior() directly compares each prior and posterior, prior_sensitivity() quantifies contrast movement under changed priors, and estimable_effects_at() distinguishes IPD, aggregate, and absent first-order information. These are optional rather than default numerical contraction or likelihood-rank summaries, so the remaining work is useful but is a diagnostic-roadmap item, not automatically a very-high-severity correctness failure. Cites choxos/cpaic@9d150e9: Provides a direct visual diagnostic for a posterior that merely reproduces its prior.. Cites choxos/cpaic@9d150e9: Quantifies whether target contrasts move when interaction or component priors are tightened or loosened.. Cites choxos/cpaic@9d150e9: Separates no first-order information from IPD-supported and aggregate-only information before interpreting posterior contrasts..
- Decision path
-
R3:partially-solved(n=1) → R11:caveated(n=2)
How the evidence falls in time
2 papers in the reviewed corpus bear on this problem, published 2022 to 2026. Every paper found here asserts the problem is open; none reports progress.
| Year | Says | Paper |
|---|---|---|
| 2022 | confirms it is open | Bayesian models for aggregate and individual patient data component network meta-analysis |
| 2026 | confirms it is open | Analysing complex interventions using component network meta-analysis |
Source
Derived from C:L1368, C:L1327-1329, scope:field-level (Bayesian ML-NMR generally; reported for cpaic) of the reviewed corpus.