Risk-modeling network meta-regression treats a fitted risk score as observed
Little, given a prognostic cohort of about a thousand patients or more: propagating the score’s estimation uncertainty widened intervals by at most 2.4%, and plug-in coverage stayed near nominal. With 300 cohort patients the widening reached 7% at the risk tails against about 1% at the median, confirming that the understatement is worst where recommendations are made, and plug-in coverage fell to 0.924. The registered rule (plug-in coverage at least 0.93 everywhere) was missed in that one scenario, so the refuting sentence fails as registered. Read the study below.
Statement
Risk-modeling approaches to treatment effect heterogeneity fit a prognostic model for baseline risk in one dataset, optionally recalibrate it in the trial data, then enter the fitted risk score into a network meta-regression as a prognostic covariate and as an effect modifier. Where the stages are fitted separately the synthesis conditions on that score as though it were observed, so the sampling and shrinkage uncertainty of the prognostic model and of the recalibration never reaches the treatment-effect posterior. This is the generated-regressor problem arriving in evidence synthesis: standard errors computed as if an estimated regressor were data are too small, and the understatement grows where the regressor is least well estimated, which in risk modeling is the tails of the risk distribution and therefore exactly where risk-stratified treatment recommendations are made. The three-stage version that combines a cohort-based prognostic model, recalibration in the trials, and a network meta-regression on the resulting score names a single Bayesian model implementing all three stages as the way to incorporate uncertainty from all stages and avoid spuriously overprecise conclusions, and does not build it; no published risk-modeling meta-analysis or network meta-analysis does. The magnitude of the interval understatement has not been quantified for any published network, so the reported credible intervals on risk-stratified benefit are too narrow by an unknown amount.
Why it is open
Fitting the stages jointly requires a likelihood linking the prognostic cohort to the trials, and whether that joint model is identified depends on whether the same score means the same thing in both, which is a transportability assumption rather than a modeling detail; where it fails, the recalibration parameters carry the difference and there is no settled answer on whether they should be shared, stratified or partially pooled across trials. Two requirements then conflict. The prognostic cohort is usually far larger than the trials, so a fully joint fit lets a handful of trials influence the prognostic model in ways the analysis did not intend, which is the standard motivation for cutting feedback; cutting feedback is precisely what discards the uncertainty at issue. A bootstrap over the prognostic model is available in principle but has to be propagated through recalibration and a Bayesian network fit for every replicate, which is expensive enough that nobody has reported it. Because nobody has measured the understatement, there is also no demonstrated harm to motivate paying that cost.
What has been tried
| Work | What it contributes |
|---|---|
| Chalkou, Hamza, Benkert, Kuhle et al. 2024, Research Synthesis Methods 15(4):641-656 | The three-stage model: a prognostic model for baseline risk fitted in a large cohort, recalibrated for trial participants, then used as an effect modifier in a network meta-regression combining aggregate and individual randomized data. States that implementing the three stages in a single Bayesian model would naturally incorporate uncertainty from all stages and avoid spuriously overprecise conclusions, and stops short of doing so. |
| Chalkou, Steyerberg, Egger, Manca et al. 2021, Statistics in Medicine 40(20):4362-4375 | The two-stage predecessor, in which a baseline risk score from a prognostic model enters a network meta-regression as a single prognostic factor and effect modifier; establishes the design in which a fitted score is treated as a covariate. |
| Kent, Paulus, van Klaveren, D’Agostino et al. 2020, Annals of Internal Medicine 172(1):35-45 | The PATH statement, which establishes risk modeling as the recommended route to treatment effect heterogeneity and therefore drives the use of fitted risk scores as covariates in synthesis; it does not address how the uncertainty of the fitted score should propagate. |
| Pagan 1984, International Economic Review 25(1):221-247 | The generated-regressor result: when a regressor is an estimate from a first-stage model, conventional second-stage standard errors are wrong, and the correction depends on the first-stage covariance. This is the general statement of the failure and the reason it is not a small-sample artifact. |
| Plummer 2015, Statistics and Computing 25(1):37-43 | Cuts in Bayesian graphical models: a formal account of what happens when feedback from one module to another is cut, showing the cut distribution is not a posterior and has no general convergence guarantee. This is the trade-off any staged risk-modeling synthesis is implicitly making. |
Probable solution or research direction
Quantify the understatement first, because it is cheap and decides whether the rest is worth doing: refit a published risk-modeling network meta-regression with the prognostic model and recalibration bootstrapped, and report how much the credible intervals on risk-stratified treatment effects widen, separately at the median and at the tails of the risk distribution. Then write the joint model: one Bayesian specification in which the cohort likelihood, the recalibration and the network meta-regression share parameters, and state the conditions under which the treatment-by-risk interaction is identified when the cohort and the trials differ in population. Compare the joint fit against a cut-feedback fit and against the staged fit on interval width, calibration of the predicted benefit and stability of the recalibration parameters, so the cost of protecting the prognostic model from trial feedback is measured rather than assumed. Report the risk score with its own uncertainty in the output, so that a stratified recommendation at the tail can be seen to rest on a weakly estimated score.
Our study
An estimated risk score used as a regressor
The question put to it. When a risk score estimated in a prognostic cohort is used as a regressor for treatment-effect heterogeneity, does treating it as data understate the interval, and is the understatement worst at the tails of risk?
What it found. Little, given a prognostic cohort of about a thousand patients or more: propagating the score’s estimation uncertainty widened intervals by at most 2.4%, and plug-in coverage stayed near nominal. With 300 cohort patients the widening reached 7% at the risk tails against about 1% at the median, confirming that the understatement is worst where recommendations are made, and plug-in coverage fell to 0.924. The registered rule (plug-in coverage at least 0.93 everywhere) was missed in that one scenario, so the refuting sentence fails as registered.
Results.
- Widening ratio 1.001 to 1.024 with cohorts of 1000 or 5000; 1.011 to 1.073 with 300.
- Tails against median with 300: widening 1.065 to 1.073 against 1.011 to 1.012.
- Plug-in coverage 0.924 to 0.970; propagated 0.932 to 0.974.
Design.
- Estimand
- Treatment log odds ratio at the 10th, 50th and 90th percentiles of true risk
- Methods compared
- plug-in interval treating the score as data, interval adding bootstrap variance of the score
- Data-generating mechanism
- Logistic prognostic score from cohorts of 300 to 5000, used in a six-trial individual-data meta-regression with calibration drift; plug-in and bootstrap-propagated intervals for the treatment effect at the 10th, 50th and 90th risk percentiles. 6 cells, 500 replicates each.
- Replicates
- 500 per scenario, 3,000 in total, each with 20 bootstrap refits
What this does not answer. Pairwise six-trial analysis; frequentist bootstrap rather than a joint or cut Bayesian model; well-specified score. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/MOD-18-a-generated-regressor-worst-where-it-is-used · Protocol: registered before the run
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it, noting that the quoted passage directly acknowledges that the staged approach yields spuriously overprecise final inferences and identifies joint modeling as an unimplemented remedy. It rests on a single methods paper within evidence synthesis, although the generated-regressor result it instantiates is long established outside it.
Source
Derived from reading:nma-08 of the reviewed corpus.