No valid selection criterion for cut points and powers in time-varying survival NMA
Statement
Survival network meta-analyses that relax proportional hazards are chosen by a search whose outcome the usual criteria cannot defend. For piecewise or time-partitioned models the number and placement of cut points cannot be selected by DIC or AIC at all, because changing the intervals changes the data configuration the likelihood is evaluated on, so fit statistics for two candidates are not comparable quantities. For fractional polynomials the analyst fits up to 8 first-order and 36 second-order power combinations and reports the winner, so the number of models tried acts as unreported hidden parameters. The shortcut that makes the search affordable, selecting by frequentist AIC and then refitting the winner in Bayesian form, has never been checked for agreement with a Bayesian selection on the same data. In the absence of a statistical criterion, published practice falls back on prior knowledge, transparency, visual comparison against the observed curves and compatibility with the economic model, which is a statement that no criterion is available rather than a solution. Neither the interval on the reported time-varying contrast nor the extrapolated survival curve that carries the cost-effectiveness result reflects the search that produced the model.
Why it is open
The cut-point obstacle is structural rather than computational: a likelihood built from interval-level event and exposure counts is a different likelihood for each partition, so information criteria that presuppose a fixed data configuration are not merely biased across candidates but undefined. A criterion that could compare them has to be defined on something invariant to the partition, such as the predictive density of the observed event times or a proper scoring rule on the survival function, and nothing in the applied literature has been written in those terms. Carrying the search into the reported uncertainty is harder here than in ordinary post-selection inference, because the target is not a coefficient but an extrapolated survival curve beyond the observed follow-up, where the candidates differ most and the data are silent, so a selection-adjusted interval would be widest exactly where it cannot be checked. Model averaging is the obvious alternative, but averaging over candidates that disagree about extrapolation produces a predictive band whose usefulness for a reimbursement decision has never been tested against current practice of choosing one model and defending it clinically.
What has been tried
| Work | What it contributes |
|---|---|
| Freeman, Cooper, Sutton & Crowther 2022, Statistical Methods in Medical Research 31(5):839-861 | Applies the competing non-proportional-hazards network models to a melanoma network and states the obstacle directly: comparing models with differing time intervals is not straightforward because the choice of intervals cannot be guided by model fit statistics, since the data to which the models are fitted change when the intervals change. Also records that agreement between frequentist AIC-based selection and Bayesian selection is unestablished, and falls back on non-statistical selection criteria. |
| Ouwens, Philips & Jansen 2010, Research Synthesis Methods 1(3-4):258-271 | Network meta-analysis of parametric survival curves, which introduced the multi-parameter treatment effect whose functional form now has to be selected. |
| Jansen 2011, BMC Medical Research Methodology 11:61 | Fractional polynomial network meta-analysis of survival data, the model class in which the analyst searches over first- and second-order power combinations and reports the best-fitting one. |
| Freeman & Carpenter 2017, Research Synthesis Methods 8(4):451-464 | Bayesian one-step individual participant data network meta-analysis with Royston-Parmar models, a flexible spline alternative that accommodates time-dependent effects without cut points; it requires individual data, which is what the aggregate networks facing this problem lack, and it still requires choosing the number and position of knots. |
| Spiegelhalter, Best, Carlin & van der Linde 2002, Journal of the Royal Statistical Society Series B 64(4):583-639 | Introduces DIC, the criterion applied networks use to choose among survival models; its comparability assumption is exactly what fails when the cut-point configuration changes the data the likelihood is evaluated on. |
| Berk, Brown, Buja, Zhang & Zhao 2013, Annals of Statistics 41(2) | Valid post-selection inference: intervals that remain valid over all models a search could have selected. The general machinery for pricing a model search, developed for linear models with a fixed design and not extended to survival syntheses whose candidates are fitted to differently constructed data. |
Probable solution or research direction
Define a selection criterion on a quantity that does not change with the partition: the leave-one-trial-out or leave-one-arm-out predictive density of the observed event times, or a proper scoring rule on the predicted survival function evaluated on a common time grid, so that piecewise models with different cut points and fractional polynomials of different orders can be scored against each other on the same footing. Check the frequentist-to-Bayesian shortcut empirically by running both selections on the same simulated and real networks and reporting how often they choose the same model and how much the extrapolations differ when they do not. Price the search by carrying it into the reported quantity, either by repeating the whole selection inside a bootstrap or by Bayesian model averaging over the candidate set with weights from the partition-invariant criterion, and report the resulting interval on the extrapolated survival difference and on the derived incremental cost-effectiveness ratio rather than on a coefficient. Publish the candidate set and the scores for every model tried, so the count of models fitted stops being an unreported parameter.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as sufficiently distinct from general post-selection inference in mechanism, model class and decision-relevant extrapolation. It rests on a single applied methods paper, which states both halves directly: fit statistics cannot guide the cut-point choice because the data change with the intervals, and the agreement of frequentist with Bayesian selection is unestablished.
Source
Derived from reading:nma-15 of the reviewed corpus.