No review method for assembling transferable effect-modifier sets
Statement
Every population-adjusted analysis begins by assembling a candidate list of effect modifiers and prognostic variables from external evidence, and no method governs that step. Freitag, Gurskyte and Sarri 2023 reviewed guidance from payer and non-payer organizations, published indirect comparisons and prior submissions in two indications, and found the most explicit guidance, from NICE, recommends a literature review or structured discussion with clinical experts without saying how the review should be designed, which evidence types are admissible, what criteria establish that a modification finding transfers from another population to the trials being compared, or whether a hierarchy should govern the final selection. Practice matches the guidance: of 511 unique records screened, 17 described any selection process, and those rested on prior literature, trial subgroup analyses or clinical input, with no reference to systematic identification. The demand has since outrun the method: EU joint clinical assessment guidance now requires a second comprehensive review to identify potential effect modifiers, in addition to the review of clinical evidence, and consultation with clinicians experienced in the disease area. What does not exist is a review methodology scoped to identify modifiers rather than to estimate an effect: an admissibility rule covering randomized subgroup analyses, individual participant data meta-analyses of interactions, observational heterogeneity studies and mechanistic argument; a ranking or filtering rule over the resulting candidates; and above all a transferability criterion. Appraisers therefore have no basis on which to judge whether a submitted candidate list was assembled adequately, only whether the statistics applied to it were correct.
Why it is open
The candidate list bounds everything downstream: a variable absent from it can never be balanced, tested or entered into a sensitivity analysis, so an omission at this step is invisible to every later diagnostic, yet the step sits in evidence retrieval rather than in statistics and neither field has claimed it. The indirect comparison methods literature takes the list as given and studies what to do with it; systematic review methodology is built around estimating an effect, not around identifying which covariates modify one, and has no design for a review whose output is a variable list rather than a pooled estimate. Transferability is the hard part and is genuinely unsolved rather than merely unwritten, because a modification finding is itself specific to a population, an effect scale, a treatment version and a follow-up horizon, so importing it into a new comparison requires an argument of exactly the type the indirect comparison is trying to make; the criterion is circular unless something external, such as a mechanistic argument or invariance across several source populations, breaks the loop. Because no standard exists, the list is assembled in practice from prior submissions in the same indication and from whichever subgroup analyses the available trials happened to run, which selects on reporting rather than on relevance and systematically favors indications with a long appraisal history.
What has been tried
| Work | What it contributes |
|---|---|
| Freitag, Gurskyte and Sarri 2023, Increasing transparency in indirect treatment comparisons: is selecting effect modifiers the missing part of the puzzle?, Journal of Comparative Effectiveness Research 12 | Reviews guidance documents, published indirect comparisons and prior submissions in non-small cell lung cancer and psoriasis. Finds that guidance names literature review and expert opinion without specifying review design, admissible evidence, transferability criteria or a selection hierarchy, and that of 511 records screened only 17 reported any selection process, none of it systematic. |
| Phillippo, Ades, Dias, Palmer, Abrams and Welton 2018, Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal, Medical Decision Making 38:200-211 (the peer-reviewed account of NICE DSU TSD 18) | The guidance that names literature review and clinical expert discussion as the routes to identifying effect modifiers, and that is the target of the review’s finding that the route is named but not specified. |
| Phillippo et al. 2019, Population Adjustment Methods for Indirect Comparisons: A Review of NICE Technology Appraisals, International Journal of Technology Assessment in Health Care 35:221-228 | Documents what submissions actually do with population adjustment across NICE appraisals, including the reporting of covariate handling, which is the record against which a designed review protocol would be judged. |
| Aballea et al. 2026, Between Rigor and Relevance: Why the EU HTA Guidelines on Indirect Comparisons Miss the Mark, Journal of Market Access and Health Policy 14:30 | Records that EU joint clinical assessment guidance now requires a second comprehensive review to identify potential effect modifiers plus documented consultation with disease-area clinicians, and argues that these operational requirements are imposed without methodological evidence behind them. The obligation now exists in regulation while the method does not. |
Probable solution or research direction
Specify the review as a protocol-driven step with its own reporting standard: a question of the form which covariates modify the effect of treatment class X on outcome Y on scale Z, a search that covers randomized subgroup analyses, individual participant data meta-analyses of interactions and observational heterogeneity studies, and prespecified inclusion criteria stating what counts as evidence of modification rather than of prognosis. Add an explicit transferability appraisal for each imported finding, scoring the source’s population, treatment version, follow-up, outcome definition and effect scale against the trials being compared, and record the appraisal in the submission so it can be contested item by item. Pair the review with formal expert elicitation for candidates the literature cannot supply, and require the final adjustment set to be traceable to specific rows of the review with a stated reason for every inclusion and every exclusion. Pilot the protocol in an indication with a long appraisal history, where the list assembled from prior submissions can be compared directly against the list a designed review produces, and report how much they differ; that comparison is the evidence the new EU requirement currently lacks.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as well posed and upstream of statistical adjustment-set selection, noting that the cited review explicitly identifies missing methods for review design, evidence admissibility, transferability and hierarchical selection. The evidence is a single paper, but that paper is itself a systematic survey of guidance, published analyses and submissions across two indications, so the absence it reports is documented rather than asserted.
Source
Derived from reading:core-02 of the reviewed corpus.