Design and conduct differences survive baseline covariate balancing
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
Statement
Cross-trial differences in endpoint definition, follow-up schedule, adherence, rescue therapy and care pathway are properties of the trial rather than of the patient, so baseline covariate balancing in MAIC or STC cannot remove them, and a standard PAIC output does not separate residual protocol difference from residual patient-mix imbalance. Known design features are not beyond the reach of every estimator: they can enter study-level meta-regression, estimand or measurement models, and bias models, provided the network supplies enough cross-study variation to identify them. What stays nonidentifiable is the latent part and the split between design difference and patient mix. The bias-adjustment machinery the source proposes as new is mature: elicited, design-stratified priors for internal and external bias were published in Turner et al. 2009 (doi:10.1111/j.1467-985X.2008.00547.x), and threshold analysis quantifies how much study-level bias would have to be present to change a network’s recommendation (Phillippo et al. 2018, doi:10.1111/rssa.12341); neither has been ported to the population-adjusted setting.
Why it is open
The offending quantities have no patient-level distribution, so a weight that equalizes age and biomarker status leaves an imaging schedule or a rescue-therapy rule untouched, and separating the latent part of a design difference from residual patient-mix imbalance is not identified without further information or assumptions. Where a design feature is recorded and varies across enough studies it can be modeled, but PAIC networks are usually too small for that variation to exist, and these features are written up as narrative study characteristics rather than as variables. Adherence is a further complication because it is post-randomization: balancing baseline predictors can equalize it incidentally, while adjusting for it directly can change the estimand. Once baselines are balanced the analysis looks comparable, which makes the residue easy to overlook, and how much it matters is not established, since severity turns on whether the difference changes the estimand or the treatment as implemented.
What has been tried
| Work | What it contributes |
|---|---|
| Turner, Spiegelhalter, Smith and Thompson, 2009, Bias modelling in evidence synthesis, JRSS-A 172:21-47 | Constructs prior distributions from elicited opinion to represent internal bias (lack of rigour) and external bias (lack of relevance) in each study and performs a bias-adjusted meta-analysis; this is the hierarchical bias model with design-stratified priors that the source lists as a new direction. |
| Phillippo et al., 2018, Sensitivity of treatment recommendations to bias in network meta-analysis | Threshold analysis: quantifies how much study-level bias would have to be present before a network’s treatment recommendation changes, giving existing machinery for the design-feature direction the source proposes. |
Probable solution or research direction
Port the existing bias-adjustment apparatus rather than build it. Turner et al.’s elicited, design-stratified bias priors and Phillippo et al.’s threshold analysis are both defined for aggregate-data meta-analysis and NMA; what is missing is their combination with a weighting or standardization step and with an explicit target population. Design-feature meta-regression stays a reasonable route where the network holds enough studies to identify a study-level coefficient, and that replication, not the modeling, is the binding constraint. Negative-control endpoints are worth trying as a residual-heterogeneity signal, but there is no use of them in this field to build on, and a negative control can at best flag residual incompatibility; it cannot attribute that incompatibility to patient mix rather than to trial design.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. Cross-trial differences in endpoint definition, follow-up schedule, adherence, rescue therapy and care pathway are properties of the trial, not the patient, so baseline covariate balancing cannot remove them and standard PAIC output does not distinguish them from residual patient-mix imbalance. Bayesian bias-adjustment models with elicited, design-stratified priors exist and are well developed for meta-analysis and NMA (Turner et al. 2009), but have not been ported to the population-adjusted setting.
The source said. “Population adjustment balances baseline patient covariates, but cross-trial differences in endpoint definition, follow-up schedule, adherence, rescue therapy, and care pathway are properties of the trial rather than of the patient and remain once balancing is complete. Nothing in a standard PAIC output distinguishes residual patient-mix imbalance from residual protocol difference.”
What was wrong with it. The source presented hierarchical bias models with design-stratified priors as an emerging response and cited nothing; the literature auditor found them a seventeen-year-old evidence-synthesis literature, so the entry is now sourced to Turner et al. 2009 (doi:10.1111/j.1467-985X.2008.00547.x) and Phillippo et al. 2018 (doi:10.1111/rssa.12341) and the gap is their absence from PAIC, not from the literature. The same auditor could verify neither the claim that reviews find these sources under-handled in ITC and PAIC work nor any literature at all on negative-control endpoints in this field, and the code auditor struck the source’s assertion that these features cannot enter the estimator even when known.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- closure-upheld
- 27 findings from the reading assert this problem is open; the closure was re-examined and stands Put to an independent reviewer with the closing auditors’ reasoning, the work they relied on, and the new findings in view together. The new findings repeatedly establish the closure’s conceded residual: baseline-only MAIC and STC cannot adjust trial-level protocol, measurement, delivery, or care-pathway differences. They do not contradict Turner et al.’s established bias-adjustment framework or Phillippo et al.’s threshold-analysis machinery, and several findings explicitly acknowledge that alternative modeling requires additional information or assumptions. Empirical demonstrations that these differences matter, including the 2026 imaging-schedule finding, strengthen the residual but do not make the already-mature bias-model direction newly open. The closure correctly distinguishes an unresolved PAIC-specific identification and implementation problem from the overstated claim that the relevant bias machinery is new or absent. (reviewer confidence: high)
- Flagged for reopening
-
29 findings from the reading assert this problem is open while the registry carries
overstatedFlagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md. - Literature and prior-art check
- open / overstated. Retrieved Turner, Spiegelhalter, Smith & Thompson (2009), JRSS-A 172:21-47: ‘We present simple methods of allowing for differences in rigour (or lack of internal bias) and relevance (or lack of external bias) in evidence synthesis… Using elicited opinion, we construct prior distributions to represent the biases in each study and perform a bias-adjusted meta-analysis.’ Hierarchical bias models with design-stratified elicited priors are therefore a 17-year-old, mature evidence-synthesis literature, extended by Phillippo et al. (2018) threshold analysis and by the 2025 RoB NMA tool. Presenting them as an emerging idea is the overstatement. The core identification problem is genuinely open: design and conduct features have no patient-level distribution, so no weighting or standardization estimator can separate residual protocol difference from residual patient-mix imbalance; this is structurally untestable, so no paper can solve it. Two sub-claims fail verification. I found no review asserting that design and conduct heterogeneity is specifically under-handled in ITC/PAIC; the reviews I retrieved (Phillippo 2019 NICE TAs, Freitag 2023, van Beekhuizen 2025 EUnetHTA) concern covariate/effect-modifier selection and overall ITC suitability. And I found no literature at all on negative-control endpoints in ITC/PAIC, so that absence claim is unverifiable. Cites doi:10.1111/j.1467-985X.2008.00547.x: Supplies the real reference for the entry’s unsourced ‘Bayesian hierarchical bias models and design-stratified bias priors’ direction. It already exists for evidence synthesis; the gap is its absence from PAIC, not its absence from the literature.. Cites doi:10.1111/rssa.12341: Existing quantitative machinery for asking how much study-level bias would have to be present to change a network’s treatment recommendation; a real prior-art anchor for the ‘design-feature’ direction..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. The core limitation is real, but several claims exceed it. C2 and C4 are correct for baseline-only MAIC and STC: patient reweighting cannot change endpoint definitions, follow-up, rescue rules, care delivery, or latent study-level features. C3 is generally but not logically true because balancing complete baseline predictors could incidentally balance adherence; adherence itself is postrandomization, and direct adjustment may change the estimand or introduce bias. The statement that known design features cannot enter an estimator is false: they can enter study-level meta-regression, measurement or estimand models, and bias models when replication and identifying assumptions permit. Negative controls can signal residual incompatibility only under valid negative-control assumptions and cannot by themselves attribute it to patient mix versus trial design. C5 through C7 are prevalence or current-use claims with no cited source and are unverifiable in this offline technical audit. C8 is plausible but limited by study-level confounding and sparse replication. Very high severity is justified when the differences change the estimand or treatment implementation, not automatically for every conduct difference.
- Decision path
-
R5:default-open → R8:overstated-majority(n=2)
How the evidence falls in time
29 papers in the reviewed corpus bear on this problem, published 2015 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2015 | confirms it is open | Addressing risk of bias in trials of cognitive behavioral therapy |
| 2015 | confirms it is open | Heterogeneity, consistency and model fit should be assessed in Bayesian network meta-analysis |
| 2016 | partly addresses it | Using the contribution matrix to evaluate complex study limitations in a network meta-analysis: a case study of bipolar maintenance pharmacotherapy review |
| 2017 | confirms it is open | Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal |
| 2017 | confirms it is open | Can Matching-Adjusted Indirect Comparison Methods Mitigate Placebo Response Differences Among Patient Populations in Adjunctive Trials of Brivaracetam and Levetiracetam? |
| 2017 | confirms it is open | 5-Aminosalicylates to maintain remission in Crohn’s disease: Interpreting conflicting systematic review evidence |
| 2018 | confirms it is open | Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: Assumptions, models, effect scales, data scenarios, and implementation details (Nguyen, Ackerman, Schmid, Cole, Stuart, PLoS ONE 2018;13(12):e0208795) |
| 2018 | confirms it is open | Network meta-analysis in health psychology and behavioural medicine: a primer |
| 2019 | partly addresses it | Treatment sequence network meta-analysis in Crohn’s disease: a methodological case study |
| 2019 | confirms it is open | Acupuncture in sham device controlled trials may not be as effective as acupuncture in the real world: a preliminary network meta-analysis of studies of acupuncture for hot flashes in menopausal women |
| 2019 | confirms it is open | Using Matching-Adjusted Indirect Comparisons and Network Meta-analyses to Compare Efficacy of Brexanolone Injection with Selective Serotonin Reuptake Inhibitors for Treating Postpartum Depression |
| 2020 | confirms it is open | Summarising salient information on historical controls: A structured assessment of validity and comparability across studies |
| 2020 | confirms it is open | A Review of Two Regulatory Approved Anti-CD19 CAR T-Cell Therapies in Diffuse Large B-Cell Lymphoma: Why Are Indirect Treatment Comparisons Not Feasible? |
| 2020 | confirms it is open | Are Randomized Controlled Trials on Pharmacotherapy and Psychotherapy for Positive Symptoms of Schizophrenia Comparable? A Systematic Review of Patient and Study Characteristics |
| 2021 | confirms it is open | Methods for population adjustment with limited access to individual patient data: A review and simulation study |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2022 | confirms it is open | Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
| 2022 | confirms it is open | Two-stage matching-adjusted indirect comparison |
| 2022 | confirms it is open | Medical Therapy, Radiofrequency Ablation, or Cryoballoon Ablation as First-Line Treatment for Paroxysmal Atrial Fibrillation: Interpreting Efficacy Through the Shiny Method |
| 2023 | confirms it is open | Comparison of indirect treatment methods in migraine prevention to address differences in mode of administration |
| 2023 | confirms it is open | Letter to the Editor Regarding “Fluticasone Furoate/Umeclidinium/Vilanterol (FF/UMEC/VI) Triple Therapy Compared with Other Therapies for the Treatment of COPD: A Network Meta-Analysis” |
| 2024 | confirms it is open | Response to “Letter to the Editor Regarding ‘Matching-Adjusted Indirect Comparison of the Long-Term Efficacy Maintenance and Adverse Event Rates of Lebrikizumab vs. Dupilumab in Moderate to Severe Atopic Dermatitis’” |
| 2024 | confirms it is open | When does the placebo effect have an impact on network meta-analysis results? |
| 2024 | confirms it is open | Broad versus narrow research questions in evidence synthesis: A parallel to (and plea for) estimands |
| 2024 | confirms it is open | Efficacy and effectiveness of antipsychotics in schizophrenia: network meta-analyses combining evidence from randomised controlled trials and real-world data |
| 2024 | confirms it is open | Impact of sedative and appetite-increasing properties on the apparent antidepressant efficacy of mirtazapine, selective serotonin reuptake inhibitors and amitriptyline: an item-based, patient-level meta-analysis |
| 2025 | confirms it is open | Network connectivity, between-study heterogeneity and timepoint challenges in generalized myasthenia gravis: a feasibility assessment of indirect treatment comparisons |
| 2026 | confirms it is open | Can We Trust PAICs in Rare Diseases? Methodological Challenges and Limitations |
| 2026 | confirms it is open | Considerations in reviewing network meta-analyses of heterogeneous clinical trial designs: a methodological review in Crohn’s disease and ulcerative colitis |
Source
Derived from A:L61, C:L214-215, C:L233 of the reviewed corpus.