No calibrated rule says when within- and across-trial interactions may be pooled
The difference’s standard error is sqrt(sW^2 + sA^2), not sA, so under equality the rule pools with probability at most 0.383 (0.31 to 0.37 in the networks studied): it refuses valid pooling most of the time in every regime. When it pools under ecological bias, the pooled interval can cover as little as 22%. Its RMSE matches never pooling to within 0.3%, forgoing the 2% to 19% gain pooling offers when the estimates agree. Read the study below.
Statement
An individual participant data network meta-analysis carries two sources of information about a treatment-covariate interaction: the within-trial contrast between participants in the same trial, and the across-trial association between trial-level covariate means and trial-level effects. Standard practice separates them and rests modifier claims on the within-trial part, because the across-trial part is identified only under an untestable assumption of no unmeasured trial-level confounding. Pooling the two buys precision, and the only published criterion for deciding whether they agree closely enough to be pooled is a visual comparison plus a rule that the within-trial estimate should lie within half a standard error of the across-trial estimate, described in its source as relatively strict and justified by pointing at interactions later debunked rather than by any calibration. No study reports the operating characteristics of that rule or of any alternative: not its power to detect a real discrepancy, not its false-positive rate, and not the coverage of the combined estimate conditional on having passed. The decision changes conclusions rather than sharpening them; in the worked example that accompanies the rule, pooling would have moved a chemotherapy-plus-surgery interaction from +0.172 to -0.120, reversing its sign. The problem is not confined to full individual-data networks, since any mixed-granularity model that lets aggregate covariate variation inform an interaction makes the same pooling decision, usually without making it explicitly.
Why it is open
The identifying assumption behind the across-trial component is not testable from the data being pooled, so agreement between the two estimates is evidence about that assumption in one direction only: disagreement can come from ecological confounding or from sampling noise, and agreement can occur under confounding whenever the bias is small next to the standard errors. Any threshold rule therefore has operating characteristics that depend on the between-trial variance and the number of trials, and both are small in exactly the networks where the extra precision is wanted. Calibrating a rule requires simulating networks in which the ecological bias is set and varied, which requires a generative model for trial-level confounding that this literature has not standardized. There is also no agreement on what a pooling rule should protect: a test tuned to preserve coverage of the interaction coefficient is not the same as one tuned to protect the downstream decision that uses the modifier to select patients or to transport an effect.
What has been tried
| Work | What it contributes |
|---|---|
| Freeman, Fisher, Tierney & Carpenter 2018, Research Synthesis Methods 9(3):393-407 | Framework for identifying treatment-covariate interactions in individual participant data network meta-analysis, separating within-trial from across-trial information and supplying the only published pooling criterion: agreement is taken to be shown if the within-trial interaction lies within half a standard error of the across-trial interaction. |
| Hua, Burke, Crowther, Ensor et al. 2017, Statistics in Medicine 36(5):772-789 | Shows that one-stage individual participant data meta-analysis models must separate within-trial from across-trial information to avoid ecological bias in treatment-covariate interactions, which is the step that creates the pooling decision this problem is about. |
| Riley, Debray, Fisher, Hattle et al. 2020, Statistics in Medicine 39(15):2115-2137 | Statistical recommendations for conduct and planning of individual participant data meta-analysis of treatment-covariate interactions, which set the default of relying on within-trial information; they do not supply a calibrated rule for when across-trial information may be added. |
| Berlin, Santanna, Schmid, Szczech et al. 2002, Statistics in Medicine 21(3):371-387 | Demonstrates that group-level meta-regression can misattribute an ecological association to individual-level effect modification, which is the harm a pooling rule has to prevent. |
| Fisher, Copas, Tierney & Parmar 2011, Journal of Clinical Epidemiology 64(9):949-967 | Critical review of methods for assessing patient-level interactions in individual participant data meta-analysis with guidance for practitioners; catalogues the estimation options without proposing an agreement test with known operating characteristics. |
Probable solution or research direction
Specify the decision as a test with a stated null and a stated alternative: the null that the within-trial and across-trial interaction estimate the same quantity, the alternative indexed by an ecological bias term. Simulate networks in which that bias term is set, crossing the number of trials, the between-trial variance of the covariate means, the covariate distribution overlap and the strength of the true within-trial interaction, and report for the half-standard-error rule and for candidate alternatives the rejection rate under the null, the power at each bias size, and the coverage and bias of the pooled interaction conditional on a pass. Compare threshold rules against continuous alternatives, such as a hierarchical model that shrinks the across-trial contribution toward the within-trial estimate with a data-driven weight, and against simply never pooling. Report the results in terms of the quantity the interaction is used for, namely the transported treatment effect in a target population, so that a rule can be chosen on decision cost rather than on interaction coverage alone.
Our study
The half-standard-error pooling rule for interactions is stated in the wrong standard error
The question put to it. The published rule pools within-trial and across-trial interaction estimates when they differ by less than half the across-trial standard error. What are its pass rate, the bias and coverage of what it reports, and how does it compare with a test stated in the difference’s own standard error?
What it found. The difference’s standard error is sqrt(sW^2 + sA^2), not sA, so under equality the rule pools with probability at most 0.383 (0.31 to 0.37 in the networks studied): it refuses valid pooling most of the time in every regime. When it pools under ecological bias, the pooled interval can cover as little as 22%. Its RMSE matches never pooling to within 0.3%, forgoing the 2% to 19% gain pooling offers when the estimates agree.
Results.
- Pass probability under equality: 0.314 to 0.374, bounded above by P(|Z| < 0.5) = 0.383 in any network.
- Coverage of the pooled interval given pooling falls with ecological bias: 0.72 at delta 0.1 and 0.22 at delta 0.2 with 20 trials and dispersed trial means, where the rule still pools in 6% and 0.04% of analyses.
- Averaged over its decisions, the rule’s RMSE is within 0.3% of never pooling in every configuration.
- Never pooling had the lowest worst-case RMSE over ecological bias 0 to 0.2 in five of six configurations; always pooling had it in one.
- An individual-level check of 2000 networks per configuration reproduced the standard errors, the pass probability (0.366 against 0.374 exact) and the conditional coverage (0.951 against 0.950).
Design.
- Estimand
- The within-trial treatment-covariate interaction
- Methods compared
- never pool, always pool, half-SE rule, size-5% test on the difference, size-20% test
- Data-generating mechanism
- Exact bivariate-normal computation of pass probability, bias, RMSE and coverage for never pooling, always pooling, the half-SE rule and tests on the difference at 5% and 20%, over 24 configurations: 5, 10 or 20 trials of 200, trial covariate means spread with SD 0.25 or 0.75, ecological bias 0 to 0.4.
- Replicates
- Exact computation; individual-level check of 2000 networks in each of four configurations
What this does not answer. Independent normal estimates with known standard errors; no between-trial heterogeneity; all trials with individual data. The ML-NMR case, where aggregate trials inform only the across-trial component, is not computed. No agreement rule can detect trial-level confounding that is small relative to the across-trial standard error. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/MOD-15-a-rule-stated-in-the-wrong-standard-error · Protocol: registered before the run
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as the distinct decision problem that begins once within-trial and across-trial information have been separated, noting that the quoted half-standard-error rule is explicitly heuristic. The evidence is a single methodological framework paper, which is a weaker basis than a multi-source proposal; the claim that no calibrated rule exists rests on the absence of any competing published rule.
Source
Derived from reading:nma-30 of the reviewed corpus.