Studies

Closing the catalog, one problem a week

The catalog records what is open. This is the attempt to close some of it: one study a week, each aimed at a single entry, taken in priority order.

Scope is population-adjusted indirect comparison only, meaning MAIC, STC, ML-NMR, ML-UMR and NMI. Standard network meta-analysis is out of scope because it does not adjust for population differences, which is where these problems live.

Completed

ADJ-04 — Weighted conformal prediction keeps its marginal guarantee under shift by returning infinite intervals where the source has no data

Question. Does weighted conformal prediction give valid intervals for target patients under covariate shift, including where the source has no data?

Answer. No. Weighted conformal intervals kept marginal coverage over the target (0.904 to 0.919 at nominal 0.9), but beyond the source’s largest covariate value they covered 0.70 at a small shift and 0.83 at a moderate one, and at the largest shift they covered 0.99 only because 83% of those intervals were infinite. The method makes the support problem visible; it does not solve it. Estimating the density ratio changed nothing.

What it does not answer. Prediction intervals for individual outcomes rather than a marginal treatment effect; one covariate and one outcome model; no BART or Gaussian process arm; no support diagnostic scored as a classifier. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

ADJ-09 — An invariance screen for a bridging assumption, scored as a classifier

Question. Does instability across observed trials predict failure of a bridge used in a gap with no data, and does a screen on marginal effects fire under a valid bridge through non-collapsibility?

Answer. A stability screen across observed trials cannot confirm a bridge and refutes it rarely. With 4 or 10 trials, drift visible across them that biased the gap estimate by 0.28 was flagged in only 20% to 24% of analyses, and drift confined to the gap never above the null rate; passing the screen did not improve coverage of the reported analyses. Screening marginal log odds ratios raised false abstention under a valid bridge from 0.11 to 0.19 to 0.24 through non-collapsibility; screening conditional or standardized effects removed that.

What it does not answer. One event-rate level, no measurement shift, Cochran’s Q only; environments valid by construction. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

ADJ-10 — Modifier discovery across trials: no penalty is both stable and powerful, and study-level heterogeneity creates a false modifier that more trials entrench

Question. Are effect modifiers discovered from pooled trial data stable and correct enough to use, and do study-level confounders create false modifiers that more data do not remove?

Answer. No. Lasso discovery at the minimum-error penalty found the true modifiers but added 1.7 to 2.8 false ones among seven null covariates, and sets from independent datasets agreed with Jaccard similarity 0.37 to 0.57; the one-standard-error rule was stable only by selecting nothing (a single modifier found in 3% to 5% of analyses). With an unmeasured study-level cause of effect heterogeneity, pooled discovery selected the covariate whose trial means tracked it in 69% of analyses with 3 trials and 93% with 12 at the same total size, and raised false discoveries among the other covariates from 1.7 to 2.8 per analysis to 3.6 to 3.9; trial-specific treatment effects removed both.

What it does not answer. Lasso only (no multi-study causal forests); continuous outcome; linear modification; one overlap level; post-selection inference for the target not studied here. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

ADJ-15 — Target moments released under differential privacy

Question. When a target trial releases covariate moments under differential privacy, how much does ignoring the added noise cost MAIC coverage at realistic arm sizes and budgets, does propagating its known variance repair it, and when do extra released moments stop paying for their share of the budget?

Answer. Privacy noise is not negligible at usable budgets for trial-sized targets: at epsilon 1 and 150 patients it was 8% to 60% of the estimate’s variance and ignoring it cut coverage to 0.88 to 0.94; at 40 patients to 0.73. Its variance is known exactly, and adding it restored coverage in every scenario, conservatively where noise dominated. Releasing second moments pays only at larger budgets; at small ones it made MAIC infeasible in up to 87% of replicates.

What it does not answer. Laplace mechanism only; Gaussian mechanism, released correlations and overlap not varied. Results are conditional on feasible weights. Budget choice and governance are outside the study. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

ADJ-17 — An encoding that reconstructs 91% of the covariates’ variance dropped the effect modifier entirely

Question. Can a covariate encoding learned to reconstruct the covariates drop an effect modifier while reconstructing them well, and would a declared mapping or the full covariate set avoid the resulting transport bias?

Answer. No. With the treatment effect modified along a low-variance direction of 20 covariates, STC on five principal components reconstructed 91% of the covariate variance and lost the whole modification: bias -0.20 at a 0.5 SD target shift and -0.40 at 1 SD, coverage 0.49 and 0.02. Ten components (95% reconstructed) still lost most of it. STC on all covariates or on a declared modifier score was unbiased with nominal coverage in every cell, so reconstruction quality does not certify an encoding for transport.

What it does not answer. Principal components only; no treatment-effect-preserving or nonlinear encoder, nonlinear heterogeneity or measurement shift; continuous outcome. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-03 — An omitted component interaction biases never-administered regimens and leaks into regimens that do not contain it

Question. When additivity makes a never-administered combination estimable, how far can an omitted component interaction move its predicted effect, and does it contaminate other regimens?

Answer. No. At a synergy of 0.2 (twice a contrast’s SE), 15 of 32 never-administered regimen results that the additive model reported as estimable had 95% coverage below 0.90, down to 0.11. Synergy on a never co-administered pair went wholly into the regimens containing it and was undetectable; synergy on a co-administered pair was absorbed into the component effects and biased regimens without that pair by -0.125, including a never-administered one. Modeling co-administered pairs’ interactions removed the leakage. The share of a regimen’s pairs never co-administered did not flag the failures (AUROC 0.37).

What it does not answer. One connected network, continuous outcome, fixed effect; no disconnection, covariate adjustment, selection procedures or formal synergy bound. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-06 — Miscoding the shared component of a bridged network costs identifiability, not accuracy; miscoding an internal component biases the bridged estimate

Question. When the component coding of a disconnected network bridged by a shared component is wrong, does the error perturb the estimate or change what is estimable, and do sensitivity or probabilistic coding give honest intervals?

Answer. Partly. When the bridge is real, coding the shared component as a distinct agent removed that trial’s bridging information and, when it hit both bridging trials (up to 9.5% of replicates), made the target non-estimable, which a rank screen detects exactly; otherwise the estimate stayed unbiased with nominal coverage. The miscoding that left the row space intact, dropping an internal component, biased the target by about 0.8 SE and cut coverage to 0.83 given a miscoding. Probabilistic coding restored coverage (0.967 or more); coding sensitivity did too but was the whole line whenever a plausible coding broke the bridge.

What it does not answer. One hand-built network, fixed-effect contrast-level fits, no covariate adjustment, synergy or Bayesian arm; independent miscoding within a small plausible set. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-11 — Pooling standardized and unstandardized edges biases a component effect by exactly the untransported edges’ share of the population difference

Question. How large is the bias from pooling edges standardized to the target with edges left in their own populations, and is the two-stage route still the better practical choice than carrying every edge to the target?

Answer. No. The two-stage bias equaled the aggregate edges’ share of the pooled precision times the difference between their own-population and target effects (slope 0.97, R^2 0.997), zero only when modification or separation was zero. At 0.5 SD separation it reached -0.106 with coverage 0.27, and transporting every edge to the target had lower RMSE in all six such cells (0.038 to 0.089 against 0.047 to 0.113). Pooling conditional rather than marginal individual-data edges added up to 0.033 on the log odds ratio scale.

What it does not answer. Eight trials of one component rather than a component network; no one-stage component ML-NMR; aggregate covariate laws known; one network size. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-12 — Splitting a multi-arm trial’s adjusted contrasts is conservative for one contrast and anticonservative for another

Question. When a three-arm trial’s population-adjusted contrasts share an arm and a weight vector, what is lost by treating them as independent, is the error ever anticonservative, and is dropping the trial cheaper?

Answer. Ignoring the within-trial covariance errs in a direction set by the contrast, not by how the arms’ effect modification aligns. The trial’s own B-versus-C interval became conservative in every scenario, while the network A-versus-B interval, which borrows through the trial, fell below nominal in 15 of 30 scenarios. The shared-weighting part of the covariance took both signs but never reversed a contrast’s error. Dropping the trial widened the network interval by 29% to 91%.

What it does not answer. Continuous outcome, linear modification, one three-arm trial in a fixed two-trial aggregate network; four-arm trials, the one-stage ML-NMR comparator and a bootstrap are not run, and target moments are fixed. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-13 — Separating within-trial from across-trial component effect modification

Question. Component ML-NMR fits one shared interaction on a globally centered covariate, which algebraically forces the causal within-trial interaction to equal the confounded across-trial association. How much does that cost, when, and is the damage really caused by aggregate data as the catalog claims?

Answer. Fitting one shared interaction matrix on a globally centered covariate is not pooling: for a binary modifier with study prevalence p_s it is algebraically identical to imposing that the causal within-trial coefficient equals the confounded across-trial one. In the model simulated, coverage of a nominal 95% interval for the causal interaction falls from 93.8% when that happens to be true to 17.2% when the two are equal and opposite, and further in networks whose contrasts move two components at once. Separating them costs about 5% in interval width and removes an order of magnitude of bias. The catalog’s stated cause is incomplete: the loss is present with all twelve trials supplying individual data, so aggregate data are not necessary for it, but removing individual data roughly doubles the bias, so they amplify it.

What it does not answer. The registered global analysis is uninformative and the regime split that produced a usable answer was prompted by observing that failure, so it is not prespecified. The general headline about component ML-NMR is not established: the likelihood and nuisance structures are not mapped onto the fixed- and random-effect specifications the implemented method offers, and free study-by-treatment effects or priors can absorb or alter the term the identity isolates. A maximum-likelihood analogue of component ML-NMR. Implemented ML-NMR is Bayesian and priors matter most where interactions are weakly identified, so the coverage numbers do not transfer to multinma; the algebra does, and therefore so does the direction. One binary modifier, a log link, a Poisson outcome, additive components with no synergy, no random treatment-effect heterogeneity, no inconsistency, and exactly reported study prevalences. Under the discordance patterns the split model is correctly specified, so the study measures the cost of an algebraically known restriction rather than establishing that real networks contain within/across disagreement or how large it typically is. Continuous or multiple correlated modifiers, misspecified aggregate covariate distributions, and time-to-event outcomes are untested. It does not touch IDN-05.

Read it: PDF · Markdown · OpenDocument

CMP-14 — Prior-to-posterior contraction and effective likelihood rank measure information, not bias

Question. CMP-14 asks for two default summaries, prior-to-posterior contraction per interaction and an effective likelihood rank, so an analyst sees when an aggregate-only interaction is held up by its prior. Would those summaries tell the analyst whether the reported interval covers?

Answer. Not reliably, and the reason is structural. On an exact identity-link arm of 504 scenarios and a logit arm of 72, no threshold on contraction, on either effective-rank reading, or on the estimability screen separates scenarios whose 95% interval covers nominally from those that fail. The summaries are functions of Fisher information and the prior, so within one design they are exactly constant while coverage ranges from 0 to 0.96. They partly catch the failure they were designed for: contraction flags 59% of failures caused by a tight prior. They catch none of the 142 failures caused by a biased route, where the ecological route carries between-study confounding or the additivity route a synergy, because confounding moves the posterior mean and not its information. A clean contraction says the likelihood spoke, not that it was right. Every result is exploratory.

What it does not answer. Nothing here is confirmatory: E1 was computed before its protocol existed, E2’s separation rules were rebuilt after its output was read, and the cause decomposition was added for the manuscript. E1 is exact for a design realized at its quadrature nodes and E2 is asymptotic, so no posterior was sampled and neither is an average over realized designs. The exact invariance of the summaries holds for known-variance Gaussian models; with estimated variances or random effects they would move with a departure and whether that movement could ever warn is untested. One target component, one covariate, one network geometry, a normal within-study covariate law on the logit arm, and departures acting on the mean only. The protocol went through thirteen rounds of critique; the manuscript has not been peer reviewed.

Read it: PDF · Markdown · OpenDocument

CMP-15 — Calibrating the correlation does not fix the copula: the family moves a standardized contrast by up to 0.03 on the log odds and log hazard scales

Question. Once a Gaussian copula’s correlation is calibrated to the reported correlations, can the remaining error from copula family (asymmetry and tail dependence) move a target-standardized contrast?

Answer. Yes, modestly on the log odds and log hazard scales and strongly on the log rate scale. With margins and correlation matched, the calibrated Gaussian erred by up to 0.034 on the log odds ratio and 0.032 on the log cumulative hazard ratio (above 0.02 in 5 of 16 primary cells), with opposite signs for lower- and upper-tail dependence, and by up to 0.63 on the log rate ratio. The error plateaued by 256 integration points. A wrong flexible family was worse than the Gaussian in 61% of comparisons, so the defensible report is an envelope over families.

What it does not answer. Population-level; three exchangeable covariates with common margins; no mixed or discrete margins, vines or reconstruction uncertainty. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-16 — Sharing one heterogeneity parameter across a bridged network cost little; the interval method mattered more

Question. When two subnetworks of a bridged network have different between-study heterogeneity, does a shared tau give the bridged contrast valid intervals, and do stratified or shrunk alternatives do better at realistic study counts?

Answer. Largely yes for a contrast that sums both subnetworks. With one subnetwork five times as heterogeneous, shared, stratified and shrunk heterogeneity models covered within 0.016 of each other (shared 0.906 to 0.932 and stratified 0.917 to 0.943 across 3 to 12 studies per subnetwork), because a shared tau overstates one part’s variance and understates the other’s. The undercoverage came from ignoring uncertainty in tau: Hartung-Knapp intervals covered 0.97 to 0.997 at three studies, at about 1.5 times the width.

What it does not answer. A contrast drawing on one subnetwork only would carry the shared tau’s error without the offset; frequentist REML without priors; no residual-variance arm or IPD. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-18 — Adding time-constant component hazard ratios mispredicted a combination’s RMST when effects were delayed or follow-up differed

Question. When two components have different time profiles of effect and their trials differ in follow-up, does adding their time-constant hazard ratios predict the combination’s RMST, and does a piecewise extension fix it?

Answer. No. With both component effects delayed, adding time-constant hazard ratios overstated the combination’s 3-year RMST gain by 0.055 years (truth 0.405, coverage 0.88); a piecewise model reduced the bias to 0.017. With delayed and waning components and follow-up of 1.5 versus 3 years, the time-constant prediction was biased by -0.102 (coverage 0.82) and the piecewise model by -0.072 (coverage 0.91), because neither can learn a late effect the shorter trial never observed. With equal follow-up and effects summing to a constant, the time-constant model was accurate.

What it does not answer. Two single-component trials rather than a component network; no disconnection or population adjustment; one piecewise extension; one horizon. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-20 — The aggregate Poisson likelihood is biased by the exposure-rate covariance, and exposure-weighted covariate moments remove the bias

Question. The aggregate Poisson likelihood multiplies an arm’s total exposure by its mean modeled rate, which is exact only when exposure and rate are uncorrelated. The error is the factor 1 + rho CV(T) CV(lambda) per arm. How much bias does that put on a transported rate ratio at plausible levels of differential follow-up, and do exposure-weighted covariate summaries remove it?

Answer. No, the covariance cancels only when it is common to both arms. The bias followed the identity log{(1 + rho1 CV1(T) CV1(lambda)) / (1 + rho0 CV0(T) CV0(lambda))} (slope 0.994); at plausible dispersions with a differential correlation of at least 0.2, 27 of 64 cells had material bias or undercoverage, with bias up to 0.14 on the log rate ratio and coverage down to 0.17. Integrating aggregate arms over exposure-weighted covariate moments removed the bias everywhere (largest 0.008). Even with zero correlation the unweighted likelihood undercovered at large dispersions (0.84), because the finite-sample covariance is not zero.

What it does not answer. One covariate, one lognormal exposure law, no overdispersion, the same exposure mechanism in both studies; the likelihood is the study’s own code, not multinma. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-21 — An MNAR shift moved an anchored transported contrast in a direction that differed from dataset to dataset, but the tipping set was almost always one interval

Question. When an MNAR shift in imputed effect-modifier values enters a transported anchored contrast through several routes, is the contrast monotone in the shift, and is the tipping point a single value?

Answer. Partly. The tipping set was a single interval in at least 96% of replicates (non-single 3.0% to 4.4%, under the registered 5%), so a single tipping point sufficed here. But the routes offset only on average: the mean slope of the anchored contrast against the MNAR shift was -0.004 to 0.030, while per dataset its SD was 0.10 to 0.14 and it was positive in 48% to 62% of datasets. The direction in which an MNAR departure moves the contrast was not predictable from the design, so sensitivity analyses must explore both directions.

What it does not answer. Two anchored individual-data trials rather than a disconnected component network; one imputation per shift; congenial imputation; one MNAR mechanism; tipping set defined by the point estimate’s sign. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-22 — Which trials supply individual data, and what selection on it does

Question. When the trials that supply individual data are selected on observed trial variables or on their own effect modification, how biased is the interaction evidence, does combining with aggregate trials help, and can a leave-IPD-out diagnostic reveal it?

Answer. Selection of which trials supply individual data on an observed trial variable caused no bias, but selection on the trials’ own effect modification biased the within-trial interaction evidence by up to 0.14 (true 0.3), with coverage down to 0.59. Combining with the across-trial association in all trials halved the bias. A leave-IPD-out influence diagnostic did not respond to the selection, so the problem is invisible without a stated selection model.

What it does not answer. Pairwise continuous-outcome meta-analysis rather than component ML-NMR; no selection-model sensitivity analysis; one network size. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMP-24 — A diagonal-weight influence ranked studies correctly only with two-arm studies and heterogeneity held fixed

Question. Does the diagonal-weight influence that cpaic’s edge_influence() reports rank studies the way refitted leave-one-study-out references do, once multi-arm covariance and re-estimated heterogeneity enter?

Answer. No, the refuting sentence fails; the registered verdict is not fit for ranking. The diagonal influence equaled the correct hat row for two-arm networks (difference 8e-15) and ranked studies well there with tau fixed at 0 (mean Spearman with refitted variance importance 0.95 to 0.97; top study missed in 5% to 6%). With three-arm studies and tau fixed, Spearman stayed at 0.90 to 0.92 but the top study was missed in 28% to 46% of networks; in a post hoc rerun the diagonal’s choice was a three-arm study in almost all misses. With tau re-estimated, mean Spearman was 0.51 to 0.70, including two-arm networks, and the top study was missed in 20% to 59%.

What it does not answer. Standard network meta-analysis rather than component networks with population adjustment; study-level deletion; cpaic’s formula reimplemented, not called; no fixed-tau reference; a block-weighted hat matrix was not tested; 120 of 6000 networks with singular refits dropped. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

CMU-02 — Do prior-sensitivity diagnostics catch the analyses that are actually wrong?

Question. In a population-adjusted network a proper prior yields a proper posterior even where the likelihood identifies nothing, so a narrow interval can reflect the prior rather than the evidence. The diagnostics that would expose this exist. What are their operating characteristics: do they flag the analyses whose intervals actually miss the truth, and how often do they cry wolf?

Answer. At the thresholds the literature suggests, they miss most of it. Pairing each warning with whether that replicate’s own interval missed, a composite of four diagnostics has sensitivity 0.37 at a false-alarm rate of 0.16, falling to 0.13 outside two engineered geometries; the one sensitive rule fires on half the sound analyses too. Used as continuous scores the same statistics reach AUC 0.51 to 0.78, so this is a failure of thresholds rather than of the statistics. What no threshold repairs is that these diagnostics measure the prior’s influence while the harm comes from the prior’s location: with disconnected evidence and a tight prior four standard deviations from the truth, going from 100 to 400 per arm drops the warning rate from 0.98 to 0.01 while coverage stays near zero.

What it does not answer. The other half of CMU-02, wiring these diagnostics into released software, is untouched; neither multinma nor cpaic is run. The model is conjugate Gaussian with an identity link, a correctly specified linear mean and known variances, so nothing transfers directly to a non-conjugate posterior where MCMC error enters and the diagnostics behave differently. Only automated thresholds are evaluated; a plot read by an experienced analyst is a different instrument. Thresholds were taken from the source literature and not tuned. CMP-14 is touched only for the generic aggregate-only interaction, not for component models, and IDN-06 is not addressed at all because this design contains no within-versus-between discrepancy.

Read it: PDF · Markdown · OpenDocument

COV-02 — The adjustment set is a property of the scale

Question. Does the set of covariates whose imbalance changes a transported contrast differ between the risk difference, risk ratio, marginal odds ratio and conditional odds ratio scales?

Answer. The covariates a transported contrast depends on change with the scale, but not in the nested way DESIGN.md predicted. Prognostic non-modifiers never affected the conditional log odds ratio and affected some marginal scale by more than 5% in 13 of 24 scenarios, the risk difference included, because with a logistic model every prognostic variable modifies the risk difference; the risk difference was in fact the most sensitive scale (median 4.5% change per unbalanced prognostic variable, against 1.6% for the marginal odds ratio). Sets differed between the risk difference, marginal and conditional odds ratio in 10 of 24 scenarios, below the registered threshold of half, so the refuting sentence holds as registered.

What it does not answer. No time-to-event scale; no sampling or estimation of the sets; independent normal covariates. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

COV-03 — What an anchored comparison must balance on a non-collapsible scale

Question. On the log odds ratio scale a purely prognostic covariate changes the marginal effect. Does an anchored MAIC that balances only covariate means, as practice does, leave material bias, and does balancing the variance of the prognostic index remove it at acceptable cost in effective sample size?

Answer. Yes. On the log odds ratio scale a difference in the spread of prognosis between trials biases an anchored MAIC that balances only means by up to 0.11, even with no effect modification. Balancing variances or the variance of the prognostic index removes the bias, but when the source is less dispersed than the target the loss of effective sample size makes RMSE worse than not balancing. G-computation was unbiased with the lowest RMSE throughout.

What it does not answer. Independent normal covariates and a correctly specified outcome model favor G-computation. Hazard ratios, skewed covariates and augmented estimators not run; MAIC intervals treat weights as fixed. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

COV-04 — Shrinkage priors on interactions do not give honest intervals for a transported effect

Question. Do posterior intervals from shrinkage or selection priors on treatment-by-covariate interactions keep frequentist coverage for the transported effect when the whole procedure is repeated?

Answer. No. Continuous shrinkage does not select, but its intervals still undercovered: 95% posterior intervals for the transported effect covered 0.794 to 0.948 with an empirical-Bayes ridge prior and 0.868 to 0.959 with a hierarchical normal prior, against 0.939 to 0.955 for a flat prior. When the interactions the target weights push the effect the same way, their joint shrinkage biases the transported effect (up to 0.19) and the narrower interval does not carry the bias. Integrating the prior scale recovers only part of the loss.

What it does not answer. Continuous outcome with conjugate priors; no horseshoe, heredity constraints or projection-predictive selection; aligned modifiers only, no null scenario. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

COV-09 — The same covariate name, two instruments

Question. When the source and target measure a matched effect modifier with different instruments, how is the MAIC estimate biased, does a reliability correction recover it, and can a bounded analysis over declared instrument differences contain the truth?

Answer. MAIC’s bias from a modifier measured by different instruments follows a closed form in the instruments’ offsets, slopes and the source reliability. A reliability correction removes it only when both trials use the same instrument; with differing instruments it left bias up to 0.18 and could undo a chance cancellation. An envelope of corrected intervals over declared instrument differences covered the truth at 1.4 to 2.9 times the naive width.

What it does not answer. Linear instruments and identity link only; misclassification of binary covariates and bridging-sample correction not run; target sampling error absent. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

COV-10 — Interaction coefficients have their own consistency equation

Question. When direct and indirect paths carry opposing treatment-by-covariate interactions, does the pooled network meta-regression mask the modifier while the effect-consistency check passes, and can an interaction-consistency check detect it?

Answer. Opposing interactions on direct and indirect paths attenuate the pooled network meta-regression interaction toward zero, fully at the balance point and partly across a continuum, while the effect-consistency check stays at its null rate. The target effect was biased by 0.13 to 0.30 whether or not the covariate was selected. A direct interaction-consistency check detects it when studies per path and covariate spread are adequate (up to 1.00) and has little power otherwise (0.16 with two studies and narrow spread).

What it does not answer. Stylized triangles with continuous estimates; design-by-treatment interaction model and larger networks not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

COV-12 — Reconstructing a target covariate from a median: when the mean is enough and when it is not

Question. When a target publication reports a skewed covariate as a median with IQR or range, how much error does reconstructing its marginal put into a target-standardized contrast, and is that error a function of the reconstructed mean alone?

Answer. Under an identity link with linear effect modification, yes: the contrast error is exactly the modification coefficient times the mean error. Under a logit link, no: the mean explains only 60% of the error’s variance, because the reconstruction’s spread and shape also enter. With strong skew and a reported range, a normal reconstruction biased the contrast by more than 1 on the identity scale and 0.4 on the log odds ratio scale, growing with sample size; a lognormal matched to the reported quantiles kept bias below 0.1.

What it does not answer. One covariate with a known conditional model; estimated outcome models and multivariate marginals with dependence (CMP-15) are not covered. The lognormal matcher suits right skew only. ML-NMR integration is represented by direct integration. Two cells were run before a regressor correction and discarded unread. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

COV-14 — Covariate sets that differ across comparators, and the ranking read from them

Question. When one individual-data trial is compared with several aggregate comparators that report different covariates, does adjusting each contrast for everything it reports give a worse ranking than adjusting all contrasts for the common set, even though each contrast is less biased?

Answer. Neither adjustment rule protects the ranking. The common (intersection) set ranked better in 15 of 36 scenarios and the maximal sets in 19; which wins depends on whether the omission bias pushes a lower-ranked comparator past a higher one, which the data do not show. The registered inversion was not confirmed in the primary scenarios, whose pattern turned out to widen the top comparator’s lead, but held in all six Fawsitt-like scenarios with similar populations. The intersection’s per-contrast bias was a median 2.4 times the maximal. A bounded analysis over the other comparators’ reported means covered at least 0.93 with similar comparator populations at little extra width.

What it does not answer. Continuous outcome and independent covariates; non-collapsible outcomes, correlated omissions, a single ML-NMR network fit and overlap as a factor were not run. Three reporting patterns sample but do not exhaust where omissions can fall. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DEC-01 — Do estimation and decision rankings of population-adjustment methods agree?

Question. When population-adjustment methods are ranked by the decisions their estimates lead to rather than by RMSE, does the ranking change, and how does it depend on where the truth sits relative to the decision threshold?

Answer. For a threshold on a known side of the truth, yes: near the threshold in poor-overlap scenarios the RMSE and expected-loss rankings correlated at only 0.45, and the disagreement was reproducible, not noise. It came from the direction of bias: a biased method decided wrongly far less often when its bias pointed away from the threshold (0.11 against 0.64). Averaged over the two sides of the threshold, as an analyst who does not know the truth must, the rankings agreed (0.93, the same best method in every poor-overlap scenario). RMSE is therefore an adequate proxy for decision error unless the side of the threshold is known.

What it does not answer. Decision on the relative effect only, without a net-benefit model; one source of replicates (COV-03) and five methods; no ML-NMR; priors on the side of the threshold not modeled. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DEC-02 — Which analysis choice flips a decision

Question. Within one decision question, which analysis choices most often flip a population-adjusted recommendation, and how does that compare with the change in the question when the target population changes?

Answer. The choices are ordered rather than dominated by one. Near a decision threshold, adjusting or not flipped a population-adjusted decision in 36% of analyses, weighting against an outcome model in 20%, linear against quadratic STC in 13% and MAIC on means against means and SDs in 7%. Changing the target population moved the true log odds ratio by 0.39 to 0.59, more than any estimator choice moved the estimate, so the target belongs to the question, not the sensitivity analysis.

What it does not answer. No prior or bridge factor; decision on the relative effect only; one outcome type. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DEC-11 — Coverage of an STC interval after selecting effect modifiers on the same data

Also bears on COV-01.

Question. When effect modifiers are selected on the analyzed data before an STC, does an interval that treats the selected model as fixed keep its coverage, and does resampling the whole procedure restore it?

Answer. Selecting effect modifiers on the analyzed data (interactions with p < 0.2) left the naive STC interval covering 0.887 to 0.927, because its SE ignored which model was selected; a bootstrap that repeats the selection restored 0.938 to 0.968. Lasso at the one-standard-error penalty dropped real modifiers and coverage fell to 0.33 with three modifiers at 1000 per arm. Fitting all candidate interactions kept coverage at 0.93 to 0.96.

What it does not answer. STC only; MAIC-based selection, overlap, larger candidate sets and target-summary uncertainty not run; 400 replicates. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DEC-17 — Which evidence drives a network contrast changes across the plausible range of heterogeneity

Question. Are the contribution shares that attribute a network meta-analysis contrast to its studies and comparisons stable over the plausible range of the between-study variance, or does conditioning on its point estimate change which evidence appears to drive the result?

Answer. No. In a four-study loop network with 5:1 within-study variances, the leading contributing comparison changed across the profile-likelihood interval for tau in 38% to 42% of networks, and the largest single study share moved by about 0.15 on average; with equal within-study variances shares did not move at all. REML put tau at zero in 30% to 67% of four-study networks. The instability comes from the data’s inability to distinguish values of tau, not from bias in its estimate.

What it does not answer. One contribution measure (absolute hat-matrix row), three treatments, two-arm studies, frequentist only. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DEC-28 — The covariate value at which the better treatment changes is often not estimable as a point, and borrowed precision can make its interval wrong

Question. Is the covariate boundary between two treatments’ optimality well enough determined at achievable interaction precision to report as a point, and do standard intervals for it hold when interaction information comes partly from between-trial variation?

Answer. No. From one trial of 400 patients at a moderate interaction the exact (Fieller) 95% confidence set for the boundary was unbounded in 66% to 69% of replicates, while the delta-method interval was always finite and covered as little as 0.86 at the weakest interaction. Borrowing interaction precision from aggregate trials at different covariate means (81% of the precision) made the sets bounded, but a between-trial ecological term of 0.1 cut their coverage to 0.68 to 0.70 for a boundary away from the trials’ mean covariate value. Treating by the point estimate at the moderate interaction lost 31% to 95% of the value of individualized treatment with one trial.

What it does not answer. Two treatments and one modifier; no modifier selection; closed-form frequentist estimators rather than ML-NMR; declared linear ecological term. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-02 — The reported weight panel cannot see where support is missing

Question. Every member of the overlap panel MAIC reports (Kish ESS, ESS percentage, entropy efficiency, maximum weight, top-share) is a symmetric function of the weights, so none can depend on where in covariate space the source fails to cover the target. Does that blindness matter in an ordinary covariate law, or do weight concentration and support geometry move together tightly enough that the panel is an adequate proxy?

Answer. It matters. Two source populations that remove the same mass at the same threshold, one in the effect-modifying coordinate and one in a prognostic coordinate, give panels that agree to within 1.6% while one estimate is biased by 0.016 on the marginal risk difference and the other by less than 0.0001, and material error is 24 times as frequent in the first. Across the two placements every panel member sits within 0.005 of chance; region-specific ESS separates them almost perfectly and sliced optimal transport cost at 0.73, but a convex hull distance maximized over coordinates is as blind as the panel. The registered within-arm primary outcome reads the other way (panel 0.574, best geometric 0.592, rule returns refuted) because within one arm the hole placement is fixed and the outcome measures sampling noise, not geometry. Both readings are reported. Balance on the omitted moment, the comparator most likely to overturn the headline, was never fairly tested.

What it does not answer. Balance on the omitted moment, the free check registered as most likely to overturn the headline, was not tested: every cell where it had anything to see was removed by the matched-multiset control. Region-specific ESS was handed the easiest instance, since the region it examines is the region the hole empties, so its near-perfect separation is an upper bound; a hole in a high-modification region the analyst did not think to examine is untested and there it would fail like the panel. No diagnostic is calibrated to a threshold. The analysis subgrid was chosen by a mechanical rule written after the run was read. The realized high-modification hole holds 22% of the modification mass against a registered 25%. MAIC weights only, a binary outcome, normal covariates, one rectangular hole, good overlap only after the null control, and two of the three published ESS definitions. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-03 — Effective sample size tracks the error an analysis got by chance better than the error it got from adjusting on the wrong covariates

Also bears on DIA-06.

Question. Every population adjustment arrives with a panel of numbers beside it: effective sample size, the same figure as a percentage, the largest weight, standardized differences before and after adjustment, a distance between populations. A committee reads them as evidence about whether to believe the estimate, and a widely quoted rule says to worry below an effective sample size of about 30 to 35. Scored as classifiers of realized error, with discrimination and calibration, against known truth and under a declared distribution of analyses, what are they worth?

Answer. At the level at which a diagnostic is actually read, one analysis at a time, effective sample size tracks the error a study got by chance far better than the error it got from adjusting on the wrong covariates: AUROC 0.847 against the outcome-noise component and 0.813 against chance covariate imbalance between the source arms, both of which are functions of the covariates, the assignment and the weights, against 0.653 for the transport component that adjustment exists to remove. At the quoted cutoff of 35 it catches between a quarter and a third of material errors in every misspecification stratum, 0.309 where an effect modifier is omitted and the omission is diagnosable in principle, at specificity 0.926. No evaluated fixed cutoff met the registered requirement of sensitivity 0.80 at specificity 0.50 in every stratum. Three of the panel’s members are provably one statistic, since ESS/n = 1/(1+CV^2(w)) exactly and area under the ROC curve is invariant to monotone transformation; a fourth, the post-weighting balance on matched moments, is zero at the solution of the calibration equations and among converged analyses cannot discriminate anything. On a decision-curve basis every rule in the panel is beaten by scrutinizing every analysis, up to an action threshold of 0.3.

What it does not answer. The second half of DIA-03 is untouched: no PSIS-LOO is computed at any hierarchical level and grouped study-level leave-one-out is no more implemented here than in the packages, so the mismatch between the level at which cross-validation is reported and the level at which the prediction is made remains open. DIA-06 is answered only in part; two estimator families and no adjustment are compared on identical replicates, but ML-NMR, NMI, doubly robust estimators and flexible learners are absent. The outcome is continuous with an identity link, which is what buys the exact error decomposition and is also the largest restriction: nothing here transfers to non-collapsible effect measures or time-to-event outcomes, where part of the error is a collapsibility artifact rather than an adjustment failure. Covariates are normal, comparisons are anchored, conditional constancy holds, and target moments are exact. The deployment weights are a declared judgment and nothing in this program measured them, which is why the primary results are within strata.

Read it: PDF · Markdown · OpenDocument

DIA-07 — Where the individual data sit decides the error of a population-adjusted network contrast at the same total sample size

Question. At a fixed total sample size, does the error of a population-adjusted network contrast depend on the network’s topology and on which trial supplies individual data, beyond total information?

Answer. No. At 1200 patients in every network, the probability of getting the sign of the decision contrast wrong ranged from 0.36 to 0.93 across four topologies and positions of the one individual-data trial (moderate modification, target shifted one SD), and the spread exceeded 0.10 in every scenario. An individual-data trial off every path to the contrast carried zero weight and changed nothing; standardizing one edge could raise the bias (0.20 to 0.30 in a line network) by removing an error that had offset the others.

What it does not answer. Deterministic calculation; continuous outcome, linear modification, four hand-built topologies, one IPD trial, fixed-effect model; no generated topology families, component networks or ML-NMR. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-08 — Does the generator’s effect-modification structure decide the method ranking?

Question. When effect modification departs from the shared linear form most simulations use, does the ranking of population-adjustment methods change?

Answer. Not at these magnitudes. The RMSE ordering under shared linear modification was reproduced under threshold, interaction and quadratic modification at every overlap level (rank correlation 0.9 to 1.0, the only change a swap between the two worst methods). STC with linear interactions ranked first everywhere even though the departures biased it by up to 0.12, because weighting methods lost more to variance at poor overlap. The generator decides a ranking only when a departure’s bias exceeds the precision differences between methods.

What it does not answer. Moderate departure magnitudes only; treatment-specific, subnetwork and within-versus-between departures need a network and were not run; no ML-NMR. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-09 — Competing risks: a correct hazard ratio, a wrong cumulative incidence

Question. When population adjustment transports a time-to-event outcome with a competing cause, does a correctly transported cause-specific hazard ratio give the right cumulative incidence in the target, and what repairs it?

Answer. For competing risks, population adjustment transports the cause-specific hazard ratio (unbiased in every scenario) but not the cumulative incidence a decision model uses: when the target’s competing hazard differed from the source’s beyond the measured covariate, A’s absolute incidence at 2 years was biased by up to 0.20 (true value 0.25) with coverage near zero. The between-arm difference was affected less. Recalibrating the competing hazard to the target’s reported competing-event proportion removed the bias. The registered rule (bias beyond 0.05 wherever the competing hazard differed) was met in 20 of 24 method-scenarios; the exceptions were at half the competing hazard, with bias 0.041 to 0.049.

What it does not answer. Only the competing-risks family; cure fractions, recurrent events, ordinal and joint net-benefit outcomes not run. Exponential hazards, one covariate, no ML-NMR. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-10 — Reconstruction error from a digitized Kaplan-Meier curve was too small to change which method ranks best for RMST

Question. When the comparator arm must be reconstructed from a published Kaplan-Meier figure, does the reconstruction error change which method performs best, as it would if it penalized likelihood-based methods more, or does it only shift every method’s error?

Answer. Yes, for RMST within follow-up the omission is harmless. Reconstructing the comparator arm from a pixel-rounded Kaplan-Meier figure changed neither the ranking of Kaplan-Meier and Weibull RMST (same in all 8 cells) nor any method’s RMSE by more than 1%: it added 0.003 to 0.040 months of bias to Kaplan-Meier RMST and 0.006 to 0.059 to Weibull RMST against a sampling SD of about 0.5 months. The likelihood-based method was more exposed, as predicted, but immaterially.

What it does not answer. Reconstruction error only; RMST within follow-up only (extrapolated survival and hazard ratios not ranked); no human digitization variability; other access mechanisms not simulated. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-13 — Do bias mechanisms in an unanchored MAIC combine additively?

Also bears on QBA-22.

Question. When an omitted confounder, comparator outcome misclassification and covariate measurement error act together on an unanchored MAIC, is their joint bias the sum of their separate biases, and does a one-at-a-time analysis bound it?

Answer. No. An omitted confounder and misclassification of the comparator’s outcome interacted by up to 0.16 on the log odds ratio scale: with both at their largest the joint bias was about half the sum of the separate biases, and with the confounder reversed the joint bias had the opposite sign to that sum. Measurement error in a matched covariate combined nearly additively with either. The joint bias exceeded the largest single-mechanism bias in 22 of 36 combinations, so one-at-a-time analyses neither add up to nor bound it.

What it does not answer. Population-level only; no bias-analysis procedure, false reassurance or tipping sets; three mechanisms; selection and correlated parameters not varied. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIA-14 — A quantitative bias analysis verdict scored as a classifier

Question. How often does a QBA declare a MAIC decision robust when the true bias would reverse it, how does that depend on the elicited region rather than the method, and does oracle recovery reveal it?

Answer. A QBA’s robust verdict is only as good as the elicited region. When the region contains the true bias the grid never reassures falsely (an exact property); with elicitation errors scaled to the region it reassured falsely at most 17% of the time the true bias reversed the decision, but with the common region centered at zero bias it did so 78%, 37% and 9% of the time at half-widths 0.05, 0.15 and 0.3. Probabilistic QBA avoided false reassurance by calling 62% to 85% of sound decisions fragile. Oracle recovery was nominal throughout and revealed none of this.

What it does not answer. One scalar bias parameter; effect modification and overlap not varied; the elicitation model is assumed. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIS-03 — A baseline-risk bridge across a disconnected network

Question. When disconnected subnetworks are linked through exchangeable baseline risks, does study-level design nuisance pass into the cross-gap contrast, and can design covariates separate it from prognosis?

Answer. A baseline-risk bridge transports whatever makes the subnetworks’ baselines differ. When a design nuisance (such as an endpoint definition) coincided with the subnetwork it passed one for one into the cross-gap contrast (bias 0.2 to 0.7) and no adjustment was possible. When a quarter of trials used the other definition, adjusting for it removed the nuisance but not prognostic drift between subnetworks (bias 0.19 to 0.20).

What it does not answer. Study-level summaries, no Bayesian arm-based model or commensurate prior, one design covariate. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIS-11 — A predictive winner among models that share a false bridge

Question. When candidate models share a bridge assumption across a disconnected network, does a predictive criterion computed on observed trials rank them by their cross-gap error?

Answer. No. Candidate models that share the bridge had identical cross-gap bias (0 to 0.5 across scenarios), and the leave-one-trial-out winner was set by within-subnetwork structure: meta-regression won when a trial covariate changed the contrast, the common-effect model when not, with the same distribution at every bridge error. A predictive criterion on observed trials cannot rank models by an error they share.

What it does not answer. Frequentist predictive densities rather than LOO-PSIS on ML-NMR; no grouped leave-subnetwork-out or synthetic deletion; population comparison not simulated. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

DIS-21 — A matched bridge’s admission threshold sets its bias: at the threshold used in practice the bridged contrast was biased by twice the true effect

Question. How do the bias of a matched bridge across a disconnected network and the existence of the bridged comparison depend on the admission threshold, expressed in multiples of the distance between arms of one randomized trial?

Answer. No. With trial populations shifted across the gap, the bridged contrast (truth -0.2) was biased by -0.03 at the within-trial arm distance, where the bridge existed in only 8% of replicates, -0.13 at three times it and -0.44 at ten times it, near the threshold used in practice, where it always existed and its interval never covered. An index omitting the most prognostic covariate was biased by -0.25 even at the tightest threshold. Pooling pairs that reuse arms made intervals undercover even without bias (0.76 at ten units with good overlap).

What it does not answer. Monte Carlo networks only, no real-network deletions; one distance with two covariate sets; no component network meta-analysis bridge; admitted pairs pooled as independent. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

EST-06 — When two sponsors’ MAICs disagree: how often, why, and which common target resolves it

Question. When two sponsors run MAIC on the same two trials, each weighting its own data to the other trial, how often do their conclusions reverse, how much of that is a real difference in target population rather than sampling error or differing adjustment sets, and which common-target remedy recovers the decision population’s effect?

Answer. Reversal is mostly a real target difference and it is common: with the modifier’s zero-effect point between the two populations, sponsors disagreed in sign in 23% to 56% of analyses at separations up to 1 SD and modification up to 0.4, and moving the point outside the populations cut that by a median 0.31. The two sponsors’ estimates are correlated, which a reversal calculation must use. Under linear modification the plain average of the two analyses is as accurate for the midpoint population as transporting both trials there.

What it does not answer. Linear modification and a continuous outcome; with nonlinear modification or a non-collapsible scale the average is not the midpoint effect. The arbitrated indirect comparison and measurement differences between trials were not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

EST-11 — Scored against its own target, the unadjusted comparison ranked best; against a declared target it ranked worst

Question. Does scoring each method against its own implicit target give the same method ranking as scoring against a declared decision target, across a menu of targets and with correct or misspecified second-stage transport?

Answer. No. The method with the lowest RMSE against its own implicit target differed from the one with the lowest RMSE against the declared decision target in 5 of 6 cells. The unadjusted Bucher comparison, scored against the mixture of populations it estimates, won the native metric everywhere because it has the least variance, and was biased by 0.13 to 0.25 against the declared target in five cells. When the comparator’s modification differed from the individual-data treatment’s, methods native to the comparator trial’s population missed a declared target one covariate SD away by 0.11, and a two-stage transport assuming shared modification did not remove it.

What it does not answer. Three declared targets, one covariate, one sample size, exact target moments, four methods; no ML-NMR or ML-UMR. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

EST-12 — Treatment hierarchies across target populations

Question. Does a treatment hierarchy move more across plausible target populations than across sampling variability, so that a hierarchy must name its population?

Answer. Yes: where effect modification changed the true ranking across plausible targets, the estimated hierarchy moved 1.6 to 2.4 places per treatment across targets against 0.1 to 1.4 places across sampling replicates, so a hierarchy must name its population. But estimated hierarchies also moved 0.2 to 2.1 places across targets where the true ranking did not change, through the uncertainty of meta-regression slopes away from the data’s center, so a changed hierarchy is not by itself evidence of population dependence.

What it does not answer. Study-level meta-regression rather than ML-NMR; P-scores only; one covariate; fixed-effect. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

HET-02 — Matched single-arm studies built against one comparator arm

Question. When several single-arm studies are each matched to the same comparator arm and their pseudo-contrasts enter a network as independent, how much does that overstate precision, and does it change the conclusion?

Answer. Entering pseudo-contrasts built against one comparator arm as independent counts that arm once per study. Its reported SE shrank as studies were added while the estimate’s spread did not, so coverage fell from 0.88 with one study to 0.57 with eight (comparator arm of 50). Modeling the shared covariance kept coverage at 0.95. In this fixed-effect network the point estimate barely changed; the error is in the stated precision.

What it does not answer. Fixed-effect continuous network without a weighting step; heterogeneity and conflicting evidence, where the point estimate can also move, not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

HET-03 — Shared and class-specific heterogeneity cannot be told apart below about 40 studies per class

Question. How many studies per class are needed before model comparison reliably distinguishes a shared between-study variance from class-specific ones?

Answer. No. Even with a fivefold difference in between-study variance, AIC chose class-specific variances in only 18% of networks with five studies per class, 40% with twelve and 79% with forty. Balanced selection accuracy reached 0.8 only at 40 studies per class and only for AIC; BIC and a likelihood-ratio test did not within 40. At the network sizes population-adjusted analyses use, the heterogeneity structure cannot be chosen from the data, so the shared default is untested rather than supported.

What it does not answer. Two classes, contrast-level data, frequentist criteria; no Bayesian model comparison, informative priors, ML-NMR or class-specific treatment SD. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

HET-04 — A population difference arriving as inconsistency

Question. When the studies supplying the direct and indirect paths of a loop differ in an effect modifier, how often does unadjusted node splitting flag the population difference as inconsistency or miss a real conflict that the difference cancels, and what does an adjusted split achieve?

Answer. Unadjusted node splitting tests inconsistency plus population mismatch. With no true inconsistency it flagged 14% to 98% of analyses when the direct and indirect study sets differed in an effect modifier, and it fell to its null rate when the mismatch cancelled a real conflict. A split adjusted for the modifier held size (4% to 6%) and recovered some masked conflicts, but its precision fell as the gap between the study sets grew.

What it does not answer. Continuous study estimates, one loop, no Bayesian node split or marginalization to a named target. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

HET-10 — Interval coverage for population-adjusted synthesis with few studies

Question. How does interval coverage of population-adjusted random-effects synthesis depend on the number of studies, when one individual-data trial supplies the adjusted contrast for several aggregate trials?

Answer. Coverage fails in two ways. Meta-analyzing per-trial adjusted contrasts that reuse one individual-data trial as if independent lost coverage as trials were added (down to 0.55 at twelve trials with DerSimonian-Laird), because the shared error does not average out. Pooling the aggregate trials first and subtracting the transported contrast once fixed that; with DerSimonian-Laird it stayed within 0.93 to 0.97 from four trials, and Hartung-Knapp was conservative (up to 1.00) at two or three.

What it does not answer. Pairwise synthesis with one individual-data trial; continuous outcome; no one-stage or Bayesian analysis; one overlap level. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

IDN-01 — What each transitivity screen can see

Question. Do the available transitivity screens (covariate dissimilarity, loop inconsistency, heterogeneity), run together, detect the violations that bias a network estimate, and can their pattern of flags identify the mechanism?

Answer. No. Each screen sees one direction of violation, and an unmeasured modifier that makes one treatment look better in every comparison that includes it is invisible to all three: at size 0.4 it biased the network estimate by 0.40 with coverage near zero while the screens together fired at their null rate (0.28). Loop breaks and drift were detected with power rising in the number of studies, but the pattern of flags identified the mechanism poorly, and a measured modifier was always flagged although adjustment removed its bias.

What it does not answer. Stylized triangle networks with continuous estimates; calendar drift as random shifts; abandonment records not scored. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

IDN-05 — A violation twice the size that would change a decision raises the shared effect modifier check’s alarm rate from 1.3% to 4.5%

Question. Population adjustment across a network assumes that a covariate modifies the effect of every active treatment in a class in the same way. There is one published within-network check on that assumption: fit the model with a single interaction shared across the class, refit with treatment-specific interactions one covariate at a time, and compare the posteriors and the model fit. Its authors say it likely has low power. Nobody has measured it, there is no threshold that separates a pass from a failure, and there is no statement of what a pass licenses. Scored against a known truth under a declared distribution of networks, what is it worth?

Answer. The check does not have the resolution to test the assumption it is read as testing. Against a violation already twice the size that would change a decision, the DIC rule fires in 0.045 [0.029, 0.070] of analyses, against 0.013 [0.005, 0.029] when the assumption holds exactly: a violation that doubles the error that matters moves the alarm rate by about three percentage points. The result does not depend on where the cut is put. Scored threshold-free, the DIC difference ranks a violated network above an intact one with an AUROC of 0.557 [0.517, 0.596] at that violation, so no threshold recovers a ranking the statistic does not contain. The posterior standard deviation of the interaction contrast runs 0.30 to 1.64 against the 0.153 that would make a transported estimate materially wrong, which is why. The equivalence reading, the only one that could confirm rather than merely fail to contradict, fired zero times in 3,200 opportunities, including every replicate where the assumption held exactly; that is a statement about precision, and partly about the prior, since independent normal(0, 2.5) priors already place 0.965 of the contrast beyond the margin. Following the check is still better than not: relaxing exactly what it fires on beats relaxing everything at every target displacement and beats imposing the restriction at every displacement except zero, where the two are indistinguishable.

What it does not answer. The design is a best case in several specific ways that favour the check, and failures attributable to those idealizations are conservative; no general lower-bound claim is made, because three features cut the other way. The true marginal reference risk in the target is supplied to the estimator. Covariates are normal with known target moments and study covariate means are drawn independently of study baselines and of the random treatment effects, so the ecological route identifying the aggregate-only treatment’s interaction is unbiased by construction. Conditional constancy holds and only one covariate drifts, linearly. Against that: the fits are fixed-effect while half the cells generate random treatment effects, which is a second specification error that inflates firing; the likelihood is integrated over 64 quadrature points and refitting at 256 changes the DIC verdict in 8.3% of a 24-network subset, which is noise belonging to the implementation and not to the evidence, and 256 points is not demonstrated to be converged; and a single individual-level study is an assumption about the network rather than necessarily a pessimistic one. Beyond that: what a second individual-level study would buy is not measured, that factor having been dropped to pay for treatment-effect heterogeneity. With two active treatments and two covariates, multiplicity across covariates and the selection involved in reading the widest of several pairwise differences are not measured. Where treatment-effect heterogeneity is present the estimand is not cleanly defined, since truth is set at a random effect of zero while a fixed-effect model targets a function of the realized studies; primary results are repeated on the sixteen homogeneous cells and do not change. The hierarchical middle ground between shared and independent interactions could not be fitted because the package does not implement it, and we did not implement it ourselves: that is a reason for its absence, not a justification. The five decision rules scored here are our formalizations of an exploratory published practice, not published rules, and their operating characteristics are ours. Fifty replicates per cell gives a per-cell standard error of 0.065, so nothing is read off a single cell.

Read it: PDF · Markdown · OpenDocument

IDN-06 — An aggregate-only treatment’s interaction is identified by assumption, by an ecological gradient or by the prior, and each fails differently

Question. When a treatment has only aggregate data, is its effect-modifier interaction identified by the data, by the shared-interaction assumption, or by the prior, and do credible intervals cover when the assumption fails?

Answer. By none of them reliably. With one aggregate trial the data carry no information about the treatment’s interaction, yet the shared-interaction model’s credible interval was as narrow as the individual-data trial’s and covered 0.68 and 0.15 at discordance 0.15 and 0.3; a separate model returned its prior. With four aggregate trials the separate model covered nominally until an ecological term of 0.1 entered the between-trial gradient, then 0.71. The shared model’s apparent recovery there came from the ecological bias offsetting the discordance.

What it does not answer. Conjugate normal models on contrast-level data rather than multinma fits; one covariate; fixed prior scales. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

IDN-07 — Calibrating a held-out-trial falsification rule

Question. When one trial is withheld and its contrast predicted from the rest, which pass or fail rule holds its type I error, how much power does it have, and what does omitting a variance component or mis-targeting the prediction do?

Answer. A held-out-trial check has a controllable size if the prediction targets the withheld trial’s own population and its variance includes between-trial heterogeneity: the full rule’s size was 0.035 to 0.073 (registered limit 0.07, exceeded by 0.003 in one scenario). Omitting heterogeneity raised size to 0.104, and aiming the prediction at the remaining trials’ population raised it to 0.96. Power against a violation as large as the effect was only 0.21 to 0.47, so a pass licenses very little.

What it does not answer. Continuous outcome and G-computation only; no MAIC weight variance, no Bayesian predictive or conditional-moment rule; five aggregate trials. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

IDN-09 — A negative-control outcome for transport bias

Question. Can a negative-control outcome detect and calibrate transport bias in an unanchored MAIC, and when does the calibration fail?

Answer. Only in a knife-edge case. A negative-control outcome with no treatment effect detects a population difference (flag rate up to 0.83) but calibrates the primary contrast correctly only when its dependence on the unmeasured covariate equals the primary outcome’s and that covariate does not modify the treatment effect (bias 0.001). Otherwise the calibrated contrast was biased by -0.18 to 0.40: it missed the modification part, overcorrected with a more strongly dependent control, and cancelled by coincidence in one scenario.

What it does not answer. One control outcome rather than a bank with empirical calibration; unanchored only; availability of control outcomes not surveyed. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

IDN-10 — A transported control-arm check carries no information about the anchored contrast

Question. Does agreement between a MAIC-transported control arm and the target’s observed control arm tell an analyst whether the anchored relative effect is biased?

Answer. No. The check tests prognostic transport, which an anchored contrast does not need, and cannot see effect-modification transport, which it does. A prognostic shift raised the alarm rate to 0.48 while the contrast was nearly unbiased (bias -0.03 to -0.06); a modifier shift biased the log odds ratio by 0.30 with the alarm rate at 0.044. Pooled over 18,000 analyses the check’s AUROC for a biased contrast was 0.50.

What it does not answer. Binary outcome only; survival-curve and RMST versions, unanchored comparisons, offsetting prognostic mechanisms and outcome-definition or follow-up differences are not simulated. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

IDN-19 — A partition-sensitivity display for shared effect modifiers is stable exactly when every candidate partition is wrong

Question. Does the choice of which treatments share an effect-modifier interaction move a target contrast, does the widest estimable partition fail silently, and does the range over defensible partitions contain the truth?

Answer. No. When an aggregate-only treatment’s interaction differed from both candidate classes’ (by 0.3) while theirs agreed, every admissible partition returned the same answer biased by -0.30; the range over partitions was narrow (0.14), contained the truth in 5% of replicates, and the only available fit test rejected at its nominal 5%. The widest partition covered 0.37. The range widened and contained the correct partition only when the candidate classes disagreed with each other.

What it does not answer. Small hand-built network, continuous outcome, linear modification, three partitions, sufficient statistics drawn directly; no overlap factor or ML-NMR fits. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MIS-01 — Missing covariate values in the individual-data trial of a MAIC

Question. When a matched covariate is missing for some patients in the individual-data trial, when is complete-case MAIC biased, and do inverse-probability-of-observation weighting and multiple imputation repair it?

Answer. Complete-case MAIC bias is set by the missingness mechanism, not the rate. It is unbiased when missingness depends only on matched covariates, including the missing covariate itself, and biased already at 15% missingness when it depends on an unmatched effect modifier, the outcome with modification, or an unmatched covariate in one arm only. Weighting and imputation repair the missing-at-random cases (imputation with arm interactions most reliably) but are biased by up to 0.22 when missingness depends on the missing covariate itself, where complete case is not.

What it does not answer. Missing outcomes, binary outcomes, STC, overlap levels and pattern-mixture sensitivity analysis not run; single-driver mechanisms only. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MIS-02 — TADA’s bootstrap undercovered under heavy censoring and poor overlap, and a censoring model fitted through administrative follow-up left part of the censoring bias in place

Question. Does the TADA estimator, participation weights times inverse-probability-of-censoring weights with bootstrap variance, stay unbiased and well calibrated when censoring is heavy, overlap poor and the two weight sets correlated?

Answer. No, the refuting sentence fails. In the registered primary cell (60% censored before 24 months, poor overlap, positively correlated participation and censoring weights) TADA with bootstrap SE had SE ratio 0.80 and coverage 0.886, with bias 0.233 (MCSE 0.145): the registered verdict is fails by variance. The product of the two weight sets kept an effective sample size of 25 against 112 for the participation weights alone, and TADA’s RMSE (3.25) exceeded that of the participation-weighted Kaplan-Meier with no censoring weights (2.23). The fixed-weight sandwich did no worse (SE ratio 0.80, coverage 0.892).

What it does not answer. RMST only, with complete-case IPCW rather than time-varying weights in a Cox model; one covariate drives both weight sets, so their correlation and the censoring weights’ spread are confounded; no stacked M-estimation variance; no censoring by unmeasured prognosis; target moments exact. The horizon-fitted censoring model is a post hoc correction reported beside the registered result. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MIS-03 — What conditioning on sampled target moments costs

Also bears on EST-07.

Question. Population adjustment treats a target trial’s published covariate means and standard deviations as exact, though they are sample estimates. By how much does that cost interval coverage at realistic sample sizes, and what would a publication have to report for a correction to work?

Answer. Two things, established by different means. Analytically: the variance a standard interval omits is (1-2k)Var_T(tau)/n_T to first order, where k is how far the target trial’s own treatment effect is modified by the same covariates as the source trial’s, so the sign is set by a quantity the usual inputs do not identify; and a correction adding only the positive term is closer to the right variance than doing nothing only when k < 1/4. Empirically, and only in an exploratory subset: the omission runs from 9.5% too much variance to 14.9% too little and moves coverage between 92.1% and 96.2%. The registered analysis is uninformative and the confirmatory coverage question remains open.

What it does not answer. The registered coverage question is not confirmatorily answered: every usable coverage result comes from a post-run restriction conditioning on an observed outcome. One estimator, one outcome type, one information structure. Covariates are multivariate normal, which is also the assumption the deployable correction is derived under, so it is measured under conditions favorable to it; skewed, bounded and categorical covariates are untested. The outcome is continuous and the effect additive, so marginal and conditional effects coincide, and nothing transfers directly to binary or time-to-event outcomes, where MAIC is most used. Effect modification lies exactly in the span of the matched moments, so this is a variance result under correct identification and says nothing about bias. The target moments and the target treatment effect come from the same trial, which is the common case but not the only one. STC, ML-NMR, ML-UMR and NMI have different sensitivities and are not covered. EST-07’s other components, model uncertainty in reconstructed correlations, inclusion-criteria ambiguity and secular drift, are not sampling error and are untouched. The alignment parameter imposes exact non-negative proportionality between source and target modifier coefficients, so modifier sets differing in direction or overlapping only partially are untested.

Read it: PDF · Markdown · OpenDocument

MIS-04 — Choosing among population-adjustment models on the same data

Also bears on MOD-04.

Question. When the adjustment model is chosen from the analyzed data, does the chosen model’s interval keep its coverage, and what does averaging over candidates change?

Answer. For choices among MAIC and STC specifications the candidates are close: choosing the most precise method covered 0.938 to 0.956, like a prespecified one. Choosing the most favorable estimate shifted it by up to 0.26 on the log odds ratio scale while its interval still covered 0.927 or more, because the favored method often had a wide interval. Averaging with between-model variance over-covered (up to 0.991) because the candidates share data; that difference makes the refuting sentence fail as registered.

What it does not answer. Outcome and weighting models only; treatment partitions and bridges not studied; equal weights. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MOD-01 — Standardizing over an independence reconstruction of the target biased the marginal log odds ratio by up to 0.21; a correlation matrix removed the bias, and the error was linear in one covariance term

Question. Standardization needs the target’s joint covariate law while publications report marginals; when the dependence the marginals leave free is reconstructed wrongly, how much does the standardized marginal contrast move, and when is the reconstruction immaterial?

Answer. Yes, the refuting sentence fails: the reconstruction matters, and it is fixable. In the registered primary cells (all-positive prognostic signs, quadratic effect modification, correlation 0.6) G-computation over an independence reconstruction of the target biased the marginal log odds ratio by 0.126 on average and 0.210 at worst, against a material 0.05, with coverage 0.898; with the correlation matrix borrowed from the trial the estimate was within 0.002 of standardization over the true joint law. The independence error was linear in the covariance term it drops (R-squared 0.92 to 0.99 by block), and the copula family beyond the correlation matrix moved the estimate by at most 0.004.

What it does not answer. Reported marginals exact; outcome model correctly specified; continuous covariates; the borrowed correlation is optimistic because the trial shares the target’s dependence. Registration was by DESIGN.md sections 4 to 8 and R/07-decision.R, committed before the grid; there is no separate protocol document. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MOD-02 — Flexible outcome models reduced but did not remove transport bias where the target left the data, and could not see a modification they were not built to see

Also bears on MOD-10.

Question. Do flexible outcome models remove the transport bias a parametric STC incurs under threshold, hinge and covariate-interaction surfaces, and with valid intervals, as overlap worsens?

Answer. Largely yes, the refuting sentence holds. Under a hinge in the treatment effect at a target 1.5 SD from the source, parametric STC was biased by -0.144 and a GAM with smooth treatment modification by -0.085 (coverage 0.93): about 40% less bias, short of the registered halving, with the same RMSE (0.314 against 0.318) because its variance was larger. Under a covariate-by-covariate modification every method was biased by about -0.18, since additive smooths cannot represent a product. A threshold prognostic term largely cancelled in all methods.

What it does not answer. No BART, random forest, doubly robust or ML-NMR arm; logit link, one sample size; model-based intervals only. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MOD-15 — The half-standard-error pooling rule for interactions is stated in the wrong standard error

Question. The published rule pools within-trial and across-trial interaction estimates when they differ by less than half the across-trial standard error. What are its pass rate, the bias and coverage of what it reports, and how does it compare with a test stated in the difference’s own standard error?

Answer. The difference’s standard error is sqrt(sW^2 + sA^2), not sA, so under equality the rule pools with probability at most 0.383 (0.31 to 0.37 in the networks studied): it refuses valid pooling most of the time in every regime. When it pools under ecological bias, the pooled interval can cover as little as 22%. Its RMSE matches never pooling to within 0.3%, forgoing the 2% to 19% gain pooling offers when the estimates agree.

What it does not answer. Independent normal estimates with known standard errors; no between-trial heterogeneity; all trials with individual data. The ML-NMR case, where aggregate trials inform only the across-trial component, is not computed. No agreement rule can detect trial-level confounding that is small relative to the across-trial standard error. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MOD-16 — Arm-separate matching cancels the aggregate trial’s chance imbalance; pooled matching carries it

Question. Arm-separate matching weights each individual-data arm to the corresponding aggregate arm; pooled matching uses one weight function. When does each give a biased anchored comparison, and is the bias of arm-separate matching the product of unmatched prognostic strength and aggregate arm imbalance, as the design claims?

Answer. The product claim is wrong. Given an aggregate-trial imbalance kappa, pooled matching is biased by kappa (beta/2 + gamma_X + gamma_U rho_T) and arm-separate matching by kappa gamma_U (rho_T - rho_S); simulation matched both (slopes 1.00 and 0.96). Arm-separate matching reproduces the aggregate imbalance in the individual-data contrast so it cancels, and is biased only when an unmatched prognostic covariate relates to the matched one differently in the two trials. It had smaller bias than pooled matching in every scenario with an imposed imbalance and 10% to 27% lower RMSE under chance imbalance.

What it does not answer. Continuous outcome, linear model, one matched and one unmatched covariate, shared effect modification; the non-collapsible case, where arm-separate matching compares arms in different populations, is not studied, and the arm-separate interval’s conservatism is not repaired. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

MOD-18 — An estimated risk score used as a regressor

Question. When a risk score estimated in a prognostic cohort is used as a regressor for treatment-effect heterogeneity, does treating it as data understate the interval, and is the understatement worst at the tails of risk?

Answer. Little, given a prognostic cohort of about a thousand patients or more: propagating the score’s estimation uncertainty widened intervals by at most 2.4%, and plug-in coverage stayed near nominal. With 300 cohort patients the widening reached 7% at the risk tails against about 1% at the median, confirming that the understatement is worst where recommendations are made, and plug-in coverage fell to 0.924. The registered rule (plug-in coverage at least 0.93 everywhere) was missed in that one scenario, so the refuting sentence fails as registered.

What it does not answer. Pairwise six-trial analysis; frequentist bootstrap rather than a joint or cut Bayesian model; well-specified score. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-02 — Effective sample size counted in events: right direction, no gain over ESS times the event rate

Question. With rare binary outcomes, does an effective sample size counted in events (EESS) predict when a MAIC interval fails better than the reported ESS multiplied by the event rate, and in which direction do the two diverge?

Answer. EESS is the reciprocal of the sandwich variance of a weighted log odds, and it differs from ESS times the event rate by E_w[w]E_w[p]/E_w[wp]: below 1 when weights concentrate on high-risk patients, above 1 on low-risk ones (the design had the direction reversed). As a predictor of failure the two were equivalent (AUROC 0.839 and 0.858) and ESS alone was poor (0.662). Their predictive value comes from zero-event analyses; once an estimate exists, non-coverage is near nominal and neither predicts it.

What it does not answer. One prognostic covariate, MAIC on means; the penalized, Firth, Bayesian and fail-closed estimators were not run, and unanchored comparisons and multivariable separation are outside the design. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-05 — The transported mean is safe and the responder probability is not

Question. When a bounded continuous outcome is transported by G-computation with a pooled normal residual, is the mean contrast safe while the responder-probability contrast is biased, and does a floor make the bias depend on the target population?

Answer. Yes. With a pooled normal residual the transported mean contrast was unbiased while the responder-probability contrast was biased by up to 0.10 when the arms’ dispersions differed, with opposite signs at a threshold near the median and one in the tail. Arm-specific SDs fixed it only for normal errors without a floor; Tobit fixed the floor but not skew; logistic regression of the responder indicator was unbiased everywhere. A floor biased the mean contrast by under 2% of the effect, somewhat more at the larger target shift.

What it does not answer. One source trial, no ceiling, no intervals, no ML-NMR. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-06 — Change scores and endpoints do not diverge under population adjustment

Question. When baseline severity modifies the treatment effect, do change-score and endpoint analyses give different target-population contrasts after population adjustment, in proportion to the product of the interaction and the baseline shift?

Answer. No. Baseline is pre-treatment, so the target follow-up and change contrasts are one quantity under any adjustment. Across 24 scenarios, MAIC and G-computation never diverged beyond Monte Carlo error, and were identical to machine precision in unanchored comparisons. The only divergence was in naive unanchored comparisons, equal to 1.00 times the baseline shift and independent of the baseline-by-treatment interaction.

What it does not answer. Continuous outcome with a linear conditional mean only; nonlinear links, threshold-based responder estimands and joint repeated-measures models are not tested. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-07 — Informative censoring on the two sides of an indirect comparison

Question. When censoring depends on latent risk in the source and comparator trials to different degrees, does the bias cancel in the indirect RMST contrast, what does source-side IPCW achieve, and does a one-sided comparator sensitivity analysis cover the truth?

Answer. Informative censoring cancels in an indirect RMST contrast only when the two mechanisms match. Unanchored, with differing mechanisms the bias was -2.3 to +1.4 months against a true difference of 0.62, reversing the comparison in up to 97% of analyses; equal mechanisms nearly cancelled. Anchored comparisons were protected (bias within 0.22) because both arms of each trial shared its mechanism. Adjusting only the source by inverse probability of censoring weighting helped partly where only the source was informative and broke the cancellation where both were.

What it does not answer. No covariate shift between trials; time-constant censoring hazards; one sensitivity parameterization; one baseline proxy. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-08 — Overdispersed and zero-inflated counts transported by G-computation

Question. For count outcomes transported to a target, does overdispersion bias the estimate or only its interval, and when the fraction of structural zeros differs between populations, what does a count model transport wrongly and what repairs it?

Answer. Overdispersion leaves the transported rate ratio unbiased and makes the Poisson interval too narrow (coverage down to 0.61); a sandwich or negative binomial restores it. A structural-zero fraction that differs between source and target biases the transported absolute rate in every count model, including zero-inflated Poisson, by up to 0.56 on the log scale; the rate ratio is protected only without effect modification. Recalibrating the zero part to the target’s reported zero proportion repairs the absolute rate when the source has structural zeros to learn from.

What it does not answer. One covariate; frailty, exposure weighting, recurrent events and zero-inflated negative binomial not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-10 — A proportional-odds violation biases an expected-utility difference in both directions, depending on where the utility increments sit

Question. Does a proportional-odds violation bias a target-standardized expected-utility difference in proportion to the utility increments at the violated cut-point, and does relaxing proportional odds remove it?

Answer. Not as predicted. A proportional-odds violation biased the target-standardized expected-utility difference whichever end carried the utility increments: by -0.025 and -0.047 with the increments at the violated cut-point and by +0.026 and +0.024 at the other end, with coverage 0.67 to 0.85. The single coefficient understates the effect at the violated cut-point and overstates it elsewhere. Separate cut-point logistic models were unbiased with coverage 0.93 to 0.98 in every cell.

What it does not answer. One individual-data trial rather than an ML-NMR network; separate cut-point regressions as the relaxed model; fixed utilities; no dichotomized-publication arm. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-13 — Reconstruction ensembles were not imputations: their spread overstated the error in restricted mean survival and understated it in the extrapolated tail

Also bears on CMP-17.

Question. When individual data are reconstructed repeatedly from a published Kaplan-Meier curve, does the spread of the ensemble match the true reconstruction error, so that pooling it like multiple imputation is valid?

Answer. No, the refuting sentence fails: neither ensemble behaved as a multiple imputation. For RMST to 24 months at a 6-month risk table the variance calibration ratio (one for an imputation) was 31.8 to 44.3 for six analyst variants and 2.3 to 11.2 for 20 observation-model draws, the registered ‘both far from one’ outcome. Across cells the draws overstated the error of RMST (2.1 to 13.1) and 12-month survival (1.1 to 3.5) and understated it for Weibull survival at 48 months (0.15 to 0.77; 90% intervals covered the true-data value in 42% to 77%). Reconstruction error was small against sampling error (single-reconstruction RMSE 2% to 8% of the sampling SD for RMST, 11% to 26% for 48-month survival), so population coverage stayed at 0.91 to 0.98.

What it does not answer. The draws are not a posterior: Guyot’s event-count rounding is held fixed and there is no likelihood for the published ordinates. One arm and no downstream comparison, so propagation into a population-adjusted or component comparison (CMP-17) is not measured; no human digitization error; one sample size. The mechanism analyses are post hoc. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OUT-14 — Assessment grids that differ between trials

Question. When two trials assess progression on different visit schedules, how much does treating recorded event times as exact bias a transported RMST difference, and do midpoint imputation or an interval-censored model remove it?

Answer. Treating visit-recorded event times as exact biases a cross-trial RMST difference by half the spacing times the event probability by the horizon, with the sign set by which trial visits less often. A 3-month difference in spacing produced a spurious difference of 1.4 months with no true effect, and removed 1.0 of a true 1.2 months in another scenario. Moving events to the interval midpoint or fitting an interval-censored model reduced the bias below 0.1 month.

What it does not answer. Binary event-by-time endpoints, different maximum follow-up, reconstruction error from digitized curves and visit-window jitter were not simulated. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OVL-01 — Unsupported target mass matters only where the effect is modified

Also bears on OVL-03.

Question. When part of the target population lies outside the source trial’s support, do weight and covariate diagnostics identify the analyses that fail, or is the modification in the unsupported region, which they cannot see, what decides it?

Answer. Unsupported target mass caused bias only where the effect bent in the unsupported direction (up to 0.29 for G-computation and 0.30 for MAIC with 27% of the target unsupported), and none with linear modification or orthogonal alignment. Weight and covariate diagnostics, identical across those scenarios by construction, discriminated failed analyses at chance within an overlap level (AUROC 0.48 to 0.52). An outcome-informed score reached 0.59 to 0.66, missing the registered margin by 0.01; neither was usable. Trimming to the supported region was unbiased for the restricted estimand.

What it does not answer. Continuous outcome, two covariates; ML-NMR, ML-UMR and NMI not run; failure defined by non-coverage. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OVL-02 — An infeasible MAIC is signaled by its own balance table, not by its effective sample size

Question. MAIC has no solution when the target mean lies outside the convex hull of the source covariates. Does the optimizer say so, and can ESS, the maximum weight or the residual balance tell infeasible analyses from hard feasible ones?

Answer. The residual imbalance of the returned weights does, almost perfectly: above 0.001 in 99.9% of infeasible and 0.1% of feasible analyses. The optimizer reported convergence on 17% of infeasible problems. ESS ranks analyses well but the conventional rule of ESS below 10% of n flagged 73% of feasible analyses near the hull boundary as well as every infeasible one, so it cannot separate impossible from hard; and hard feasible analyses near the boundary undercovered badly anyway.

What it does not answer. Only feasibility of the mean constraint; support for the target joint law cannot be checked from published marginals. Normal covariates, MAIC on means, one sample size; the proposed diagnostics’ computability taxonomy was not simulated. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

OVL-06 — Planning a MAIC from the estimator’s influence function, not from n or Kish ESS

Question. Before any data exist, which planning quantity predicts the precision a MAIC will achieve: nominal sample size, Kish effective sample size, or the estimator’s influence-function variance evaluated at posited source and target laws?

Answer. The influence-function variance. It predicted the achieved SD within 10% in 85% of 72 scenarios, missing only at 150 patients with 10% prevalence. Kish ESS was within 10% in 58% and understated the achieved SD in 99% of scenarios, so it is optimistic, not the conservative quantity it is usually taken to be; nominal n was within 10% in 21%.

What it does not answer. The planner is assumed to know the source and target laws. Binary outcome, anchored-side precision and MAIC on means only; the unanchored bias floor and survival outcomes are not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-06 — A misclassified covariate that is both balanced and a modifier

Question. When a binary effect modifier is misclassified in the source, the target or both, which corrections does population adjustment need, and is correcting the outcome model alone enough?

Answer. No. MAIC on a misclassified binary modifier was biased even with identical assays in both studies (-0.03 to -0.13 on the log odds ratio scale), because reweighting recorded categories leaves each category’s true composition at the source’s; correcting the target prevalence alone did not help. Correcting only STC’s outcome model left bias of -0.012 to -0.051. Fully corrected STC and MAIC with weights chosen on the latent prevalence were unbiased (within 0.006).

What it does not answer. Known assay parameters; one binary covariate; no interval assessment. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-07 — Outcome ascertainment in an external comparator

Question. Does an interval-censored analysis cover the ascertainment differences between a trial and a routine-care comparator, or do missed detections and risk-dependent visiting still bias the RMST contrast?

Answer. Partly. With every progression detected, an interval-censored model removed the bias from the external arm’s sparser visits (within 0.05 months) and was not biased by visits that depended on patients’ risk at the strength simulated. When the external arm missed progression at some visits, every method was biased, by 0.25 to 0.38 months at a per-visit sensitivity of 0.85 and 0.67 to 0.94 at 0.7, against a true difference of 1.59 months. Missed detection needs its own bias parameter.

What it does not answer. No lag, linkage loss or false positives; one frailty and visiting strength; no sensitivity analysis over the detection parameter; no population adjustment. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-11 — Which failures double robustness covers

Question. Does a doubly robust MAIC protect against an omitted confounder, and what happens to a sensitivity grid built on a misspecified outcome model?

Answer. Double robustness protects against one misspecified nuisance model and nothing else. With an omitted confounder the augmented estimator carried exactly the same bias as correctly specified weighting or outcome-regression estimators, and with both models wrong the misspecification and omitted-variable biases added. A sensitivity grid built on a misspecified analysis is therefore shifted by the misspecification bias. With exact-balance weights, augmentation by an outcome model built from the balanced terms reproduces MAIC.

What it does not answer. Continuous outcome only; no cross-fitting; drMAIC E-value screen and a formally bias-indexed doubly robust estimator not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-13 — A simulated-covariate bias analysis by STC and by MAIC

Question. Is a MAIC version of the simulated-covariate bias analysis infeasible or too imprecise, as its authors assumed when restricting it to STC?

Answer. The MAIC extension of the simulated-covariate bias analysis was feasible in at least 99.8% of replicates across assumed target prevalences from 0.1 to 0.9, lost effective sample size to 85 to 109 of 300 only at the extreme, and had a reported SE 3% to 14% above STC’s. Both routes were unbiased. The restriction to STC was conservative rather than necessary. Both routes’ multiply imputed intervals widened along the sweep more than the estimate’s real spread (SE up to 0.28 against about 0.13), so coverage exceeded nominal.

What it does not answer. One binary unmeasured covariate, correct assumed relations, unanchored A side only, no decision reading. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-20 — One-at-a-time sensitivity analysis of two bias layers reports robustness that the joint analysis does not support

Question. Do one-at-a-time sensitivity analyses of population transportability and component invariance bound their joint effect on a cross-gap contrast, or can the layers offset and report spurious robustness?

Answer. No. With an omitted modifier (population layer) and component drift (bridge layer) acting on one cross-gap log odds ratio of -0.20, each one-at-a-time curve stayed within 0.16 and never reversed the decision, while the joint elicited box reached a bias of -0.41 and 7% of it reversed the decision; 23% to 26% of the box lay outside the one-at-a-time envelope at every scale. Component-by-modifier drift, which current tooling excludes, moved the estimate by 0.04 alone because it acts through its product with the modifier imbalance.

What it does not answer. Population-level calculation on one logistic model; no sampling, component network fit or survival scale; joint covariate law known, so the cost of eliciting it is not measured; component non-additivity and backbone interactions not included. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-23 — Bootstrap intervals over a bias grid when resamples fail

Question. When a bootstrap sensitivity analysis drops resamples whose weights fail near the edge of support, how does the interval behave, and is conditioning on success immaterial?

Answer. No. Near the edge of the source’s support, up to 67% of bootstrap resamples failed to produce MAIC weights, and the percentile interval from the survivors covered only 0.29 to 0.65 (0.14 to 0.56 counting failed original fits as giving no interval). The surviving resamples are those that contain the few patients who make the weights possible, so their spread understates the estimate’s. The failure share should be reported as a result and the interval withheld where it is material.

What it does not answer. One covariate, continuous outcome; cost and surrogate approaches and identified sets not studied. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-24 — Does a hazard-scale bias translate proportionally into RMST?

Question. Can a sensitivity analysis stated as a hazard-ratio bias be reinterpreted as a proportional bias in RMST, the quantity the decision uses?

Answer. No. The RMST change caused by a hazard-scale bias depends on the baseline hazard’s shape and the event level: at the same survival by 24 months the same bias moved RMST up to 1.44 times as much for a decreasing as for an increasing hazard, and the proportional (first-order) reinterpretation understated the change by about 25% with few events and overstated it by up to 8% with many. A hazard-ratio sensitivity analysis must be translated with the baseline curve or stated on the RMST scale.

What it does not answer. Only the hazard-to-RMST translation; the PAIC layer, pseudo-IPD and curve-based arms, reconstruction error and tipping sets not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-25 — Carrying a hazard-ratio bias into net benefit: the extrapolation family moved the decision more than the bias did

Question. Does propagating a bias-corrected hazard ratio and its interval into an economic model reproduce the decision uncertainty, or does the bias need to share a parameter space with the extrapolation model?

Answer. No. The probability of negative net benefit was 0.38 under point propagation of the reported hazard ratio and 0.67 when the bias (log hazard ratio within 0.15) and the treatment-effect extrapolation family were varied jointly. Almost all of the difference came from the extrapolation: a symmetric bias region alone moved the probability to 0.42, while a waning or cure extrapolation reversed the decision at the reported estimate. The worst joint combination (net benefit -9,618) was worse than any one-at-a-time scenario (-6,657).

What it does not answer. Deterministic survival curves instead of fitted extrapolations; three families, no splines; one symmetric bias parameter; costs and utilities fixed; one price. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

QBA-26 — A single-covariate benchmark against omitted composite structure

Question. Does the bias from dropping the strongest measured covariate bound the residual bias from several omitted variables, and does a scaled benchmark calibrate?

Answer. No. With three or six omitted variables each no stronger than the strongest measured covariate, the leave-one-covariate-out benchmark covered the true residual bias in at most 7% of analyses, the bias being 1.4 to 5.7 times the benchmark. Scaling by the square root of the number of omitted variables did not fix it; scaling by the number itself did when each omitted variable was weaker than the strongest measured one. The benchmark also shrank as omitted structure grew, through odds-ratio non-collapsibility.

What it does not answer. Omitted variables shifted in the same direction; one overlap level; negative-control route not run. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

SFW-10 — Stacking the weighting step into the MAIC sandwich makes intervals narrower, not better

Question. Does a stacked sandwich that accounts for weight estimation give better MAIC intervals than the fixed-weight sandwich or the ESS-based conventional variance, and what population does a trimmed MAIC actually estimate?

Answer. No. Stacking lowered the standard error in every untrimmed scenario (by up to 11%), because estimating calibration weights removes outcome variation the covariates explain; where MAIC intervals already undercover, at poor overlap, that made them worse (coverage down to 0.844). The fixed-weight sandwich was in band in 24 of 30 scenarios, the ESS-based variance in 18 and the stacked sandwich in 17, so current guidance stands. Trimming moved the estimand by up to 0.19 on the log odds ratio scale.

What it does not answer. No bootstrap and one sample size, so where stacking’s asymptotic advantage appears is not located. Binary outcome, three covariates, MAIC on means. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

SFW-12 — outstandR’s G-computation estimate moved by about 0.05 on the log odds ratio across defensible target reconstructions

Question. How much does outstandR’s G-computation estimate depend on how it reconstructs the aggregate trial’s covariate distribution from published means and SDs, when the true covariates are skewed and tail-dependent?

Answer. Partly. Across five defensible reconstructions of the aggregate trial’s covariates (normal or gamma margins; the IPD’s, the target’s, or no correlation), outstandR’s G-computation estimate ranged by 0.049 to 0.052 on the log odds ratio scale within a dataset: at the registered materiality threshold, but about a fifth of its sampling SD. The default (normal margins, IPD correlation) was biased by -0.02 to -0.06 on skewed covariates; gamma margins removed most of it. Tail dependence in the true copula changed the range by at most 0.002.

What it does not answer. Pairwise anchored comparison, two covariates, correct outcome model, point estimates only; no Bayesian G-computation, multiple-imputation marginalization or network use. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

SFW-14 — drMAIC 0.1.0 returns the MAIC estimate as its doubly robust estimate, with an interval six times too narrow

Question. Does drMAIC 0.1.0 implement a doubly robust MAIC with valid intervals: is its augmented estimate robust to a wrong weighting model when the outcome model is right, and do its intervals cover?

Answer. No. drMAIC 0.1.0 averages the outcome model over the individual-data patients instead of the target population, so its doubly robust estimate equals its MAIC estimate in every replicate (difference below 2e-15) and keeps MAIC’s bias when the weights are wrong and the outcome model right (0.066 against 0.004 for correct augmentation). Its analytic SE divides by n twice and mixes probability and log odds scales: 0.028 against an empirical SD of 0.17 to 0.22, coverage 0.20 to 0.26. Its percentile bootstrap covers 0.78 to 0.79.

What it does not answer. One package version, binary outcome, one sample size; G-MAIC not run; the time-to-event path (a weighted median labeled a hazard ratio) read from code, not simulated. Peer review has not been done.

Read it: PDF · Markdown · OpenDocument

Under way

Problem	Study	Status
OUT-11	What does a transported hazard ratio summarize under non-proportional hazards, and how do the recommended replacements behave?	running
CMP-26	Does cpaic’s two-stage component bridge fail under its assumptions’ violations only as known mechanisms predict?	designed
CMU-01	Does multinma’s split-chain integration check flag material integration error in a flexible survival ML-NMR contrast?	designed
CMU-03	Does it matter whether an SBC replicate for ML-NMR is drawn from the coded likelihood or end to end, and which test quantities detect a broken implementation?	designed
DIA-16	On a deletion suite with individual data on both sides, does a matching bridge recover the held-out randomized result?	designed
DIA-17	When a masked trial’s relative effect is recovered, are its arm outcomes recovered too?	designed
DIA-18	Once sampling error and heterogeneity are subtracted, does error in predicting later trials grow with calendar gap?	designed
IDN-08	Does the error of reconstructing a cut link depend on which link is cut, and on the separation across the gap?	designed
MOD-09	When does ML-NMR beat MAIC on error before fitting says so, and what does fitting it cost?	designed
MOD-11	Does a Dirichlet-process mixture over study populations improve a transported contrast at ITC study counts, and does tempering move more than the interval width?	designed
QBA-02	Can a randomized benchmark’s discrepancy distribution be used directly as a residual-bias prior?	designed
SFW-06	Does ML-NMR cost per effective draw grow with integration rows, or is sampler geometry what binds?	designed
SFW-08	Which predictive question does a mixed IPD and aggregate leave-one-out elpd answer, and does PSIS approximate it at the unit that matters?	designed
SFW-13	Does mlumr’s ML-UMR agree with an independent g-computation and carry exactly the analytic omitted-covariate bias?	designed
COV-01	Can significance screening find an effect modifier at the strengths where omitting it biases the transported effect?	designed
DIA-06	Do weighting, outcome-model, doubly robust and forest estimators fail at the same point as overlap goes?	designed
QBA-22	Which single number summarizing a joint bias region best predicts that a decision would flip?	designed
EST-07	What does treating reported target moments as known constants cost on a non-collapsible scale?	designed
MOD-10	Does a flexible treatment-effect surface inside a multilevel network likelihood pay for itself once part of the target lies outside the source support?	designed
CMP-17	Does the extrapolated tail’s reconstruction error reach a downstream survival comparison, or does the downstream likelihood dilute it?	designed
OVL-03	Does a support diagnostic integrated over a target law reconstructed from published marginals still find the analyses that fail?	designed

How a problem gets chosen

Not every open problem can be settled by a study, and the difference is not cosmetic. A simulation can measure bias, coverage and error rates, and it can show how badly a known nonidentification bites at realistic sample sizes. It cannot establish that a package is missing, that clinicians misread a plot, that a field under-reports something, or that a quantity is unidentifiable in principle. Those need software, a case study, a meta-research corpus, or a proof.

Every one of the 336 catalog entries was judged on both questions separately, by an external model, before the program started: is this a population-adjustment problem, and what kind of study would settle it. The queue below is the intersection.

Why an entry is not in the program	n
not a population-adjustment problem	101
needs analytic, not a simulation or case study	69
needs software, not a simulation or case study	22
needs consensus, not a simulation or case study	17
needs corpus, not a simulation or case study	16
needs none, not a simulation or case study	1

The queue

110 problems, in order. This is an ordering, not a schedule: a study that turns out to answer two entries closes both, and a study whose design review finds it would answer a neighbouring question rather than the stated one gets sent back before it is run.

#	Problem	Methods	Study	Answers	Feasibility	Catalog priority
1	CMP-13 Shared Gamma conflates within- and between-study modification	ML-NMR	simulation	yes	4/5	Very high
2	CMP-14 Aggregate-data-only interactions may be prior-driven	ML-NMR	simulation	yes	4/5	Very high
3	CMU-02 Priors, not data, can drive posteriors in weakly identified models	ML-NMR, ML-UMR, NMI	simulation	yes	4/5	Very high
4	EST-07 Reported target moments are treated as known constants	MAIC, STC, ML-NMR, ML-UMR	simulation	yes	4/5	Very high
5	MIS-03 Nothing propagates sampling error in reported target moments	MAIC, STC, ML-NMR, ML-UMR	simulation	yes	4/5	Very high
6	DIA-03 PAIC diagnostics are not evaluated as classifiers of realized error	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	yes	3/5	Very high
7	IDN-05 The shared effect modifier check has low and uncalibrated power	ML-NMR	simulation	yes	3/5	Very high
8	IDN-06 ML-NMR interactions can rest solely on aggregate-data variation	ML-NMR	simulation	yes	3/5	Very high
9	OUT-11 Non-proportional hazards and the transported hazard ratio	MAIC, STC, ML-NMR	simulation	yes	3/5	Very high
10	DIA-06 Failure signatures across PAIC estimator families are uncharacterized	MAIC, STC, ML-NMR, ML-UMR	simulation	yes	2/5	Very high
11	MOD-02 Convenience functional forms and untested flexible alternatives	STC, ML-NMR, ML-UMR	simulation	yes	2/5	Very high
12	CMP-20 Poisson aggregate likelihood needs zero exposure-rate covariance	ML-NMR	simulation	yes	5/5	High
13	COV-03 Design-dependent roles of prognostic covariates	MAIC, STC, ML-NMR, ML-UMR	simulation	yes	4/5	High
14	COV-12 No aggregate integration method for median-based covariate summaries	ML-NMR, ML-UMR	simulation	yes	4/5	High
15	COV-14 Per-comparator adjustment sets yield non-comparable PAIC contrasts	MAIC, STC	simulation	yes	4/5	High
16	MOD-16 No criterion chooses pooled or arm-separate matching in anchored MAIC	MAIC	simulation	yes	4/5	High
17	OUT-02 Rare events, separation, and zero-event studies in weighted synthesis	MAIC	simulation	yes	4/5	High
18	CMP-12 No two-stage adjusted estimator carries multi-arm covariance	MAIC, STC, ML-NMR	simulation	yes	3/5	High
19	DEC-28 No inference is reported for an estimated optimal-treatment boundary	ML-NMR, ML-UMR	simulation	yes	3/5	High
20	SFW-06 ML-NMR runtime can constrain repeated sensitivity analyses at scale	ML-NMR	simulation	yes	3/5	High
21	CMP-06 Measurement error and component misclassification are ignored	ML-NMR	simulation	yes	2/5	High
22	CMU-03 Bayesian component ML-NMR and bridge models lack SBC validation	ML-NMR, ML-UMR, NMI	simulation	yes	2/5	High
23	EST-11 No simulation crosses target choice with second-stage transport error	ML-UMR	simulation	yes	2/5	High
24	CMP-24 Edge influence is approximate for multi-arm and random effects	ML-NMR, NMI	simulation	yes	3/5	Medium
25	MOD-01 Conventional STC reports a conditional effect at target means	STC	simulation	partly	5/5	Very high
26	OVL-01 Weak overlap leaves target regions the source data cannot support	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	4/5	Very high
27	CMP-11 Two-stage component PAIC adjusts only the IPD edges	MAIC, STC, ML-NMR	simulation	partly	3/5	Very high
28	COV-01 Deciding which covariates are true effect modifiers	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
29	DEC-11 Intervals omit selection, reconstruction and target-summary error	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
30	DIA-08 PAIC simulations generate only shared, linear effect modification	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
31	DIA-09 Narrow outcome families and hazard-ratio-only survival scoring	MAIC, STC, ML-NMR, ML-UMR	simulation	partly	3/5	Very high
32	DIA-14 QBA is judged on oracle recovery, not honest classification	MAIC, STC, ML-UMR	simulation	partly	3/5	Very high
33	EST-06 The MAIC paradox: swapping the IPD trial can reverse the comparison	MAIC	simulation+case	partly	3/5	Very high
34	IDN-01 Transitivity and conditional constancy cannot be tested in-sample	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
35	IDN-07 No agreed criterion for when a held-out-trial discrepancy falsifies	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
36	MIS-01 Missing aggregate summaries and unavailable IPD have no remedy	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
37	OVL-02 ESS is an insufficient overlap diagnostic and no battery is agreed	MAIC	simulation	partly	3/5	Very high
38	QBA-11 Outcome-model misspecification and doubly robust QBA	MAIC, STC, ML-UMR	simulation	partly	3/5	Very high
39	CMP-17 Survival pseudo-IPD reconstruction uncertainty is not propagated	ML-NMR	simulation+case	partly	2/5	Very high
40	CMP-18 Proportional or time-constant component effects may fail	ML-NMR	simulation	partly	2/5	Very high
41	CMP-21 Missing data and MNAR sensitivity are unaddressed in component PAIC	MAIC, STC, ML-NMR	simulation	partly	2/5	Very high
42	DEC-02 No factorial study ranks which choice most often flips a decision	MAIC, STC, ML-NMR, ML-UMR	simulation	partly	2/5	Very high
43	DIA-02 No agreed ESS definition and no support-geometry diagnostics	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	Very high
44	DIA-13 QBA bias mechanisms are not crossed with the primary factors	MAIC, STC, ML-UMR	simulation	partly	2/5	Very high
45	IDN-10 Transported control-arm outcomes are uncalibrated where it matters	MAIC, STC, ML-UMR	simulation+case	partly	2/5	Very high
46	MOD-09 No criterion says when ML-NMR should be preferred to MAIC or STC	MAIC, STC, ML-NMR	simulation+case	partly	2/5	Very high
47	QBA-13 Simulated-covariate QBA is an STC route, not a general solution	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	Very high
48	QBA-22 Total bias is non-additive, so one-at-a-time analyses mislead	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	Very high
49	QBA-24 Survival QBA beyond the hazard ratio with limited aggregate data	MAIC, STC, ML-UMR	simulation+case	partly	2/5	Very high
50	SFW-10 maicplus variance, trimming and estimand disclosure gaps	MAIC	simulation	partly	2/5	Very high
51	CMP-26 Unreleased cpaic has no tracked simulation benchmark or replication	MAIC, ML-NMR	simulation+case	partly	1/5	Very high
52	DIA-17 Held-out IPD is not used to benchmark transport against truth	MAIC, STC, ML-NMR, ML-UMR, NMI	case-study	partly	1/5	Very high
53	IDN-08 No curated benchmark validates bridged cross-gap comparisons	MAIC, NMI	case-study	partly	1/5	Very high
54	QBA-20 Component PAIC needs joint population-and-bridge bias analysis	ML-NMR, ML-UMR	simulation	partly	1/5	Very high
55	OUT-14 Pseudo-IPD cannot express delayed entry or assessment windows	MAIC, STC, ML-NMR	simulation	partly	4/5	High
56	ADJ-04 Flexible outcome surfaces and support diagnostics in standardization	STC, ML-NMR	simulation	partly	3/5	High
57	CMP-15 Target joint covariate distribution is only approximately known	ML-NMR	simulation	partly	3/5	High
58	CMP-16 Unreleased cpaic shares one tau and one IPD-row residual variance	ML-NMR	simulation	partly	3/5	High
59	COV-02 No evidence on how the modifier set changes across effect scales	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation+case	partly	3/5	High
60	COV-09 Measurement non-equivalence of covariates across trials	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	High
61	DIA-10 Data access is simulated as benign rather than informative	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	High
62	EST-12 Treatment hierarchies lack a declared target-population referent	ML-NMR	simulation+case	partly	3/5	High
63	HET-04 Node splitting compares paths without standardizing to a target	ML-NMR	simulation	partly	3/5	High
64	MIS-02 No robustness theory for censoring-weighted survival transport	MAIC	simulation	partly	3/5	High
65	OUT-08 Overdispersed counts and recurrent events as total disease burden	MAIC, ML-NMR, ML-UMR	simulation	partly	3/5	High
66	OUT-10 Proportional odds, category probabilities, and ordinal utilities	ML-NMR	simulation	partly	3/5	High
67	OUT-13 Reconstruction uncertainty from digitized Kaplan-Meier curves	MAIC, STC, ML-NMR	simulation+case	partly	3/5	High
68	OVL-06 No standardized sample-size planning guidance or software for PAIC	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	High
69	QBA-26 Benchmarking and negative controls as data-based calibration	MAIC, STC, ML-UMR	simulation	partly	3/5	High
70	SFW-14 drMAIC has no methods paper and no independent evaluation	MAIC	simulation	partly	3/5	High
71	ADJ-15 Privacy guarantees and governance for shared trial-level data	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	High
72	COV-04 Selection uncertainty in PAIC interaction shrinkage is uncalibrated	STC, ML-NMR, ML-UMR	simulation	partly	2/5	High
73	DIA-07 Scenario grids omit network topology and information placement	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	High
74	HET-02 Multi-arm covariance is lost when contrasts are adjusted pairwise	MAIC, ML-NMR	simulation	partly	2/5	High
75	HET-10 No calibrated small-study interval method for population-adjusted NMA	MAIC, STC, ML-NMR	simulation	partly	2/5	High
76	IDN-19 No principled basis for shared effect-modifier treatment classes	MAIC, STC, ML-NMR, ML-UMR	simulation+case	partly	2/5	High
77	MIS-04 Model and bridge selection uncertainty never enters the interval	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	High
78	MOD-04 No post-selection inference for model choice in population adjustment	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	High
79	OVL-03 No support diagnostic aggregates over a PAIC target distribution	STC, ML-NMR, ML-UMR	simulation	partly	2/5	High
80	SFW-08 No agreed leave-one-out unit for mixed IPD and aggregate likelihoods	ML-NMR	simulation+case	partly	2/5	High
81	SFW-12 outstandR is pairwise only with no network or disconnected validation	STC, ML-NMR	simulation+case	partly	2/5	High
82	SFW-13 cpaic and mlumr are unreleased and independently unvalidated	MAIC, STC, ML-NMR, ML-UMR	simulation+case	partly	2/5	High
83	DIS-21 Match acceptance thresholds are uncalibrated against bridge bias	NMI	simulation+case	partly	1/5	High
84	ADJ-10 Data-driven modifier and causal-structure discovery with few trials	STC, ML-NMR, ML-UMR	simulation	partly	3/5	Medium-high
85	MOD-10 Flexible interaction surfaces inside network evidence synthesis	ML-NMR, ML-UMR	simulation	partly	2/5	Medium-high
86	MOD-11 No mixture-over-populations model transports to a declared target	ML-NMR, ML-UMR	simulation	partly	2/5	Medium-high
87	CMU-01 ML-NMR integration cost is unaddressed for flexible survival models	ML-NMR	simulation	partly	1/5	Medium-high
88	ADJ-17 Representation learning for low-dimensional population encodings	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation+case	partly	1/5	Medium
89	DEC-01 Method evaluations score estimator error, not decision error	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	yes	4/5	Very high
90	MOD-15 No calibrated rule says when within- and across-trial interactions may be pooled	ML-NMR	simulation	yes	4/5	High
91	DIS-11 Predictive criteria and population comparison cannot validate a bridge	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
92	IDN-09 Negative controls for ITC lack catalogues, power and calibration	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	3/5	Very high
93	CMP-03 Strict component additivity may be clinically false	ML-NMR	simulation	partly	2/5	Very high
94	OUT-07 Informative censoring and outcome-dependent missingness	MAIC, STC, ML-NMR, ML-UMR	simulation	partly	2/5	Very high
95	QBA-06 Measurement and misclassification QBA under population adjustment	MAIC, STC, ML-NMR, ML-UMR	simulation	partly	2/5	Very high
96	QBA-23 Propagating every uncertainty layer without prohibitive cost	MAIC, STC, ML-NMR, ML-UMR, NMI	simulation	partly	2/5	Very high
97	DIA-16 No curated public suite for bridge-deletion validation with IPD	MAIC, STC, ML-NMR, ML-UMR, NMI	case-study	partly	1/5	Very high
98	QBA-02 No prospectively blinded calibration program for PAIC and QBA	MAIC, STC, ML-UMR	case-study	partly	1/5	Very high
99	QBA-25 Decision QBA: net benefit, tipping surfaces and extrapolation	MAIC, STC, ML-NMR, ML-UMR	simulation+case	partly	1/5	Very high
100	ADJ-09 Cross-environment invariance as a screen for bridges and modifiers	ML-NMR, ML-UMR	simulation	partly	4/5	High
101	CMP-22 IPD availability may be informative rather than random	MAIC, STC, ML-NMR	simulation	partly	3/5	High
102	COV-10 Effect-modifier selection does not check interaction consistency	ML-NMR	simulation	partly	3/5	High
103	DIS-03 Baseline risk is not a neutral anchor for disconnected evidence	MAIC, STC, ML-UMR	simulation	partly	3/5	High
104	HET-03 Single shared heterogeneity variance across classes and populations	ML-NMR, ML-UMR	simulation	partly	3/5	High
105	QBA-07 Outcome ascertainment and the observation process in external data	MAIC, STC, ML-UMR	simulation	partly	2/5	High
106	DIA-18 Constancy assumptions are never tested across calendar time	MAIC, STC, ML-NMR, ML-UMR, NMI	case-study	partly	1/5	High
107	OUT-06 Change scores, trajectories, and responder-threshold estimands	MAIC, STC, ML-NMR, ML-UMR	simulation	partly	4/5	Medium-high
108	MOD-18 Risk-modeling network meta-regression treats a fitted risk score as observed	ML-NMR	simulation+case	partly	3/5	Medium-high
109	OUT-05 Unequal variances, skew, and bounded scales in continuous outcomes	STC, ML-NMR, ML-UMR	simulation	partly	3/5	Medium-high
110	DEC-17 Evidence-provenance measures do not propagate heterogeneity uncertainty	ML-NMR, ML-UMR	simulation+case	partly	2/5	Medium-high

What each study has to do

Simulation studies follow ADEMP: aims, data-generating mechanisms, estimands, methods, performance measures (Morris, White and Crowther 2019). Two rules follow from it and are enforced rather than encouraged.

Every performance measure carries a Monte Carlo standard error.

A bias of 0.02 is uninterpretable without knowing whether the Monte Carlo error is 0.001 or 0.03. The replicate count is derived in the protocol from a target Monte Carlo standard error, not chosen because it is a round number.

Failed replicates are results.

A method that converges on 60% of replicates and is unbiased on those is not an unbiased method. Convergence is reported for every method in every scenario.

The protocol is committed before the run.

It fixes the design and states in advance what result would count as showing the problem is real and what would count as showing it is not. The commit that adds it is the timestamp.

Code, protocols and results are at github.com/choxos/ITC-open-problems/tree/main/studies.