# Protocol: separating within-trial from across-trial component effect modification

**Target problems.** CMP-13 *Shared Gamma conflates within- and between-study
modification* and IDN-06 *ML-NMR interactions can rest solely on aggregate-data
variation*.

**Status.** Registered before any result was seen; the commit that adds this file is the
timestamp. Section 10 records amendments.

**Reporting standard.** ADEMP (Morris, White and Crowther 2019,
[doi:10.1002/sim.8086](https://doi.org/10.1002/sim.8086)).

---

## 1. The problem, and the algebra that makes it exact

Component multilevel network meta-regression estimates a treatment-by-covariate
interaction from individual and aggregate data together. Current practice fits one shared
interaction matrix on a globally centered covariate. For a binary modifier with
study-specific prevalence $p_s$,

$$\gamma\,(X - 0.5) \;=\; \underbrace{\gamma\,(X - p_s)}_{\text{within trial}} \;+\; \underbrace{\gamma\,(p_s - 0.5)}_{\text{across trials}} .$$

So a single shared $\gamma$ does not merely pool two sources of information. It **imposes
that the within-trial and the across-trial coefficients are equal.**

The two are not the same kind of quantity. The within-trial coefficient $\gamma_W$ is
identified by randomized comparisons inside a trial and is causal. The across-trial
coefficient $\gamma_B$ is the association between a study's case mix and its treatment
effect across trial populations that nobody randomized, and is confounded by everything
else that differs between them. Separating them by centering covariates at the trial mean
is standard in individual-participant meta-analysis (Hua et al. 2017,
[doi:10.1002/sim.7171](https://doi.org/10.1002/sim.7171); Freeman et al. 2018,
[doi:10.1002/jrsm.1300](https://doi.org/10.1002/jrsm.1300)). It has not been ported to
component ML-NMR.

`multinma`'s `center` argument does not do this. It centers regression terms about the
**overall** mean, which is numerical conditioning; only per-study centering separates the
two sources. This was checked against the package documentation for version 0.9.1.9002.

The algebra above already establishes that the shared model is misspecified whenever
$\gamma_W \neq \gamma_B$. What is unknown, and what this study measures, is **how much
that costs and when**.

## 2. Aims

1. Measure bias and interval coverage for the causal within-trial component interaction
   under a shared $\Gamma$, and locate the discordance at which the damage becomes
   material.
2. Test the catalog's stated *mechanism*: CMP-13 says aggregate studies pull the shared
   estimate. If so, the bias should vanish when every trial supplies individual data.
3. Determine what happens to a component that has no within-trial information at all,
   which is IDN-06's case, and whether an estimator that declines to report is preferable
   to one that supplies a number from an ecological gradient.

## 3. Data-generating mechanism

Twelve two-arm randomized trials over four regimens built from two components:
control $=(0,0)$, A $=(1,0)$, B $=(0,1)$, AB $=(1,1)$. One binary modifier $X$ with
study prevalence $p_s$; write $z = X - p_s$. The individual log rate is

$$\eta = \alpha_s + \beta z + \sum_{c \in \{A,B\}} a_c \{\, \delta_c + \gamma_{W,c}\, z + q_c(p_s) \,\},$$

with $\beta = \log 1.5$, $\delta_A = \log 0.70$, $\delta_B = \log 0.75$,
$\alpha_s = \log 0.30 + u_s$ and $u_s \sim \mathrm{Unif}(-0.2, 0.2)$. Outcomes are Poisson
with unit exposure. For the $\rho$ patterns $q_c(p_s) = \gamma_{B,c}(p_s - 0.5)$ with
$\gamma_B = \rho\,\gamma_W$.

**Aggregate arms are generated by exact integration**: the arm total is Poisson with mean
$n\{(1-p_s)e^{\eta|X=0} + p_s e^{\eta|X=1}\}$, which is also exactly the distribution of
the sum of the individual outcomes. Generating them by a regression on $p_s$ would test a
method nobody uses, since ML-NMR integrates precisely to avoid that.

### Factors

| Factor | Levels |
|---|---|
| Network structure | isolating, bundled |
| Discordance | $\rho \in \{1, 0.5, 0, -0.5, -1\}$, plus null, between-only, nonlinear |
| IPD availability | all 12, six, four, four lowest-prevalence, none containing component B |
| Participants per arm | 100, 400 |
| Prevalence spread | narrow (0.35 to 0.65), wide (0.15 to 0.85) |

Fully factorial: **320 scenarios**, 2000 replicates each, 640,000 datasets.

**Network structure** is a factor because of a review finding. In the originally proposed
network every contrast added exactly one component, so each interaction was recoverable
from a directly component-isolating comparison and the study would have tested ordinary
network meta-regression while claiming to test component models. The `bundled` network
contains `control:AB`, which moves both components at once, and `A:B`, which removes one
while adding the other, so contrasts are aliased across components. Both structures still
identify both component main effects.

**The `no-B-ipd` scheme** gives individual data only to trials in which component B never
appears in any arm. Selecting by which component a contrast *changes* does not achieve
this, and the reason is worth recording: the within-trial estimator identifies $\gamma_W$
from the covariate slope inside an arm, which is $\beta + \sum_c a_c \gamma_{W,c}$, not
from the treatment contrast. A single arm containing component B therefore identifies
$\gamma_{W,B}$ once $\beta$ is known, even where no contrast in that trial changes B.

### What the mechanism makes true, and therefore what the study cannot see

The modifier is binary, the link is log, the outcome is Poisson, the component structure
is additive with no synergy, there is no random treatment-effect heterogeneity and no
inconsistency, and study prevalences are reported exactly. Under the $\rho$ patterns the
split model is correctly specified and the shared model is correctly specified only at
$\rho = 1$. The study therefore quantifies the cost of an algebraically known restriction;
it cannot establish that real networks contain within/across disagreement, or how large it
typically is. The `nonlinear` pattern is the one case where no linear model is correct.

Under `nonlinear`, only $\gamma_W$ is assessed. The review established that the implied
pseudo-true $\gamma_B$ is a likelihood-weighted projection over each component's own
informative trials rather than the unweighted zero the design assumed, so assessing
$\gamma_B$ coverage there would compare against the wrong number.

## 4. Estimands

**Primary.** The within-trial component interaction $\gamma_{W,c}$, $c \in \{A,B\}$: the
difference in conditional log rate ratio for adding component $c$ between otherwise
comparable randomized individuals with $X=1$ and $X=0$ in the same trial. Truth is the
coefficient the pattern specifies; no Monte Carlo evaluation is needed.

## 5. Methods

All fit the same integrated component likelihood by maximum likelihood with analytic
gradients, study intercepts profiled analytically, and Newton polishing until the maximum
absolute score is below $10^{-6}$.

| Method | Interaction parameterization | What may inform $\gamma_W$ |
|---|---|---|
| **shared-info** (status quo) | one $\gamma_c$ on $X - 0.5$ | everything, with $\gamma_W = \gamma_B$ imposed |
| **shared-sandwich** | identical point estimate | as above, with a 12-study cluster sandwich |
| **joint-split** | $\gamma_{W,c}$ on $z$ and $\gamma_{B,c}$ on $p_s - 0.5$ | everything, including aggregate curvature |
| **ipd-anchored** | $\gamma_{W,c}$ from individual data only, free intercept per study-arm | randomized within-arm covariate contrasts only |

`shared-sandwich` exists so the comparison cannot be won merely by giving the proposed
method a more robust variance estimator.

`ipd-anchored` **declines to report** a component that appears in no individual-data arm,
returning it as not estimable rather than supplying a number from an ecological gradient.
That is the behavior IDN-06 asks for, and whether it is preferable to supplying a number
is one of the things being tested.

**Non-convergence**: nonzero optimizer code, maximum absolute score above $10^{-6}$, any
absolute estimate above 5, minimum information eigenvalue at or below $10^{-8}$, condition
number above $10^{10}$, or a missing interval. A failed replicate contributes rows with the
reason recorded.

**Failures are not dropped silently.** Every measure is reported over converged replicates
and again counting a failure as non-coverage; any scenario where the two differ by more
than one percentage point is flagged. Not-estimable is counted separately from failure,
because declining to answer is a design feature of one method and a different thing from
failing to converge.

## 6. Performance measures

Primary: coverage of the nominal 95% interval for $\gamma_{W,c}$. Secondary: bias,
standardized bias, empirical and model standard errors, MSE, interval width, rate of
declining to report, and convergence. Every measure with a Monte Carlo standard error.

**2000 replicates.** At coverage 0.95 the Monte Carlo SE is
$\sqrt{0.95 \times 0.05/2000} = 0.0049$, so a two-point coverage error is four Monte Carlo
SEs from nominal. Methods share the replicate, so paired contrasts are more precise still.
A timed pilot measured 42.6 ms per replicate for all four methods, giving about 57 minutes
on eight cores for the full design.

## 7. Decision rule, fixed in advance

**Material at mild discordance.** At $\rho = 0.5$, shared-$\Gamma$ coverage for at least
one component is below 0.90 with its Monte Carlo 95% interval excluding 0.95, in at least
one network structure at $n = 400$.

**Material only at strong discordance.** Coverage stays within 0.93 to 0.97 at
$\rho = 0.5$ but falls below 0.90 at $\rho = 0$ or $\rho = -1$.

**Not material.** Coverage stays within 0.93 to 0.97 at every $\rho$, and median absolute
standardized bias across discordant patterns is below 0.10.

**Uninformative** if any method converges on fewer than 95% of replicates in more than 10%
of scenarios; or conditional and unconditional coverage differ by more than 0.01 anywhere;
or either the shared or the split model fails its own negative controls, meaning coverage
outside 0.93 to 0.97 at $\rho = 1$ or under `null`, where both are correctly specified.

### The catalog's mechanism, tested separately

CMP-13 states that **aggregate studies** pull a shared $\Gamma$ toward the across-trial
association. That is a claim about cause, and it is separable from the claim about size.

- **Supported** if, under `all` IPD, absolute standardized bias for shared $\Gamma$ is
  below 0.10 in every discordant pattern.
- **Refuted** if, under `all` IPD, it exceeds 0.20 in at least half of them, which would
  show the conflation is a property of the parameterization rather than of the data type.

## 8. Threats, and what was done about each

| Threat | What was done |
|---|---|
| The study tests ordinary meta-regression while claiming to test component models | Network structure is a factor; `bundled` contrasts move two components at once |
| The aggregate-only case IDN-06 describes is never produced | `no-B-ipd` leaves a component with no within-trial information anywhere |
| Only extreme discordances are examined, so the sign is fixed by the design | $\rho$ is swept through 1, 0.5, 0, $-0.5$, $-1$ |
| The proposed method wins on its variance estimator | `shared-sandwich` gives the status quo a cluster-robust interval on the identical point estimate |
| Failed fits favor whichever method fails on its hardest replicates | Both conditional and unconditional summaries, with declining-to-report counted separately |
| The decision rule cannot be evaluated | Rewritten from a five-clause count over 72 cells to statements checkable against measured Monte Carlo error |

## 9. Scope

This is a **maximum-likelihood analogue** of component ML-NMR. Implemented ML-NMR is
Bayesian, and priors matter most exactly where interactions are weakly identified, so the
coverage numbers here do not transfer directly to `multinma`. What does transfer is the
algebra of section 1, which is a property of the parameterization and not of the
estimation paradigm, and therefore the direction of the bias. This limitation is stated
rather than bridged; a Bayesian evaluation is a separate study.

Also out of scope: continuous or multiple correlated modifiers, misspecified or uncertain
aggregate covariate distributions, component synergy, random component-effect
heterogeneity, inconsistency, time-to-event outcomes, and links other than log.

Accordingly this study **answers CMP-13** for the maximum-likelihood analogue and
**answers a named part of IDN-06**, namely what happens when an interaction has no
within-trial support. It does not touch IDN-05.

## 10. Amendments

### 2026-07-28, after the run: the two regimes are analyzed separately

**What changed.** The analysis splits the grid on the `ipd` factor: a primary regime of 256
scenarios in which every component has within-trial information (`all`, `six`, `four`,
`four-low`), and 64 scenarios of `no-B-ipd` in which one component has none.

**Why.** In `no-B-ipd` the shared and joint models are near-unidentified for that component.
They fail to return an interval on 11% of replicates against 0.02% elsewhere, their model
standard errors run three to ten times their empirical ones, and coverage tends to one
because the intervals are enormous. Pooled with the rest, that regime makes the negative
controls fail and the pooled numbers describe neither regime.

**Why this is not the move study 1 was criticized for.** Study 1 of this program restricted
its analysis by the *observed coverage* of a method, which is selection on an outcome, and
peer review was right to call that exploratory. This split is on a **level of a factor fixed
before the run**. Nothing about the observed results decides which scenarios go where, and
both regimes are reported in full with their own conclusions.

**What it does not change.** No threshold, no factor, no performance measure, no replicate
count, and no scenario's data.

### 2026-07-28: the cluster-robust comparator fails its own controls

`shared-sandwich` was included so the comparison could not be won by giving the proposed
method a better variance estimator. It fails its own negative controls in all 128 control
scenarios, covering 0.754 to 0.929 where the shared model is correctly specified. With
twelve clusters the cluster-robust variance is badly anticonservative. It is therefore
reported as a finding in its own right and is not used as the status quo; the status quo is
the model-based interval, which passes its controls.