Effect Modification and SPFA Testing

Overview

This vignette focuses on the Shared Prognostic Factor Assumption (SPFA) - a critical assumption in ML-UMR - and how to test for effect modification when this assumption may be violated.

The Shared Prognostic Factor Assumption (SPFA)

What is SPFA?

SPFA assumes that prognostic factors have the same effect on outcomes regardless of treatment assignment. Mathematically:

\text{SPFA: } \beta_{PF}^{(A)} = \beta_{PF}^{(B)} = \beta_{PF}

Where: - \beta_{PF}^{(A)} = regression coefficients for prognostic factors under Treatment A - \beta_{PF}^{(B)} = regression coefficients under Treatment B

When SPFA Holds

Under SPFA, the linear predictors are: - Treatment A: g(\theta_i^A) = \alpha^A + \mathbf{x}_i^T \beta_{PF} - Treatment B: g(\theta^B) = \alpha^B + \mathbf{x}^T \beta_{PF}

The only difference between treatments is the intercept (\alpha^A vs \alpha^B).

When SPFA is Violated (Effect Modification)

When SPFA is violated, prognostic factors affect outcomes differently depending on treatment: - Treatment A: g(\theta_i^A) = \alpha^A + \mathbf{x}_i^T \beta_{PF}^{(A)} - Treatment B: g(\theta^B) = \alpha^B + \mathbf{x}^T \beta_{PF}^{(B)}

This is called effect modification - the treatment effect varies depending on patient characteristics.

library(mlumr)
library(dplyr)
library(ggplot2)

Simulating Effect Modification Scenarios

The generate_simulation_data() function provides three scenarios:

“none”: SPFA holds perfectly (\beta^A = \beta^B)
“moderate”: Weak violation (\beta^B \approx 0.7 \times \beta^A)
“strong”: Strong violation (\beta^B \approx 0.5 \times \beta^A)

Scenario 1: SPFA Holds (No Effect Modification)

set.seed(123)

data_no_em <- generate_simulation_data(
  n_index = 300,
  n_comparator = 200,
  n_covariates = 2,
  effect_modification = "none",  # SPFA holds
  population_shift = 0.5,
  correlation = 0.3,
  seed = 123
)

print(data_no_em)
#> ML-UMR Simulation Data
#> ======================
#> 
#> Design:
#>   Likelihood: binomial 
#>   Index treatment (IPD): n = 300 
#>     Events: 132 
#>   Comparator treatment (AgD): 1 subgroups/studies
#>     Total n = 200 , Total events = 128 
#>   Number of covariates: 2 
#>   Effect modification: none 
#> 
#> True Parameters:
#>   mu_index = -0.5 
#>   mu_comparator = -0.3 
#>   beta_index: 0.5, 1 
#>   beta_comparator: 0.5, 1 
#> 
#> True Marginal Effects:
#>   LOR (index pop): -0.156 
#>   LOR (comparator pop): -0.156 
#>   RD (index pop): -0.038 
#>   RD (comparator pop): -0.037
cat("\nTrue Parameters:\n")
#> 
#> True Parameters:
cat("  beta_index:", paste(round(data_no_em$truth$beta_index, 3), collapse = ", "), "\n")
#>   beta_index: 0.5, 1
cat("  beta_comparator:", paste(round(data_no_em$truth$beta_comparator, 3), collapse = ", "), "\n")
#>   beta_comparator: 0.5, 1

Scenario 2: Moderate Effect Modification

data_moderate_em <- generate_simulation_data(
  n_index = 300,
  n_comparator = 200,
  n_covariates = 2,
  effect_modification = "moderate",
  population_shift = 0.5,
  seed = 456
)

print(data_moderate_em)
#> ML-UMR Simulation Data
#> ======================
#> 
#> Design:
#>   Likelihood: binomial 
#>   Index treatment (IPD): n = 300 
#>     Events: 143 
#>   Comparator treatment (AgD): 1 subgroups/studies
#>     Total n = 200 , Total events = 132 
#>   Number of covariates: 2 
#>   Effect modification: moderate 
#> 
#> True Parameters:
#>   mu_index = -0.5 
#>   mu_comparator = -0.3 
#>   beta_index: 0.5, 1 
#>   beta_comparator: 0.35, 0.7 
#> 
#> True Marginal Effects:
#>   LOR (index pop): -0.102 
#>   LOR (comparator pop): 0.046 
#>   RD (index pop): -0.025 
#>   RD (comparator pop): 0.011
cat("\nTrue Parameters (30% difference):\n")
#> 
#> True Parameters (30% difference):
cat("  beta_index:", paste(round(data_moderate_em$truth$beta_index, 3), collapse = ", "), "\n")
#>   beta_index: 0.5, 1
cat("  beta_comparator:", paste(round(data_moderate_em$truth$beta_comparator, 3), collapse = ", "), "\n")
#>   beta_comparator: 0.35, 0.7

Scenario 3: Strong Effect Modification

data_strong_em <- generate_simulation_data(
  n_index = 300,
  n_comparator = 200,
  n_covariates = 2,
  effect_modification = "strong",
  population_shift = 0.5,
  seed = 789
)

print(data_strong_em)
#> ML-UMR Simulation Data
#> ======================
#> 
#> Design:
#>   Likelihood: binomial 
#>   Index treatment (IPD): n = 300 
#>     Events: 135 
#>   Comparator treatment (AgD): 1 subgroups/studies
#>     Total n = 200 , Total events = 99 
#>   Number of covariates: 2 
#>   Effect modification: strong 
#> 
#> True Parameters:
#>   mu_index = -0.5 
#>   mu_comparator = -0.3 
#>   beta_index: 0.5, 1 
#>   beta_comparator: 0.25, 0.5 
#> 
#> True Marginal Effects:
#>   LOR (index pop): -0.098 
#>   LOR (comparator pop): 0.157 
#>   RD (index pop): -0.024 
#>   RD (comparator pop): 0.039
cat("\nTrue Parameters (50% difference):\n")
#> 
#> True Parameters (50% difference):
cat("  beta_index:", paste(round(data_strong_em$truth$beta_index, 3), collapse = ", "), "\n")
#>   beta_index: 0.5, 1
cat("  beta_comparator:", paste(round(data_strong_em$truth$beta_comparator, 3), collapse = ", "), "\n")
#>   beta_comparator: 0.25, 0.5

Testing SPFA: Model Comparison Approach

Step 1: Prepare Data

# Use the moderate effect modification scenario
ipd <- set_ipd_unanchored(
  data = data_moderate_em$ipd,
  treatment = "treatment",
  outcome = "outcome",
  covariates = c("x1", "x2")
)

agd <- set_agd_unanchored(
  data = data_moderate_em$agd,
  treatment = "treatment",
  outcome_n = "n_total",
  outcome_r = "n_events",
  cov_means = c("x1_mean", "x2_mean"),
  cov_sds = c("x1_sd", "x2_sd"),
  cov_types = c("continuous", "continuous")
)

network <- combine_unanchored(ipd, agd)

network <- add_integration_unanchored(
  network,
  n_int = 64,
  x1 = distr(qnorm, mean = x1_mean, sd = x1_sd),
  x2 = distr(qnorm, mean = x2_mean, sd = x2_sd)
)
#> Computing correlation matrix from IPD...
#> Added 64 integration points for AgD

Step 2: Fit Both Models

# Model 1: SPFA (shared beta)
fit_spfa <- mlumr(
  data = network,
  spfa = TRUE,
  prior_intercept = prior_normal(0, 10),
  prior_beta = prior_normal(0, 10),
  iter_warmup = 1000,
  iter_sampling = 2000,
  chains = 4,
  seed = 123
)

# Model 2: Relaxed SPFA (treatment-specific beta)
fit_relaxed <- mlumr(
  data = network,
  spfa = FALSE,
  prior_intercept = prior_normal(0, 10),
  prior_beta = prior_normal(0, 10),
  iter_warmup = 1000,
  iter_sampling = 2000,
  chains = 4,
  seed = 123
)

Step 3: Compare Using DIC

# Calculate DIC for both models
dic_spfa <- calculate_dic(fit_spfa)
dic_relaxed <- calculate_dic(fit_relaxed)

# Compare models
comparison <- compare_models(fit_spfa, fit_relaxed)

# Decision rules:
# - Delta_DIC > 5: Meaningful difference
# - Lower DIC indicates better fit

Interpreting Model Comparison Results

When to Use SPFA Model

Use the SPFA model (simpler) when: - SPFA model has lower or similar DIC - Clinical knowledge supports SPFA - Effect modification parameters are small

Advantages: - More parsimonious (fewer parameters) - More precise estimates - Easier interpretation

When to Use Relaxed Model

Use the relaxed model when: - Relaxed model has meaningfully lower DIC (Delta > 5) - Clinical evidence suggests effect modification - Credible intervals for \Delta\beta exclude zero

Advantages: - Accounts for effect modification - Less biased if SPFA violated - Provides insight into heterogeneity

Examining Effect Modification Parameters

Extract Delta Beta

The relaxed model estimates \Delta\beta = \beta_{index} - \beta_{comparator}:

# Extract effect modification parameters
summary(fit_relaxed)

# Look for delta_beta in the output
# If 95% CrI excludes 0, there's evidence of effect modification

# Alternative: Direct extraction
params <- extract_parameters(fit_relaxed)
delta_params <- params[grep("delta_beta", params$variable), ]
print(delta_params)

Visualize Effect Modification

# Get posterior draws for delta_beta
draws <- extract_parameters(fit_relaxed, pars = c("delta_beta[1]", "delta_beta[2]"),
                            summary = FALSE)

# Create density plots
par(mfrow = c(1, 2))
hist(draws$`delta_beta[1]`, main = "Delta Beta[1]", xlab = "Difference")
abline(v = 0, col = "red", lty = 2)

hist(draws$`delta_beta[2]`, main = "Delta Beta[2]", xlab = "Difference")
abline(v = 0, col = "red", lty = 2)

Impact of Effect Modification on Treatment Effects

Bias in Population-Specific Effects

When SPFA is violated but we use the SPFA model:

Comparator Population Effects: Usually unbiased (similar to MAIC/STC)
Index Population Effects: Can be severely biased

This asymmetry occurs because: - AgD likelihood dominates estimation of \beta in SPFA model - Index population effects are then extrapolated using potentially wrong \beta

# Compare true vs estimated LOR under different scenarios
# (Using simulation study results from the ISPOR poster)

scenarios <- data.frame(
  Scenario = c("No EM", "Weak EM", "Strong EM"),
  True_LOR_Index = c(-0.43, -0.53, -0.72),
  SPFA_LOR_Index = c(-0.43, -0.47, -0.43),
  Relaxed_LOR_Index = c(-0.43, -0.53, -0.72),
  Bias_SPFA = c(0, 0.06, 0.29)
)

print(scenarios)

When to Worry About Bias

Effect modification is most problematic when:

Populations differ substantially in prognostic factors
Effect modifiers are unknown or unmeasured
Target population is the index population (where bias occurs)

Clinical Examples of Effect Modification

Some treatments work differently in elderly vs. younger patients:

# Scenario: Age modifies treatment effect
# - Treatment A: works better in younger patients
# - Treatment B: works similarly across ages

# In relaxed model:
# beta_age_A > beta_age_B (age has stronger effect on A outcomes)

Example 2: Biomarker-Based Effect Modification

Targeted therapies may show effect modification:

# Scenario: Biomarker status modifies treatment effect
# - Treatment A (targeted therapy): strong effect in biomarker+ patients
# - Treatment B (chemotherapy): similar effect regardless of biomarker

# In relaxed model:
# beta_biomarker_A > beta_biomarker_B

Example 3: Disease Severity

# Scenario: Baseline severity modifies treatment effect
# - Treatment A: more effective in severe patients
# - Treatment B: more effective in mild patients

# In relaxed model:
# beta_severity_A < beta_severity_B (opposite signs possible)

Best Practices for SPFA Testing

1. Always Fit Both Models

# Standard workflow
fit_spfa <- mlumr(data, spfa = TRUE, ...)
fit_relaxed <- mlumr(data, spfa = FALSE, ...)
comparison <- compare_models(fit_spfa, fit_relaxed)

2. Consider Clinical Plausibility

Even if DIC favors SPFA, consider: - Is there biological reason to expect effect modification? - Do prior studies suggest heterogeneous treatment effects? - Are the populations very different?

3. Report Both Models When Uncertain

# In your results:
# "Under the SPFA model, LOR was X (95% CrI: Y, Z).
#  Relaxing SPFA, LOR was X' (95% CrI: Y', Z').
#  Model comparison favored [SPFA/Relaxed] (Delta DIC = W)."

4. Sensitivity Analysis

# Test with different priors
fit_spfa_narrow <- mlumr(data, spfa = TRUE,
                        prior_beta = prior_normal(0, 2), ...)
fit_spfa_wide <- mlumr(data, spfa = TRUE,
                      prior_beta = prior_normal(0, 20), ...)

# Compare results across prior specifications

5. Consider Subgroup Analyses

If effect modification is suspected but data is limited:

# 1. Pre-specify potential effect modifiers
# 2. Report effects in relevant subgroups
# 3. Discuss clinical implications of heterogeneity

Limitations of SPFA Testing

1. Limited Power

With few integration points or small samples: - Difficult to detect moderate effect modification - DIC may favor simpler SPFA even with true violation

2. Cannot Test Unmeasured Effect Modifiers

SPFA testing only covers measured covariates. Unmeasured effect modifiers remain a source of bias.

3. Model Comparison is Not Hypothesis Testing

DIC comparison is for model selection, not formal hypothesis testing: - No p-value or formal significance level - “Better fit” doesn’t prove absence/presence of effect modification

Summary

Aspect	SPFA Model	Relaxed Model
Assumption	\beta^A = \beta^B	\beta^A \neq \beta^B
Parameters	Shared \beta	\beta_{index}, \beta_{comparator}, \Delta\beta
Bias if EM	High (index pop)	Low
Precision	Higher	Lower
When to use	No evidence of EM	Evidence of EM

Key Takeaways:

SPFA is testable via model comparison (DIC)
Effect modification causes bias primarily in index population effects
Always fit both models and compare
Consider clinical plausibility alongside statistical evidence
Report uncertainty when SPFA assumption is questionable

Session Information

sessionInfo()
#> R version 4.5.2 (2025-10-31)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.3 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
#>  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
#>  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
#> [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
#> 
#> time zone: UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] ggplot2_4.0.2 dplyr_1.2.0   mlumr_0.1.0  
#> 
#> loaded via a namespace (and not attached):
#>  [1] Matrix_1.7-4        gtable_0.3.6        jsonlite_2.0.0     
#>  [4] compiler_4.5.2      tidyselect_1.2.1    pspline_1.0-21     
#>  [7] jquerylib_0.1.4     systemfonts_1.3.1   scales_1.4.0       
#> [10] textshaping_1.0.4   yaml_2.3.12         fastmap_1.2.0      
#> [13] lattice_0.22-7      R6_2.6.1            generics_0.1.4     
#> [16] pcaPP_2.0-5         knitr_1.51          rngWELL_0.10-10    
#> [19] randtoolbox_2.0.5   tibble_3.3.1        desc_1.4.3         
#> [22] bslib_0.10.0        pillar_1.11.1       RColorBrewer_1.1-3 
#> [25] rlang_1.1.7         stabledist_0.7-2    ADGofTest_0.3      
#> [28] gsl_2.1-9           cachem_1.1.0        xfun_0.56          
#> [31] fs_1.6.6            sass_0.4.10         S7_0.2.1           
#> [34] cli_3.6.5           pkgdown_2.2.0       withr_3.0.2        
#> [37] magrittr_2.0.4      digest_0.6.39       grid_4.5.2         
#> [40] mvtnorm_1.3-3       copula_1.1-6        lifecycle_1.0.5    
#> [43] vctrs_0.7.1         evaluate_1.0.5      glue_1.8.0         
#> [46] numDeriv_2016.8-1.1 farver_2.1.2        ragg_1.5.0         
#> [49] stats4_4.5.2        rmarkdown_2.30      purrr_1.2.1        
#> [52] tools_4.5.2         pkgconfig_2.0.3     htmltools_0.5.9

mlumr package

2026-02-06

Overview

The Shared Prognostic Factor Assumption (SPFA)

What is SPFA?

When SPFA Holds

When SPFA is Violated (Effect Modification)

Simulating Effect Modification Scenarios

Scenario 1: SPFA Holds (No Effect Modification)

Scenario 2: Moderate Effect Modification

Scenario 3: Strong Effect Modification

Testing SPFA: Model Comparison Approach

Step 1: Prepare Data

Step 2: Fit Both Models

Step 3: Compare Using DIC

Interpreting Model Comparison Results

When to Use SPFA Model

When to Use Relaxed Model

Examining Effect Modification Parameters

Extract Delta Beta

Visualize Effect Modification

Impact of Effect Modification on Treatment Effects

Bias in Population-Specific Effects

When to Worry About Bias

Clinical Examples of Effect Modification

Example 2: Biomarker-Based Effect Modification

Example 3: Disease Severity

Best Practices for SPFA Testing

1. Always Fit Both Models

2. Consider Clinical Plausibility

3. Report Both Models When Uncertain

4. Sensitivity Analysis

5. Consider Subgroup Analyses

Limitations of SPFA Testing

1. Limited Power

2. Cannot Test Unmeasured Effect Modifiers

3. Model Comparison is Not Hypothesis Testing

Summary

Session Information

Effect Modification and SPFA Testing

mlumr package

2026-02-06

Overview

The Shared Prognostic Factor Assumption (SPFA)

What is SPFA?

When SPFA Holds

When SPFA is Violated (Effect Modification)

Simulating Effect Modification Scenarios

Scenario 1: SPFA Holds (No Effect Modification)

Scenario 2: Moderate Effect Modification

Scenario 3: Strong Effect Modification

Testing SPFA: Model Comparison Approach

Step 1: Prepare Data

Step 2: Fit Both Models

Step 3: Compare Using DIC

Interpreting Model Comparison Results

When to Use SPFA Model

When to Use Relaxed Model

Examining Effect Modification Parameters

Extract Delta Beta

Visualize Effect Modification

Impact of Effect Modification on Treatment Effects

Bias in Population-Specific Effects

When to Worry About Bias

Clinical Examples of Effect Modification

Example 1: Age-Related Effect Modification

Example 2: Biomarker-Based Effect Modification

Example 3: Disease Severity

Best Practices for SPFA Testing

1. Always Fit Both Models

2. Consider Clinical Plausibility

3. Report Both Models When Uncertain

4. Sensitivity Analysis

5. Consider Subgroup Analyses

Limitations of SPFA Testing

1. Limited Power

2. Cannot Test Unmeasured Effect Modifiers

3. Model Comparison is Not Hypothesis Testing

Summary

Session Information