When Do Causal ML Models Beat LM / GLM / GBM?

Summary

Who this is for: Data scientists and analysts who need to estimate what an intervention does (a call, a discount, a campaign) and want to know when causal ML methods are worth using over LM, GLM, and GBM. The basis is nine simulations with a known true effect, each with 40 repetitions.

The five most important findings

1. Data first, algorithm second. If the model contains a wrong variable or omits an important influencing factor, all methods are equally wrong, including the most flexible ones. Only domain knowledge or an instrument helps here (Cases 4, 5).
2. For the average effect, causal ML wins only when influencing factors act nonlinearly. In that case the LM is clearly biased and Double ML is almost right. The difference comes from the construction principle (first adjust, then estimate), not from a particular algorithm (Case 3).
3. A risk model is not an effect model. Calling the customers with the highest cancellation risk achieves less than a random selection, because precisely these customers are hardly retainable any more (Case 7).
4. The correct ranking is easier than the correct magnitude. A simple GLM can select the right units and still misstate the size of the effect (Case 9).
5. The causal forest pays off when the size of the effect per unit matters and its form is unknown (Cases 8, 9). In simple worlds, a well-specified GLM is better (Case 6).

How to read this report?

• Choose a method: Guide (next section), with two decision trees and a table.
• Understand the methods: Methodology, with terms, formulas, strengths and weaknesses, and the question of how to check without a known effect.
• Verify the evidence: Simulation, with nine cases, each with data, results, and a lesson.
• The numbers [n] in the text refer to the references at the end.

Guide: Which Method When?

Work through the steps from top to bottom. Stop at the first point where the answer is “not possible”: no algorithm decides steps 1–3; only step 4 chooses the method. “Unit” stands for anything that is treated (customer, person, contract, branch); the examples in the cases are set at a fictional online fitness subscription service (an app with training plans and classes).

The terms used in the guide:

• A/B test: Who is treated was determined at random (technical term: randomized). Treated and untreated units are therefore comparable.
• No random assignment (called “non-random” for short in the table): The treatment followed rules or judgments; the data simply accumulated (technical term: observational data). Treated and untreated units already differ beforehand.
• Influencing variable (confounder): A variable that influences who is treated and the outcome. Measured means it is in the data. Not measured: It is missing, and no algorithm can replace it.
• Instrument: A random nudge that shifts the treatment but affects the outcome only through it (e.g. a randomly sent invitation). Allows an estimate even though a confounder is missing.
• Average: A single effect for all units together (technical term: ATE). Answer to “Does the intervention work on average?”.
• Ranking (targeting): Deciding whom to treat, e.g. the 30% with the largest effect. Only who comes first matters; a model may over- or underestimate the effect by the same amount for everyone.
• Effect per unit (CATE): The exact size of the effect for each unit, e.g. to weigh costs and benefits per unit. Stricter than the ranking: A model can rank correctly and still estimate the size wrongly (Case 9).

Blue = question, green = recommendation, red = stop.

Step 1 – What is the decision?

Question	Target quantity	Next
Does the intervention work on average?	Average (ATE)	Step 2
Whom should I treat?	Ranking of units by effect	Step 2
How large is the effect for this unit? (e.g. for cost-benefit per unit)	Effect per unit (CATE)	Step 2
Who will cancel? (without intervention)	Risk, no effect	Pure GBM, done. Unsuitable for decisions about interventions (Case 7: top-30% uplift 0.037, worse than chance).
How strongly should I treat? (e.g. discount level)	Dose-response curve	Methodological knowledge, not simulated (see conclusion, continuous treatment).

Step 2 – How was the treatment assigned?

• At random (A/B test, lottery): Identification is secured, continue with step 4.
• By rules or by judgment of the business department (no random assignment): continue with step 3.
• Randomly assigned, but not everyone received the treatment (e.g. call assigned, customer not reached): The comparison of means by assignment measures the effect of the assignment. For the effect of the treatment itself, the assignment serves as an instrument (2SLS, Case 5 [5]). Not simulated.

Step 3 – Identification (only without random assignment)

This is where most projects fail, and no algorithm helps.

Three terms decide which variables belong in the model. They differ only in where the variable sits in the cause-and-effect picture (\(T\) = treatment, \(Y\) = outcome, \(X\) = the variable in question):

Term	Structure	What is it?	Example	In the model?
Confounder	\(T \leftarrow X \rightarrow Y\)	A common cause: \(X\) influences who is treated and the outcome. Treated and untreated units therefore already differ before the treatment.	Age: Older customers are more likely to receive a call and, for their own reasons, cancel more or less often.	Yes. Otherwise the influence of \(X\) is read as an effect of \(T\) (bias). For nonlinear influence, also in the right form (Case 2, 3).
Mediator	\(T \rightarrow X \rightarrow Y\)	An intermediate station: \(T\) acts on \(Y\) through \(X\). \(X\) only arises after the treatment and carries part of the effect.	Call \(\rightarrow\) customer sets up a training plan \(\rightarrow\) stays. “Training plan set up” is the mediator.	No, if the total effect is of interest. Otherwise the part that runs through \(X\) is missing (Case 4).
Collider	\(T \rightarrow X \leftarrow Y\)	A common consequence: \(X\) is influenced by \(T\) and by \(Y\). \(X\) arises after both.	Call and cancellation both lead to the customer ending up in the complaint log. “In the complaint log” is the collider.	No. Otherwise a spurious relationship between \(T\) and \(Y\) arises. Not simulated as a separate case; same rule as for the mediator.

Rule of thumb: What exists before the treatment and co-determines it belongs in the model. What arises after the treatment does not. Whether a variable is a confounder or a mediator is not revealed by the data – it is domain knowledge about the process.

1. Which variables go into the model? What exists before the treatment and co-determines it goes in (confounder). What arises after the treatment stays out (mediator, collider). The data do not say which is which; this is domain knowledge. In Case 4 all algorithms are equally biased when the mediator is included. Overview of good and bad control variables: [19]. Tool: Draw the process as a DAG in DAGitty; the program names the required variables [26].
2. Is a confounder missing? Cannot be proven from the data. Hints come from the placebo test (an outcome that the treatment cannot influence must show effect 0; Case 5) and the E-value (how strong a missing confounder would have to be to explain away the effect) [20]. DoWhy runs such tests automatically [27].
3. If one is missing, is there an instrument? A random nudge that acts only through the treatment. Yes: 2SLS for a constant effect, Instrumental forest for a varying effect (Case 5). Validity cannot be checked from the data. No: no causal statement. Label the number as an association, not as an effect. Exception: If before-after data or a threshold exist, see difference-in-differences and regression discontinuity under “Further methods (not simulated)”.
4. Overlap: Are there treated and untreated units in every group? Check with the propensity histogram (section “Overlap diagnostic”). Propensities near 0 or 1: restrict the population to the region with overlap. AIPW and other weighting estimators otherwise become unstable (Case 3, †). The restriction itself is not simulated here.

Step 4 – Choose a method

First find the data situation (column 2) and the question (column 3); the terms are defined at the beginning of the guide.

As a decision tree, from top to bottom (blue = question, green = recommendation, red = stop; Var. = variables). It presupposes that step 3 is passed (including overlap) and that the question concerns an effect, not pure risk. With little data (a few hundred units), row 2 of the table applies. The table additionally lists a cross-check and what to avoid:

#	Data situation	Question	Recommendation	Cross-check	Avoid	Evidence
1	A/B test	Average	Comparison of means. LM (T + X) at most for variance reduction.	–	–	Case 1, 6–8
2	A/B test, little data (a few hundred)	Ranking or effect per unit	First check whether the effect varies at all. Without a strong prior hypothesis, use the average for everyone (LM T + X, CATE-RMSE 0.141 in Case 1).	Causal forest (0.21): estimates an almost constant effect when the effect is constant	T-learner and GBM S-learner (CATE-RMSE 0.72 and 0.51, respectively): invent differences from noise	Case 1
3	A/B test, few variables, simple form plausible	Ranking or effect per unit	GLM with T x X (CATE-RMSE 0.127 vs. causal forest 0.24).	S-learner, causal forest	Pure risk model (Case 7)	Case 6, 7
4	A/B test, many variables, form unknown	Ranking	S-learner (top-30% uplift 1.64 vs. oracle 1.65).	Causal forest, R-learner (equally good selection)	Double ML and AIPW (only deliver an average), pure risk model	Case 8
5	A/B test, many variables, form unknown	Effect per unit	Causal forest (CATE-RMSE 0.28 vs. S-learner 0.45). Also provides confidence intervals.	R-learner (0.36)	GLM with T x X (0.74) or with all 2-way interactions (1.08)	Case 8
6	No randomization; confounders measured, linear	Average	LM (T + X).	Double ML	Naive comparison	Case 2
7	No randomization; confounders measured, nonlinear or unknown	Average	Double ML with a GBM as building block.	AIPW, causal forest	LM and S-learner (biased)	Case 3
8	No randomization; confounders measured	Ranking	R- or S-learner (top-30% uplift 2.70 and 2.71, respectively, vs. oracle 2.74).	Causal forest	Pure risk model	Case 9
9	No randomization; confounders measured	Effect per unit	Causal forest (ATE bias -0.020, CATE-RMSE 0.26).	R-learner (0.34)	LM/GLM (ATE bias -1.76: ranks correctly, size wrong). AIPW: no result in 21 of 40 runs.	Case 9
10	No randomization; confounder missing, instrument available	Average	2SLS (constant effect) or Instrumental forest (varying effect).	Placebo test (Case 5)	All methods without an instrument	Case 5
11	No randomization; confounder missing, no instrument	Any question about the effect	No causal method. Report the association, E-value as sensitivity analysis. Exception: With before-after data or a threshold, difference-in-differences or regression discontinuity are possible (not simulated).	–	Any method that outputs an effect number	Case 5

Step 5 – Check without knowing the truth

• A/B test: Qini/uplift curve and calibration on a hold-out dataset are valid.
• No random assignment: R-loss ranks the methods only as long as step 3 holds. In addition, placebo test and overlap histogram.
• Agreement of several methods proves nothing: In Cases 4 and 5 all methods agree and are wrong.
• State the uncertainty: An estimate should come with a confidence interval (Methodology, “Confidence intervals”). It measures only the random error: If step 3 is violated, even a narrow interval misses the target.

Details in the section “Evaluation without knowing the truth”.

Step 6 – When in doubt

Keep the simplest correct model as the baseline. Adopt the more complex method only if it is clearly better on validated data. In Cases 1, 2 and 6, complexity brings nothing.

Not tested (statements about these are methodological knowledge, not simulation): more than 100 variables with weak heterogeneity, trimming under weak overlap, weak instrument, incomplete compliance, panel/time structures, multiple treatment levels, continuous treatment, and the methods in the section “Further methods (not simulated)” (e.g. BART, BCF, TMLE).

Other decision aids

A fully automatic decision tree that chooses the right method from the data alone does not exist, and cannot exist: Whether a variable is a confounder or a mediator is not in the data (Case 4). Any automation therefore needs a cause-and-effect picture (DAG) based on domain knowledge as input. What does exist:

• DAGitty [26]: You draw the DAG; the program names the variables that must go into the model, and possible instruments. Automates step 3.1.
• DoWhy [27]: From the DAG through identification and estimation to automatic checks (placebo, random additional confounder, subsamples). Closest to an automatic workflow.
• EconML Library Flow Chart [28]: Decision tree for choosing the estimator, similar to the one in step 4. Additionally asks about compliance and whether the effect depends linearly on the features.
• Causal Roadmap [24] and Target Trial Emulation [25]: standardized workflows from epidemiology. You first describe the experiment you would have liked to run, and then check how close the data come to it.
• Automatic model selection for CATE (e.g. causaltune based on EconML): chooses the estimator via R-loss or similar measures. This works only if identification holds (see “Evaluation without knowing the truth”).

Methodology

This part describes what is estimated and with which methods, without simulation results: the core idea, the terms, the methods studied with their formulas, and the question of how to check a method on real data (where the true effect is unknown).

Core idea

LM, GLM, and GBM are normally trained to predict Y. CausalML models are built to estimate the effect of a treatment T on Y, either on average (ATE) or per unit (CATE). A good prediction model is not automatically a good effect model. Conversely, an effect model is not automatically better when the world is simple.

In real data the true effect is never known. That is why we simulate here: we define the formulas that generate the data ourselves. We therefore know the true effect of every single unit and can measure each method against it exactly.

Classification following Molnar [6]: the report takes the causal inference mindset (what causes what?) and uses supervised learning (prediction models such as GBM) only as a building block, e.g. in Double ML. Evaluation is frequentist (bias and spread over repeated samples). Other mindsets lead to other tools: Bayesian to BART and Bayesian causal forest (uncertainty per unit as a posterior), reinforcement learning to bandits, which assign the treatment continuously and learn while doing so. That is a different type of data than the one used here. The core message in this language: a good supervised-learning model is not automatically a good causal-inference model. Introductions to the field: [1, 2, 3].

Terms

Potential outcomes. Every unit has two possible outcomes: \(Y(1)\) if it is treated (\(T = 1\)) and \(Y(0)\) if not (\(T = 0\)). Only one of them is ever observed. The individual effect is \(Y(1) - Y(0)\).

Term	Definition
ATE (Average Treatment Effect)	\(\text{ATE} = E[Y(1) - Y(0)]\): effect on average across all units.
CATE (Conditional ATE)	\(\tau(x) = E[Y(1) - Y(0) \mid X = x]\): effect for units with characteristics \(x\). This is the effect “per unit”.
Treatment \(T\)	The action whose effect we are interested in (call, discount, campaign). Always 0/1 here.
Outcome \(Y\)	The outcome the action is supposed to influence. Higher = better.
Confounder	Variable that influences both T and Y (\(T \leftarrow X \rightarrow Y\)). If it is ignored, the comparison of treated vs. untreated mixes the effect of T with the effect of X. Must be in the model, and in the correct functional form.
Mediator	Variable on the path from T to Y (\(T \rightarrow M \rightarrow Y\)); it arises after T. Adjusting for it removes the part of the effect that runs through M. Must not be in the model if the total effect is of interest.
Collider	Variable that is influenced by T and Y (\(T \rightarrow C \leftarrow Y\)). Adjusting for it creates a spurious relationship between T and Y. Must not be in the model. (Not a separate case here; same rule as for the mediator: nothing that arises after T.)
Instrument \(Z\)	Variable that influences T but affects Y only through T (\(Z \rightarrow T \rightarrow Y\)) and is unrelated to unobserved disturbances. Allows effect estimation despite unobserved confounding.
Effect modifier	Variable on which the size of the effect depends (e.g. the discount only works for price-sensitive customers). Creates heterogeneity: \(\tau(x)\) is not constant.
Noise variable	Variable with no influence on T or Y. Does not harm systematically, but costs estimation precision.
Randomization	T is assigned by chance (coin flip). Then there are no confounders for T, and the simple comparison of means is unbiased.

In the diagrams (DAGs) below, an arrow \(A \rightarrow B\) means: “A enters the formula of B”. Dashed = unobserved variable.

Methods

All methods receive the training data \((Y, T, X)\) as input and return an estimated effect \(\hat\tau(x)\) for every test customer as output. Exception: the “pure” GBM, which ignores T and only predicts a risk.

Method	Model	Assumption
Naive (diff. in means)	\(\hat\tau = \bar Y_{T=1} - \bar Y_{T=0}\)	No confounding (randomization)
LM/GLM (T + X)	\(Y = \beta_0 + \tau T + \beta^\top X\)	Effect constant; Y linear in X (logit-linear for binary Y)
LM/GLM (T x X)	\(Y = \beta_0 + \tau_0 T + \beta^\top X + T \cdot \gamma^\top X\)	Effect linear in X
LM/GLM (T x all 2-way X)	like T x X, plus all products \(X_j X_k\) and their interaction with T	Effect quadratic in X
GBM / LightGBM S-learner	\(\hat f(X, T)\) = one GBM with T as a feature; \(\hat\tau(x) = \hat f(x, 1) - \hat f(x, 0)\)	No form imposed. Regularization can push the effect of T toward 0.
GBM / LightGBM T-learner	\(\hat f_1\) on treated, \(\hat f_0\) on untreated; \(\hat\tau(x) = \hat f_1(x) - \hat f_0(x)\)	No form imposed. Two separate models, each with half the data: high variance.
GBM / LightGBM pure (ignores T)	\(\hat f(X)\) without T; score = predicted risk \(-\hat f(x)\)	No effect model. This is how targeting is often done (churn model, then call the highest-risk units).
Causal forest	grf::causal_forest	No form imposed. Estimates \(\tau(x)\) directly, adjusted for propensity and baseline (honest trees). Default settings. ATE here = mean of the CATE predictions.
Double ML (xgboost)	Cross-fitting (5 folds): \(\hat\ell(X) \approx E[Y \mid X]\), \(\hat m(X) \approx E[T \mid X]\) via xgboost; \(\hat\tau = \frac{\sum_i (T_i - \hat m_i)(Y_i - \hat\ell_i)}{\sum_i (T_i - \hat m_i)^2}\)	Effect constant (or weighted mean); confounders arbitrarily nonlinear. Same xgboost as the S-learner, but T and Y are first residualized on X, so regularization does not bias the effect.
R-learner (xgboost)	Same residuals \(\tilde Y, \tilde T\) as Double ML; \(\hat\tau(x)\) = xgboost on \(\tilde Y / \tilde T\) with weights \(\tilde T^2\)	No form imposed for \(\tau(x)\); confounders arbitrarily nonlinear. Double ML with a varying effect. Needs overlap: units with propensity near 0/1 contribute little.
AIPW (causal forest)	grf::average_treatment_effect on the causal forest: \(\hat\tau = \tfrac{1}{n}\sum_i \big[\hat\mu_1 - \hat\mu_0 + \tfrac{T_i (Y_i - \hat\mu_1)}{\hat e} - \tfrac{(1-T_i)(Y_i - \hat\mu_0)}{1 - \hat e}\big]\)	Doubly robust: correct if the outcome model or the propensity model is correct. Average effect only, no effect per unit.
2SLS	Stage 1: \(\hat T\) = prediction from \(T \sim Z + X\) Stage 2: \(Y = \beta_0 + \tau \hat T + \beta^\top X\)	Valid instrument Z; effect constant. Classical IV estimator.
Instrumental forest	grf::instrumental_forest	Valid instrument Z; effect may depend arbitrarily on X.

xgboost and LightGBM: depth 3, learning rate 0.1, number of rounds via 5-fold CV with early stopping, otherwise library defaults (LightGBM: num_leaves = 8, min_data_in_leaf = 20).

Methods in detail

The methods from the S-learner onward, each with the idea, the formula, and the typical weakness. Common notation: \(\mu_t(x) = E[Y \mid X = x, T = t]\) is the expected outcome, \(e(x) = P(T = 1 \mid X = x)\) the propensity (probability of being treated), and \(\hat{\ }\) denotes estimated.

GBM S-learner (“Single”)

A single model for all units, with \(T\) as one feature among many. The effect is the difference in the prediction when \(T\) is set from 0 to 1 and everything else is held fixed.

\[ \hat\tau(x) = \hat f(x, 1) - \hat f(x, 0) \]

Weakness: If \(T\) has only a small influence, the GBM barely uses it (regularization), and the effect is pushed toward 0. Under strong confounding a residual bias remains (Case 3). Source: [9].

GBM T-learner (“Two”)

Two separate models: one only on the treated, one only on the untreated. The effect is the difference between the two predictions.

\[ \hat\tau(x) = \hat\mu_1(x) - \hat\mu_0(x) \]

Weakness: Each model sees only part of the data, and the two errors add up. With small n or groups of unequal size this is very noisy (Case 1). The X-learner (see “Further methods”) addresses exactly this. Source: [9].

Pure GBM (ignores \(T\))

An ordinary prediction model for \(Y\) without \(T\), e.g. a churn model. The score is the predicted risk.

\[ \text{Score}(x) = -\hat f(x) \quad\text{with}\quad \hat f(x) \approx E[Y \mid X = x] \]

Weakness: It answers “who is at risk?”, not “for whom does the treatment work?”. Units with high risk may be unpersuadable (Case 7). On the difference between risk and uplift models: [21].

Double ML

Two nuisance models predict from \(X\) what is to be expected without the treatment: the outcome \(\ell(x) = E[Y \mid X = x]\) and the assignment \(m(x) = E[T \mid X = x]\). What remains is only what \(X\) does not explain (the residuals). The effect is the slope between the two residuals.

\[ \tilde Y_i = Y_i - \hat\ell(X_i), \qquad \tilde T_i = T_i - \hat m(X_i), \qquad \hat\tau = \frac{\sum_i \tilde T_i \tilde Y_i}{\sum_i \tilde T_i^2} \]

Cross-fitting: The nuisance models predict each unit using models that have not seen it (5 folds). Otherwise overfitting would distort the residuals. Why it works: Errors in \(\hat\ell\) and \(\hat m\) enter only as a product, so the form of the confounders can be learned freely without biasing the effect. Weakness: a single effect for everyone (the average), no effect per unit. Source: [7]; textbook: [2].

R-learner (Double ML with per-unit effect)

The same residuals \(\tilde Y, \tilde T\) as in Double ML, but the effect is no longer a single number but a function \(\tau(x)\) learned by an arbitrary model. It is chosen so that the residuals of \(Y\) are explained as well as possible by \(\tau(X_i)\,\tilde T_i\):

\[ \hat\tau(\cdot) = \arg\min_{\tau} \sum_i \big(\tilde Y_i - \tau(X_i)\,\tilde T_i\big)^2 \]

Implemented as a weighted regression: an xgboost predicts \(\tilde Y_i / \tilde T_i\) from \(X\), with weight \(\tilde T_i^2\). Weakness: Units with \(\tilde T_i \approx 0\) (propensity near 0 or 1) contribute almost nothing; with a constant effect the model learns heterogeneity that does not exist (more variance than Double ML). Source: [8].

Causal forest

A forest of trees that are split so that the effect differs as much as possible between the leaves (not the outcome). For a new unit \(x\), the forest counts which training units \(x\) often lands in the same leaf with (weight \(\alpha_i(x)\)) and estimates the effect only from these similar units, again using the residuals as in Double ML:

\[ \hat\tau(x) = \frac{\sum_i \alpha_i(x)\, \tilde T_i \tilde Y_i}{\sum_i \alpha_i(x)\, \tilde T_i^2} \]

Honest: One half of the data determines the splits, the other half estimates the effect. This prevents the forest from learning random fluctuations as an effect. Weakness: needs a lot of data to find fine differences; with a constant effect or a linear world there is no advantage. Sources: [10, 11]; honest trees: [12].

AIPW (doubly robust)

Combines two models. The outcome model \(\hat\mu_t\) predicts both outcomes for every unit. The weighted correction offsets deviations at the units actually observed, with rare cases (small propensity) counting more.

\[ \hat\tau_{\text{AIPW}} = \frac{1}{n} \sum_i \left[ \hat\mu_1(X_i) - \hat\mu_0(X_i) + \frac{T_i\,(Y_i - \hat\mu_1(X_i))}{\hat e(X_i)} - \frac{(1 - T_i)\,(Y_i - \hat\mu_0(X_i))}{1 - \hat e(X_i)} \right] \]

Doubly robust: The result is correct if either the outcome model or the propensity model is correct. Weakness: If \(\hat e\) is near 0 or 1, the weights \(1/\hat e\) explode (Case 3, individual runs fail). Average effect only. Source: [13]; TMLE as a more stable variant: [17].

2SLS (instrument \(Z\))

If a confounder is unobserved, an instrument \(Z\) helps: it shifts \(T\) but has no other path to \(Y\). Stage 1 isolates the part of \(T\) that \(Z\) explains. Stage 2 uses only this part, which is free of the unobserved confounder.

\[ \text{Stage 1: } \hat T = \pi_0 + \pi_1 Z + \gamma^\top X \qquad \text{Stage 2: } Y = \beta_0 + \tau\,\hat T + \beta^\top X \]

Without \(X\) and with a binary \(Z\), this is the Wald estimator: the effect of \(Z\) on \(Y\), divided by the effect of \(Z\) on \(T\).

\[ \hat\tau = \frac{E[Y \mid Z = 1] - E[Y \mid Z = 0]}{E[T \mid Z = 1] - E[T \mid Z = 0]} \]

Weakness: Whether \(Z\) is valid cannot be tested from the data. If \(Z\) is weak (barely shifts \(T\)), the denominator is near 0 and the estimate is very uncertain. Source: [5].

Instrumental forest

The same principle as 2SLS, but local as in the causal forest: for each unit \(x\), a separate IV estimator is computed from the similar units (weights \(\alpha_i(x)\) of the forest). The effect may therefore depend on \(X\).

\[ \hat\tau(x) = \frac{\sum_i \alpha_i(x)\,(Y_i - \hat\ell_i)(Z_i - \hat z_i)}{\sum_i \alpha_i(x)\,(T_i - \hat m_i)(Z_i - \hat z_i)} \]

Weakness: With a constant effect it is merely noisier than 2SLS (Case 5). Source: [11].

Strengths and weaknesses by method

Summary of the simulation results per method. Which method fits which situation is given in the guide; here we give the reasons.

Method	Good for	Bad when	Evidence in report
Naive (difference in means)	A/B test, average only (Case 1, 6–8). Unbiased, needs no model assumptions.	No random assignment: influencing variables are read as an effect (Case 2: bias -2.82).	Case 1, 2, 6–8
LM/GLM (T + X)	Constant effect, linear relationships, all influencing variables measured (Case 1, 2). Hard to beat at small n (Case 1: CATE error 0.141). Transparent, auditable.	Influencing variables act nonlinearly (Case 3: bias -1.75) or the effect varies (Case 8).	Case 1, 2, 3, 8
LM/GLM with interactions (T x X)	Effect varies in a simple, known form: then better than the causal forest (Case 6: 0.127 vs. 0.24). Often ranks correctly even when the magnitude is biased (Case 9: top 30% 2.73 at oracle 2.74).	Many variables with unknown interactions (Case 8: 0.74; with all interactions 1.08). Nonlinear influencing variables bias the magnitude (Case 9: bias -1.76).	Case 6, 8, 9
GBM S-learner	Ranking in A/B tests and with measured influencing variables (Case 8: 1.64 at oracle 1.65; Case 9: 2.71 at 2.74). Little effort.	Magnitude of the effect under strong confounding: regularization biases it (bias Case 3: -0.197, Case 9: -0.117).	Case 3, 7, 8, 9
GBM T-learner	Lots of data in both groups, strongly different structure of treated and untreated.	Small n: two models on half the data each, high variance (Case 1: CATE error 0.72).	Case 1
Pure GBM (ignores T)	Pure risk prediction (who will cancel?). Not for decisions about interventions.	Selection: picks units with high risk instead of high effect (Case 7: top 30% 0.037, worse than chance).	Case 7
Double ML	Average with measured but nonlinearly acting influencing variables (Case 3: bias -0.013). Uses any GBM as a building block, fast, easy to explain.	Only an average: CATE error is high by construction when the effect varies (Case 9: 1.50).	Case 3, 9
R-learner	Effect per unit with unknown form, even without random assignment (CATE error Case 8: 0.36, Case 9: 0.34). Ranking on the level of the causal forest.	Constant effect: estimates differences that do not exist (Case 1: 0.27 vs. Double ML 0.150). Less accurate than the causal forest (Case 8, 9).	Case 1, 8, 9
Causal forest	Effect per unit with unknown form, many variables, or nonlinear influencing variables (CATE error Case 8: 0.28, Case 9: 0.26). Provides confidence intervals.	Simple form: a suitable GLM is better (Case 6). No advantage over LM at small n (Case 1). Unstable under weak overlap. Default settings, not tuned.	Case 1, 6, 8, 9
AIPW	Average with protection against errors in one of the two submodels (Case 3: bias -0.059), as long as overlap holds.	Weak overlap (propensity near 0/1): weights explode, runs fail (Case 3: 2, Case 9: 21 of 40). No effect per unit.	Case 3, 9
2SLS / Instrumental forest	Missing influencing variable, but a valid instrument (Case 5). 2SLS for a constant effect, instrumental forest for a varying one.	Weak or invalid instrument. Validity cannot be tested from the data (Case 5 assumes it).	Case 5

Evaluation methods

In real data the true effect is unknown. What can be checked depends on whether the treatment was assigned at random.

With randomized data (e.g., a randomly treated share of a campaign), the difference in means is unbiased and serves as the benchmark. On a hold-out set, any effect model can then be checked from the outside:

• Calibration: Group units into deciles by predicted effect and compute the actual difference in means within each decile. If prediction and measurement agree, the model is usable (grf: test_calibration).
• Uplift or Qini curve: Do the top X % according to the model really have the higher effect? This is the practical version of the top-30% measure in this report [21].
• Comparison with chance: The benefit of selecting by model is compared with random selection and with “treat everyone”.

A permanently small random share (e.g., 5–10 %) costs something, but it is the only clean answer.

With observational data there is no such benchmark. The following tools narrow down errors but prove nothing, and none of them is recommended across the board: each works only under its own assumptions or is too noisy to decide on its own. It is best to combine several.

• Test on your own data (“plasmode”): keep the real features \(X\) and the real assignment \(T\), regenerate \(Y\) with an effect of your own choosing, and measure the methods against it. This is what this report does on a small scale, only with the structure of your own data. The result holds only for the chosen effect form, however.

• Loss function on hold-out data: The R-loss [8, 22] (or the analogous DR-loss [14]) compares effect models without knowing the true effect. It is noisy and depends on the nuisance models \(\hat\ell, \hat m\): usable for discarding poor models, but for picking the best one only when the differences are clear (see Simulation: “Evaluation without knowing the truth”).

  Step 1: nuisance models. Two prediction models are learned on the training data, the outcome model \(\hat\ell(X) \approx E[Y \mid X]\) and the treatment model \(\hat m(X) \approx E[T \mid X]\) (propensity).

  Step 2: residuals on the hold-out set. From \(Y\) and \(T\), subtract what can already be predicted from \(X\):

  \[\tilde Y_i = Y_i - \hat\ell(X_i), \qquad \tilde T_i = T_i - \hat m(X_i)\]

  What remains is only the part of \(Y\) not explained by \(X\) and the part of \(T\) that is “random”. If the treatment acts with \(\tau(X)\), then approximately \(\tilde Y_i \approx \tau(X_i)\,\tilde T_i + \text{noise}\).

  Step 3: loss. An effect model \(\hat\tau\) is then judged by how well it fits this relationship:

  \[\text{R-loss}(\hat\tau) = \frac{1}{n}\sum_{i=1}^{n} \Big(\tilde Y_i - \hat\tau(X_i)\,\tilde T_i\Big)^2\]

  Smaller is better. For comparison, \(\hat\tau \equiv 0\) (no effect) gives \(\frac{1}{n}\sum_i \tilde Y_i^2\). A method is useful only if it falls below this value. The loss needs neither the true effect nor randomization, but it presupposes that \(X\) contains all confounders (otherwise Cases 4 and 5).

• Agreement of several methods: Similar results from LM, Double ML and causal forest are reassuring; clear deviations point to a wrong functional form or weak overlap. It is not proof (Cases 4, 5: all equally wrong).

• Overlap and balance: Histogram of the estimated propensity (values near 0 or 1 are a warning sign, Case 3) and comparison of the features of treated and untreated units after weighting. This checks only measured variables.

• Placebo test: The treatment is applied to an outcome it cannot affect (or to a period before it). An “effect” indicates remaining confounding. It requires a credible placebo outcome, which is rarely available.

• Sensitivity analysis (e.g., E-value, Rosenbaum): How strong would an unobserved confounder have to be to explain the effect away? This yields an order of magnitude, not a decision [20].

• Comparison with an experiment: If a small randomized test exists somewhere, the observational estimator is measured against it. This holds only if population and treatment are similar enough.

Not testable is whether all confounders are measured and no mediators are in the model. No quality measure detects this; predictive accuracy can even look good (see “Predictive accuracy is not effect accuracy”). This is decided by domain knowledge of the process, not by the method.

Confidence intervals

A confidence interval shows how much an estimate would vary from sample to sample. A 95 % interval is usually the estimate ± 1.96 standard errors (SE). It measures only random error, not systematic error: if a wrong variable is in the model or a driver is missing, a narrow interval will surely surround the wrong value (see Simulation, “Do the confidence intervals hold?”).

Method	How the standard error arises	In R	Simulated here
Difference in means	\(\sqrt{s_1^2/n_1 + s_0^2/n_0}\) from the variances of the two groups.	t.test(Y ~ T)	yes
LM/GLM (T + X)	Robust standard error (sandwich, HC1) of the coefficient of T. The default SE from summary() assumes equal variance for all.	lmtest::coeftest(m, vcov = sandwich::vcovHC)	yes (continuous Y only)
Double ML	Formula from the residuals: \(\sqrt{\sum_i \tilde T_i^2 \hat\varepsilon_i^2} \big/ \sum_i \tilde T_i^2\) with \(\hat\varepsilon_i = \tilde Y_i - \hat\tau \tilde T_i\). Valid despite ML nuisance models, because cross-fitting and orthogonalization make their errors harmless [7].	DoubleML package	yes
AIPW, Causal forest (ATE)	Standard deviation of the per-unit AIPW scores, divided by \(\sqrt{n}\) [11].	grf::average_treatment_effect(cf)	yes (AIPW)
2SLS	IV standard error; the residuals are computed with the actual T, not with the prediction from stage 1 [5].	ivreg::ivreg	yes
Causal forest (effect per unit)	Variance per unit from the forest (valid only with honest trees) [10]. Pointwise: the interval holds for one unit, not for all simultaneously.	predict(cf, X, estimate.variance = TRUE)	no
GLM with T x X (effect per unit)	Delta method from the covariance matrix of the coefficients.	marginaleffects::comparisons()	no
S-, T-, R-learner with GBM	No formula. Bootstrap: draw units with replacement and re-estimate the whole pipeline. Expensive, and captures only the variance, not the regularization bias.	own loop or boot	no
BART, BCF	Credible interval directly from the posterior [15].	bartCause, bcf	no

Often more useful for decisions than an interval per unit: the interval for a group, e.g., the 30 % with the largest estimated effect. To obtain it, estimate the model on training data, select the group on hold-out data, and estimate the effect there: in an A/B test as the difference in means within the group, without random assignment as the mean of the group’s AIPW scores (SE = standard deviation / \(\sqrt{n}\)). Because selection and estimation happen on separate data, the interval is honest.

When the standard errors are too small: with clustered data (several observations per customer or per branch: cluster-robust SE or bootstrap whole groups), with time series and panels (block bootstrap), with weak overlap (AIPW SE unstable) and with a weak instrument (2SLS SE unreliable).

Further methods (not simulated)

The simulated methods cover the most common families, but not all. The table maps each further method to a place in the guide: the column “Alternative to” names the simulated method it can replace, and the row of the table in Step 4. Most change only the estimation. Two intervene earlier: difference-in-differences and regression discontinuity make estimation possible even when a driver is missing, and causal discovery helps with the choice of variables. Everything in the table is literature knowledge, not tested here. With a wrong choice of variables (Step 3), the estimators fail just like the simulated ones.

Method	Alternative to	Idea	When to prefer?	R package	Source
Causal discovery (e.g., PC algorithm)	No estimation method. Helps with Step 3.1 (which variables go into the model).	Estimates the DAG from the data.	Only exploratory, to check domain knowledge. It does not replace it: whether a variable is a confounder or a mediator (Case 4) often cannot be determined from observational data.	pcalg	[4]
Difference-in-differences, synthetic control, regression discontinuity	The stop “No causal statement” (row 11), if before/after data or a threshold exist.	Identification via time courses (before/after, control group over time) or via a threshold (e.g., student discount up to age 25), instead of via measured drivers.	When a driver is missing and no instrument is available, but time or threshold data are. Own assumptions (e.g., parallel trends).	did, Synth, rdrobust, CausalImpact	[5]
Matching, weighting (IPW), entropy balancing	LM (T + X) and Double ML for the average effect without randomization (rows 6, 7).	First make treated and untreated units comparable (form pairs or weight), then simply compare.	When traceability matters: balance tables show directly whether the groups are comparable. Often more convincing for a specialist audience.	MatchIt, WeightIt	[1]
TMLE with SuperLearner	Double ML and AIPW for the average effect without randomization (row 7).	Doubly robust like AIPW, with an additional correction step (“targeting”); ensemble of many models as a building block.	When AIPW fails because of extreme propensities (Cases 3, 9 †).	tmle, SuperLearner	[17]
X-learner	S-learner and GLM with T x X in A/B tests (rows 3, 4).	T-learner whose errors are corrected with the data of the respective other group and weighted by the propensity.	Very unequally sized groups (e.g., 5 % treated).	causalToolbox (GitHub)	[9]
DR-learner	R-learner (rows 5, 8, 9).	Per-unit AIPW scores as the target, with any ML model fitted on them.	More robust than the R-learner if one of the nuisance models is wrong. Also unstable with weak overlap.	a few lines based on xgboost	[14]
BART	Causal forest and S-learner (rows 4, 5, 8, 9).	Sum of many small trees with a prior (Bayesian). Provides an uncertainty for each per-unit effect.	Small to medium n, uncertainty per unit required. Performed very well in a large methods competition [23].	bartCause, dbarts	[15]
Bayesian causal forest (BCF)	Causal forest without randomization (row 9).	BART with separate models for baseline and effect, propensity as an additional feature. The effect part is regularized more strongly toward “constant”.	Strong confounding and varying effect (as in Case 9).	bcf	[16]
Policy tree	None, but a step afterwards: turns an estimate (e.g., causal forest, rows 4, 8) into an explainable rule.	Learns a rule “whom to treat” directly as a shallow tree.	When the rule must be explainable (e.g., for a business department or a regulator).	policytree	[18]
Neural networks (TARNet, Dragonnet)	S-, T- and R-learner when the features are images or text (rows 4, 5).	Shared representation of the features, two outputs for treated / untreated.	With tabular data as here, usually no advantage over trees.	Python (e.g., catenets)	[2]

Simulation

Here the methods are measured on nine simulated worlds in which the true effect is known. First the setup, then the cases in four blocks, then the overall comparison and the conclusion.

Setup

How are methods scored?

For each case, a training dataset with new random numbers is generated 40 times. Each method is trained on it and compared with the known true effect \(\tau_i\) on a separate test dataset (n = 5000).

Metric	Formula	Meaning
ATE bias	\(\overline{\hat\tau} - \overline{\tau}\), averaged over simulations	Systematic error in the average effect. 0 = perfect.
ATE SD	Standard deviation of the estimated ATE over simulations	How strongly the estimate varies from dataset to dataset. Small = stable.
CATE RMSE	\(\sqrt{\tfrac{1}{n}\sum_i (\hat\tau_i - \tau_i)^2}\)	Typical error of the effect per customer. 0 = perfect. With a constant effect, almost the same as |ATE error|.
Y prediction RMSE	\(\sqrt{\tfrac{1}{n}\sum_i (Y_i - \hat Y_i)^2}\), \(\hat Y_i\) = prediction at the actual \(T_i\)	Classical predictive accuracy (for binary Y: square root of the Brier score). Not available for forests and IV estimators.
Top-30% uplift	\(\tfrac{1}{|S|}\sum_{i \in S} \tau_i\), \(S\) = 30 % with the highest model score	True mean effect among the customers the model would treat. Higher = better; compare with perfect selection (oracle). Only when the effect varies.

What this simulation can and cannot show. The nine worlds are deliberately simple and chosen by the author, each to make one mechanism visible. The methods run with little tuning (xgboost and LightGBM with depth 3 and number of rounds chosen by cross-validation, grf with default settings). The ranking of the methods therefore holds for these worlds, not in general; what transfers are the mechanisms, that is, why a method fails. Every number is a mean over 40 repetitions, and the result tables report the Monte Carlo standard error. Differences below about twice that are to be read as equally good. All methods run on the same datasets, so differences between them are more precise than the individual means.

Overview of the cases

The cases are arranged in four blocks. A shows the simple world. B asks whether the effect can be estimated at all with the available variables (identification) – there, usually no algorithm helps, but knowledge about the variables does. C asks how flexible the model must be when the effect varies from customer to customer (estimation). D combines both: observational data with confounding and a varying effect.

Block	Case	n	Variables	Setting	True effect	True ATE
A Simple world	1 Small n	200	3	Randomized, small sample	tau = 1 (constant)	1.00
B Which variables?	2 Confounding linear	1000	2	Observational, confounder measured, acts linearly	tau = 2 (constant)	2.00
B Which variables?	3 Confounding nonlinear	1000	2	Observational, confounder measured, acts U-shaped	tau = 1 (constant)	1.00
B Which variables?	4 Mediator	1000	2	Randomized, one variable is a mediator	tau = 1.5 (total effect, constant)	1.50
B Which variables?	5 Unobs. confounding	2000	1	Observational, confounder U not measured, randomized instrument Z	tau = 1 (constant)	1.00
C How flexible?	6 Linear heterogeneity	2000	3	Randomized	tau = 1 + 1.5 X1 (linear)	1.00
C How flexible?	7 Risk ≠ uplift	3000	2	Randomized, binary outcome (subscription kept / cancelled)	tau(X1) = bell shape around medium risk (max. approx. 0.27)	0.10
C How flexible?	8 Many variables	2000	20	Randomized, 20 variables (18 of them noise)	tau = 2 if X1 > 0 AND X2 > 0, else 0	0.50
D Both together	9 Observational + heterogeneity	2000	2	Observational, confounder measured (U-shaped), effect varies with X2	tau = 1 + 1.5 X2 (linear)	1.00

Every case follows the same structure: story → DAG → formulas that generate the data → model input → true effect → why simple methods fail → correct estimation → results → lesson.

A – Simple world

Before turning to pitfalls: what happens when the world is simple? Then the simplest correct model should win, and any additional flexibility only costs.

Case 1 – Small n, constant effect

Story. A pilot trial with only 200 customers, randomized. The world is linear and the effect is the same for everyone.

Data generation (n = 200, randomized):

\[ \begin{aligned} X_1, X_2, X_3 &\sim N(0, 1), \qquad T \sim \text{Bernoulli}(0.5) \\ Y &= 1 + X_1 + 0.5\, X_2 + 1 \cdot T + \varepsilon, \quad \varepsilon \sim N(0, 1.5^2) \end{aligned} \]

Model input: \(Y, T, X_1, X_2, X_3\).

True effect: \(\tau(x) = 1\) for everyone, ATE = 1.

Correct estimation: lm(Y ~ T + X1 + X2 + X3). The covariates are not needed against bias (randomized), but they reduce the noise and thus the variance.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Monte Carlo standard errors of the means at most: ATE bias 0.040, CATE RMSE 0.027. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE
Naive (diff. in means)	0.01	0.25	0.18	1.88
LM/GLM (T + X)	-0.01	0.19	0.14	1.52
LM/GLM (T x X)	-0.02	0.20	0.38	1.53
GBM S-learner	-0.39	0.21	0.51	1.61
GBM T-learner	-0.02	0.20	0.72	1.64
LightGBM S-learner	-0.30	0.21	0.43	1.60
LightGBM T-learner	-0.01	0.20	0.59	1.62
Causal forest	-0.02	0.19	0.21	–
Double ML (xgboost)	0.00	0.20	0.15	–
R-learner (xgboost)	0.00	0.20	0.27	–
AIPW (causal forest)	-0.01	0.19	0.14	–
GBM pure (ignores T)	–	–	–	1.68
LightGBM pure (ignores T)	–	–	–	1.67

Lesson: With a simple effect and little data, the simplest model wins (LM, CATE-RMSE 0.141). Flexibility costs variance (T x X: 0.38, T-learner: 0.72) or bias: the S-learner pushes the effect toward 0 (ATE bias -0.39), because with little data T is rarely chosen for splits. The causal forest is robust (0.21) but adds nothing. The pure ATE estimators Double ML (0.150) and AIPW (0.136) are on par with the LM; the differences are smaller than the spread across simulations.

B – Which variables belong in the model?

In this block the effect is constant: there is no heterogeneity, only the question of whether the average effect is estimated correctly. The order is: confounders must be included (Case 2), and in the right form (Case 3). Variables that arise after T must not be included (Case 4). What is not measured cannot be replaced by any model (Case 5).

Case 2 – Confounding, linear

Story. Customer service preferentially calls dissatisfied customers (T = call). But dissatisfied customers also have a lower satisfaction score Y three months later even without a call. Prior satisfaction (X1) is therefore a confounder: it determines who is called, and it determines Y.

Data generation (n = 1000, observational data):

\[ \begin{aligned} X_1, X_2 &\sim N(0, 1) \\ P(T = 1 \mid X_1) &= \text{logit}^{-1}(-1.2\, X_1) \quad \text{(the less satisfied, the more likely a call)} \\ Y &= 10 + 3\, X_1 + 2\, T + \varepsilon, \quad \varepsilon \sim N(0, 1) \end{aligned} \]

Model input: \(Y, T, X_1, X_2\) (all observed).

True effect: \(\tau(x) = 2\) for every customer, so ATE = 2.

Why the naive comparison is wrong: \(E[Y \mid T{=}1] - E[Y \mid T{=}0] = 2 + 3 \cdot (E[X_1 \mid T{=}1] - E[X_1 \mid T{=}0])\). Called customers are on average 0.93 standard deviations less satisfied, so the naive comparison is ≈ 2 + 3 · (-0.93) = -0.81. The call appears to harm, although it helps.

Correct estimation: lm(Y ~ T + X1) is exactly the true model. The coefficient of T is the effect.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Monte Carlo standard errors of the means at most: ATE bias 0.030, CATE RMSE 0.030. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE
Naive (diff. in means)	-2.82	0.19	2.82	2.84
LM/GLM (T + X)	0.01	0.06	0.04	1.00
LM/GLM (T x X)	0.01	0.06	0.12	1.01
GBM S-learner	-0.20	0.06	0.35	1.04
GBM T-learner	-0.14	0.07	0.60	1.06
LightGBM S-learner	-0.21	0.07	0.43	1.06
LightGBM T-learner	-0.22	0.07	0.86	1.09
Causal forest	-0.01	0.06	0.15	–
Double ML (xgboost)	-0.02	0.06	0.04	–
R-learner (xgboost)	-0.02	0.06	0.10	–
AIPW (causal forest)	-0.01	0.07	0.05	–
GBM pure (ignores T)	–	–	–	1.37
LightGBM pure (ignores T)	–	–	–	1.38

Lesson: The problem is confounding, not missing ML. If the confounder acts linearly, the LM that includes it is exactly right and the best model (ATE bias 0.009). The causal forest reaches the target (-0.007) but is less accurate per customer. The GBM learners are biased (S-learner -0.196), because regularization gives T too little weight as a feature.

Case 3 – Confounding, nonlinear

Story. As in Case 2, but the confounder acts nonlinearly. A motivational call (T) goes preferentially to very young and very old customers (X1 = age, standardized). Precisely these groups also train less without a call (Y = activity score, e.g., training minutes per week). The relationship is U-shaped: it depends on \(X_1^2\), not on \(X_1\).

Data generation (n = 1000, observational data):

\[ \begin{aligned} X_1, X_2 &\sim N(0, 1) \\ P(T = 1 \mid X_1) &= \text{logit}^{-1}(-1 + 1.2\, X_1^2) \quad \text{(young and old customers are more likely to be called)} \\ Y &= 10 - 1.5\, X_1^2 + X_2 + 1 \cdot T + \varepsilon, \quad \varepsilon \sim N(0, 1) \end{aligned} \]

Model input: \(Y, T, X_1, X_2\) (all observed; nothing is missing, only the form is unknown).

True effect: \(\tau(x) = 1\) for everyone, ATE = 1.

Why the LM fails even though the confounder is included: lm(Y ~ T + X1 + X2) can only fit a straight line in \(X_1\). The U shape is not in \(X_1\) but in \(X_1^2\), and called customers have on average a 1.18 larger \(X_1^2\). The LM attributes this difference to the call: coefficient of T ≈ 1 − 1.5 · 1.18 = -0.78 (n = 200,000). The call appears to harm.

Correct estimation: lm(Y ~ T + I(X1^2) + X2) – if you know the form. Without that knowledge: a method that learns the influence of the confounders flexibly and estimates the effect separately from it (causal forest, Double ML).

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). † No result in 2 runs (extreme propensities); mean over the remaining runs. Monte Carlo standard errors of the means at most: ATE bias 0.024, CATE RMSE 0.024. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE
Naive (diff. in means)	-1.75	0.15	1.75	2.40
LM/GLM (T + X)	-1.75	0.15	1.75	2.18
LM/GLM (T x X)	-1.75	0.15	1.77	2.19
GBM S-learner	-0.20	0.08	0.30	1.11
GBM T-learner	-0.38	0.10	1.34	1.11
LightGBM S-learner	-0.19	0.08	0.29	1.21
LightGBM T-learner	-0.49	0.10	1.51	1.24
Causal forest	-0.02	0.08	0.18	–
Double ML (xgboost)	-0.01	0.08	0.07	–
R-learner (xgboost)	-0.02	0.08	0.13	–
AIPW (causal forest) †	-0.06	0.07	0.08	–
GBM pure (ignores T)	–	–	–	1.19
LightGBM pure (ignores T)	–	–	–	1.29

Lesson: This is the case in which CausalML clearly wins on the average effect. The LM has the confounder in the model and is nevertheless strongly biased (ATE bias -1.75), because the form is wrong. The S-learner learns the U shape but remains biased (-0.197), because regularization mixes the effect of T with the influence of X1. Double ML with the same xgboost is practically unbiased (-0.013): it first adjusts T and Y for X and estimates the effect only afterwards. The causal forest is likewise almost unbiased (-0.019). AIPW suffers from the weak overlap (-0.059, no result in 2 of 40 runs): with propensities close to 1, the weights \(1/(1-\hat e)\) become huge. What matters is thus not the learning algorithm but the construction of the estimator. If you know the form, the LM with the right term wins again.

Case 4 – Bad control (mediator)

Story. A campaign (T) is sent out at random. It affects additional revenue Y (booked premium courses) in two ways: directly, and indirectly because customers who received the campaign open the app more often afterwards (M = number of app visits in the following three months). M is in the data and predicts Y extremely well. But M is a mediator: it arises only after T and is part of the effect.

Data generation (n = 1000, randomized):

\[ \begin{aligned} X_1 &\sim N(0, 1), \qquad T \sim \text{Bernoulli}(0.5) \\ M &= 0.5\, X_1 + 1 \cdot T + \eta, \quad \eta \sim N(0, 1) \\ Y &= X_1 + 0.5\, T + 1 \cdot M + \varepsilon, \quad \varepsilon \sim N(0, 1) \end{aligned} \]

Model input: \(Y, T, X_1, M\). For comparison, LM and causal forest are also run with \(X_1\) only (“w/o mediator”).

True effect: The question “Is the campaign worthwhile?” refers to the total effect: direct 0.5 plus via M 1 · 1 = 1.5 for all customers.

Why adjusting for M shrinks the effect: For fixed M, the model compares customers who received the campaign with those who did not at the same number of app visits. This cuts off the path through M, and only the direct effect 0.5 remains (simulated with n = 200,000: 0.49). Bias = −1.0: two thirds of the effect disappear.

Correct estimation: lm(Y ~ T + X1) or simply the comparison of means – without M. No algorithm can detect this from the data; it follows only from the knowledge that M arises after T.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Monte Carlo standard errors of the means at most: ATE bias 0.024, CATE RMSE 0.014. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE
Naive (diff. in means)	0.02	0.15	0.12	2.06
LM/GLM (T + X)	-0.99	0.08	0.99	1.00
LM/GLM (T x X)	-0.99	0.08	1.00	1.00
GBM S-learner	-1.09	0.09	1.10	1.04
GBM T-learner	-0.93	0.09	1.04	1.05
LightGBM S-learner	-1.08	0.09	1.09	1.04
LightGBM T-learner	-0.91	0.09	1.04	1.06
Causal forest	-0.99	0.08	1.00	–
Double ML (xgboost)	-0.99	0.09	0.99	–
R-learner (xgboost)	-0.99	0.09	0.99	–
AIPW (causal forest)	-0.99	0.08	0.99	–
GBM pure (ignores T)	–	–	–	1.06
LightGBM pure (ignores T)	–	–	–	1.06
LM/GLM (T + X), w/o mediator	0.00	0.10	0.08	1.42
Causal forest, w/o mediator	-0.01	0.10	0.31	–

Lesson: No algorithm detects a bad control variable. All models with M estimate only the direct effect (LM: ATE bias -0.99, causal forest: -0.99). At the same time they predict Y much better (Y prediction RMSE of the LM 1.00 with M, 1.42 without M): best prediction, wrong effect. In practice this is the most common error of this kind: variables that arise after the intervention do not belong in the effect model.

Case 5 – Unobserved confounding

Story. Customers can voluntarily take part in an 8-week training program (T). More motivated customers (U) are more likely to take part and also train more without the program (Y = activity score). U is not recorded anywhere. As part of an experiment, however, an invitation Z was sent at random, which strongly increases participation. Z is an instrument.

Data generation (n = 2000, observational data with randomized invitation):

\[ \begin{aligned} U, X_1 &\sim N(0, 1), \qquad Z \sim \text{Bernoulli}(0.5) \\ P(T = 1 \mid Z, U) &= \text{logit}^{-1}(-1.5 + 3\, Z + 1.5\, U) \\ Y &= X_1 + 2\, U + 1 \cdot T + \varepsilon, \quad \varepsilon \sim N(0, 1) \end{aligned} \]

Model input: \(Y, T, X_1\) for all standard methods; \(Y, T, X_1, Z\) for 2SLS and the instrumental forest. U is never in the input.

True effect: \(\tau(x) = 1\) for everyone, ATE = 1.

Why all standard methods fail: Participants are more motivated (higher U). Without U, no model can separate which part of the difference comes from the program and which from U. The naive comparison yields 2.68 instead of 1. More flexibility (GBM, causal forest) does not help, because the missing information is not in the data.

Why the instrument helps: Z is assigned at random, hence independent of U. It changes the participation rate from 0.26 to 0.74 and acts on Y only through T. The difference in Y between invited and non-invited customers, divided by the difference in participation rates, is the effect (Wald estimator): \(\frac{E[Y \mid Z=1] - E[Y \mid Z=0]}{E[T \mid Z=1] - E[T \mid Z=0]} \approx\) 0.97. The three conditions: Z affects T (testable), Z is random (here by design), Z acts only through T (not testable, an assumption).

Correct estimation: 2SLS with instrument Z (T ~ Z + X1, then Y ~ T_hat + X1). The instrumental forest is only needed if the effect additionally depends on X.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Monte Carlo standard errors of the means at most: ATE bias 0.032, CATE RMSE 0.044. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE
Naive (diff. in means)	1.70	0.09	1.70	2.31
LM/GLM (T + X)	1.71	0.08	1.71	2.08
LM/GLM (T x X)	1.71	0.08	1.71	2.08
GBM S-learner	1.56	0.09	1.59	2.09
GBM T-learner	1.71	0.08	1.75	2.10
LightGBM S-learner	1.58	0.10	1.61	2.09
LightGBM T-learner	1.71	0.08	1.75	2.10
Causal forest	1.71	0.08	1.76	–
Double ML (xgboost)	1.71	0.08	1.71	–
R-learner (xgboost)	1.71	0.08	1.71	–
AIPW (causal forest)	1.70	0.09	1.70	–
GBM pure (ignores T)	–	–	–	2.48
LightGBM pure (ignores T)	–	–	–	2.48
2SLS (uses Z)	0.02	0.18	0.15	–
Instrumental forest (uses Z)	-0.06	0.20	1.00	–

Lesson: All standard methods, including the causal forest (ATE bias 1.71), are equally strongly biased because U is missing. Only the instrument rescues the estimate. The classical 2SLS regression suffices for this (0.021); with a constant effect the instrumental forest only adds variance (CATE-RMSE 1.00 vs. 0.149). The advantage is one of design (a good instrument), not of algorithm. The instrument here is strong (participation 0.26 → 0.74); with a weak instrument both IV estimators would be considerably less certain.

C – When does flexibility pay off?

In this block all data are randomized, so there is no confounding. Instead, the effect varies from customer to customer, and the question is how flexible the model must be to capture this. The order runs from “a GLM suffices” (Case 6) through “modeling T is decisive” (Case 7) to “without a known form, the causal forest wins” (Case 8).

Case 6 – Linear heterogeneity

Story. A discount (T) is assigned at random. It works the more strongly, the more price-sensitive the customer is (X1). Y is revenue. X1 is an effect modifier; the effect changes linearly with X1.

Data generation (n = 2000, randomized):

\[ \begin{aligned} X_1, X_2, X_3 &\sim U(-1, 1), \qquad T \sim \text{Bernoulli}(0.5) \\ Y &= 2\, X_2 + 2\, X_3 + \tau(X_1)\, T + \varepsilon, \quad \varepsilon \sim N(0, 1.5^2) \\ \tau(X_1) &= 1 + 1.5\, X_1 \end{aligned} \]

Model input: \(Y, T, X_1, X_2, X_3\).

True effect: between −0.5 (\(X_1 = -1\)) and 2.5 (\(X_1 = 1\)), ATE = 1.

Correct estimation: lm(Y ~ T * X1 + X2 + X3). The LM with T x X (all variables) is almost exactly that: it has only two superfluous interactions with X2 and X3.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Top-30% uplift of a perfect selection (oracle): 2.05. Monte Carlo standard errors of the means at most: ATE bias 0.017, CATE RMSE 0.009, Top-30% uplift 0.006. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE	Top-30% uplift
Naive (diff. in means)	-0.01	0.11	0.87	2.30	1.01
LM/GLM (T + X)	0.01	0.07	0.87	1.56	1.04
LM/GLM (T x X)	0.02	0.06	0.13	1.50	2.05
GBM S-learner	-0.07	0.06	0.40	1.54	1.98
GBM T-learner	0.01	0.06	0.47	1.54	1.92
LightGBM S-learner	-0.06	0.06	0.39	1.54	1.99
LightGBM T-learner	0.01	0.06	0.46	1.54	1.93
Causal forest	0.02	0.06	0.24	–	2.02
Double ML (xgboost)	0.01	0.07	0.87	–	1.01
R-learner (xgboost)	0.01	0.06	0.34	–	1.99
AIPW (causal forest)	0.01	0.07	0.87	–	1.01
GBM pure (ignores T)	–	–	–	1.67	0.78
LightGBM pure (ignores T)	–	–	–	1.66	0.78

Lesson: If the effect varies in the way the model assumes, the GLM with interaction wins (CATE-RMSE 0.127 vs. causal forest 0.24). Without the interaction, the LM sees no heterogeneity (0.87). Caveat: in a preliminary test (8 repetitions) with a nonlinear baseline \(5 \sin(\pi X_2)\) instead of \(2 X_2\), the causal forest was on par with the GLM; the GLM wins only if the rest of the model is also correct. The R-learner (Double ML with xgboost as effect model) lies in between (0.34), clearly better than the average-effect estimator Double ML (0.87).

Case 7 – Risk ≠ uplift (binary)

Story. Retention calls (T) are assigned at random. Y = 1 means the customer stays. X1 is a cancellation risk score: the higher it is, the more likely the customer is to cancel the subscription. The call works only at medium risk (“persuadables”). High-risk customers leave anyway, and low-risk customers stay anyway. X1 is both a risk driver and an effect modifier.

Data generation (n = 3000, randomized):

\[ \begin{aligned} X_1, X_2 &\sim N(0, 1), \qquad T \sim \text{Bernoulli}(0.5) \\ P(Y = 1 \mid T = 0, X) &= \text{logit}^{-1}(0.5 - 1.5\, X_1) \\ P(Y = 1 \mid T = 1, X) &= \text{logit}^{-1}\big(0.5 - 1.5\, X_1 + 1.5\, e^{-X_1^2 / (2 \cdot 0.4^2)}\big) \end{aligned} \]

Model input: \(Y, T, X_1, X_2\).

True effect (on the probability scale): \(\tau(x) = P(Y{=}1 \mid T{=}1, x) - P(Y{=}1 \mid T{=}0, x)\). It is a bell shape with maximum 0.27 at \(X_1 \approx\) 0.15 and falls to ≈ 0 for high and low risk. ATE = 0.10.

Why the usual approach fails: A churn model (“pure” GBM) predicts the risk correctly and selects the 30% with the highest X1. Precisely there the effect is almost 0. This is an assumption of this case: if the effect increases with risk instead of lying at medium risk, the risk model works passably.

Correct estimation: a logit model with T * exp(-X1^2/0.32) – if you know the form. Without knowledge of the form: any model that models T.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Top-30% uplift of a perfect selection (oracle): 0.238. Monte Carlo standard errors of the means at most: ATE bias 0.003, CATE RMSE 0.002, Top-30% uplift 0.003. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE	Top-30% uplift
Naive (diff. in means)	-0.005	0.020	0.103	0.478	0.102
LM/GLM (T + X)	-0.004	0.016	0.080	0.395	0.170
LM/GLM (T x X)	-0.004	0.016	0.073	0.394	0.204
GBM S-learner	-0.019	0.013	0.043	0.393	0.231
GBM T-learner	-0.003	0.016	0.071	0.395	0.217
LightGBM S-learner	-0.018	0.014	0.043	0.393	0.231
LightGBM T-learner	-0.004	0.016	0.071	0.395	0.217
Causal forest	-0.003	0.017	0.059	–	0.220
Double ML (xgboost)	-0.004	0.016	0.103	–	0.102
R-learner (xgboost)	-0.003	0.016	0.062	–	0.221
AIPW (causal forest)	-0.004	0.017	0.103	–	0.102
GBM pure (ignores T)	–	–	–	0.399	0.037
LightGBM pure (ignores T)	–	–	–	0.399	0.037

Lesson: This is the most important case for practice. The standard approach (churn model → call the highest risk) achieves an uplift of 0.037 and is thus worse than random selection (0.102; Naive = random ranking). What matters is to model T. S-learner (0.23), GLM with interaction (0.20) and causal forest (0.22) are all close to the oracle; with only one relevant variable, no complex method is needed.

Case 8 – Many variables

Story. Randomized campaign (T). There are 20 customer features. The campaign works only if two conditions hold simultaneously (X1 > 0 and X2 > 0). This is not known in advance. The baseline depends nonlinearly on X3–X6; X7–X20 are noise.

Data generation (n = 2000, randomized, 20 variables):

\[ \begin{aligned} X_1, \dots, X_{20} &\sim N(0, 1), \qquad T \sim \text{Bernoulli}(0.5) \\ Y &= \underbrace{2 \sin(X_3) + X_4 X_5 + X_6}_{\text{baseline}} + \tau(X)\, T + \varepsilon, \quad \varepsilon \sim N(0, 1) \\ \tau(X) &= 2 \cdot \mathbb{1}\{X_1 > 0 \text{ and } X_2 > 0\} \end{aligned} \]

Model input: \(Y, T, X_1, \dots, X_{20}\).

True effect: 2 for the quarter of customers with \(X_1 > 0\) and \(X_2 > 0\), otherwise 0. ATE = 0.5.

Why it is difficult: The effect is an interaction of two variables, and which two they are is stated nowhere. A linear T x X cannot represent a “corner”. The GLM with all two-way interactions can approximate it roughly, but needs about 400 parameters to do so.

Correct estimation: lm(Y ~ sin(X3) + X4:X5 + X6 + T + T:I(X1 > 0 & X2 > 0)) – with knowledge one does not have in practice.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). Top-30% uplift of a perfect selection (oracle): 1.65. Monte Carlo standard errors of the means at most: ATE bias 0.014, CATE RMSE 0.009, Top-30% uplift 0.010. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE	Top-30% uplift
Naive (diff. in means)	0.03	0.09	0.87	2.26	0.50
LM/GLM (T + X)	0.02	0.05	0.86	1.61	0.51
LM/GLM (T x X)	0.02	0.05	0.74	1.59	1.19
GBM S-learner	-0.03	0.04	0.45	1.23	1.64
GBM T-learner	0.01	0.05	0.75	1.32	1.41
LightGBM S-learner	-0.03	0.04	0.45	1.22	1.65
LightGBM T-learner	0.01	0.04	0.75	1.32	1.41
Causal forest	0.01	0.04	0.28	–	1.65
Double ML (xgboost)	0.01	0.04	0.86	–	0.50
R-learner (xgboost)	0.01	0.04	0.36	–	1.64
AIPW (causal forest)	0.02	0.05	0.86	–	0.50
GBM pure (ignores T)	–	–	–	1.33	0.32
LightGBM pure (ignores T)	–	–	–	1.33	0.32
LM/GLM (T x all 2-way X)	0.01	0.05	1.08	1.36	1.02

Lesson: Here flexibility pays off. Linear T x X does not capture the form (CATE-RMSE 0.74), and the GLM with all interactions overfits (1.08). The causal forest has the smallest CATE error (0.28 vs. S-learner 0.45). When selecting the top 30%, however, the S-learner is equally good (1.64 vs. 1.65). The R-learner lies in between in CATE error (0.36) and also selects equally well (1.64). Estimating heterogeneity via Double ML thus does not match the causal forest; it is, however, a large gain over pure average-effect Double ML.

D – Both together

Considering blocks B and C separately is convenient, but practice combines them: an intervention is not assigned at random, and its effect differs from customer to customer. This is the typical situation in targeting from observational data.

Case 9 – Observational data and varying effect

Story. As in Case 3: the motivational call (T) goes preferentially to very young and very old customers (X1 = age), who also train less without a call (activity score Y, U-shaped). In addition, customers benefit to different degrees from the call, depending on a second quantity (X2, e.g., pre-existing conditions). X1 is the confounder, X2 the effect modifier.

Data generation (n = 2000, observational data):

\[ \begin{aligned} X_1, X_2 &\sim N(0, 1) \\ P(T = 1 \mid X_1) &= \text{logit}^{-1}(-1 + 1.2\, X_1^2) \\ \tau(x) &= 1 + 1.5\, X_2 \\ Y &= 10 - 1.5\, X_1^2 + X_2 + \tau(x)\, T + \varepsilon, \quad \varepsilon \sim N(0, 1) \end{aligned} \]

Model input: \(Y, T, X_1, X_2\) (all observed).

True effect: \(\tau(x) = 1 + 1.5\, X_2\), ATE = 1.

Why the GLM fails: As in Case 3, the U shape is in \(X_1^2\), and called customers have a 1.18 larger \(X_1^2\). lm(Y ~ T * X2 + X1) attributes this to the call: the effect at \(X_2 = 0\) is -0.78 instead of 1. The slope in \(X_2\), by contrast, is captured correctly by the GLM (1.51 instead of 1.5), because \(X_2\) is independent of the confounding. The GLM thus ranks customers well but gets the level wrong.

Correct estimation: Learn confounding and heterogeneity flexibly at the same time: causal forest or R-learner (Double ML with effect model). With a known form, lm(Y ~ T * X2 + I(X1^2)) suffices.

Results:

Means over all simulations. Bold = best method per column (ATE bias: closest to 0; ATE SD not marked, because a small spread is worth nothing when the bias is large). † No result in 21 runs (extreme propensities); mean over the remaining runs. Top-30% uplift of a perfect selection (oracle): 2.74. Monte Carlo standard errors of the means at most: ATE bias 0.023, CATE RMSE 0.021, Top-30% uplift 0.010. Differences below about twice that count as equally good.

Method	ATE bias	ATE SD	CATE RMSE	Y prediction RMSE	Top-30% uplift
Naive (diff. in means)	-1.76	0.14	2.32	2.88	1.00
LM/GLM (T + X)	-1.77	0.11	2.32	2.31	1.01
LM/GLM (T x X)	-1.76	0.10	1.77	2.18	2.73
GBM S-learner	-0.12	0.06	0.38	1.10	2.71
GBM T-learner	-0.34	0.09	1.26	1.09	2.62
LightGBM S-learner	-0.11	0.05	0.38	1.14	2.71
LightGBM T-learner	-0.41	0.08	1.40	1.16	2.59
Causal forest	-0.02	0.04	0.26	–	2.72
Double ML (xgboost)	-0.03	0.06	1.50	–	1.00
R-learner (xgboost)	-0.04	0.05	0.34	–	2.70
AIPW (causal forest) †	-0.08	0.10	1.50	–	1.00
GBM pure (ignores T)	–	–	–	1.36	-0.26
LightGBM pure (ignores T)	–	–	–	1.40	-0.25

Lesson: Here the ranking and the size of the effect separate. The GLM with T x X has the wrong level (ATE bias -1.76, CATE-RMSE 1.77) but selects almost as well as the oracle selection (top-30% uplift 2.73 vs. oracle 2.74), because the bias is the same for all customers. The causal forest is the most accurate (ATE bias -0.020, CATE-RMSE 0.26). The R-learner follows (0.34), ahead of the S-learner (0.38, ATE bias -0.117). AIPW fails because of the weak overlap in 21 of 40 runs. Whoever needs only a ranking gets far with the GLM; whoever computes costs and benefits per customer needs a method that adjusts correctly for the confounding.

Overall comparison

CATE error across all cases

How to read: The number in each tile is the method’s CATE-RMSE, i.e., the typical error per customer (0 = perfect). Because the errors lie on very different scales depending on the case (Case 7 measures probabilities, Case 2 score points), the color is relative: it shows how much larger the error is than that of the best method in the same case. Green = best method in the case, red = at least 4 times the error of the best. Empty = the method does not estimate an effect (pure GBM) or does not appear in the case.

Bias of the average effect

Number = mean ATE error (estimate − truth, 0 = perfect). Color = magnitude of the error.

Predictive accuracy is not effect accuracy

Each point is a method. x-axis: how well it predicts Y; y-axis: how well it estimates the effect per customer. For both, 0 = perfect. If good prediction implied a good effect estimate, the points would lie on a rising line.

In Case 4 (mediator) and Case 3 (nonlinear confounding), good prediction models give a poor effect estimate. The pure GBM predicts Y almost as well as the S-learner but is useless as a targeting model (Cases 7, 8). LightGBM vs. xgboost: practically identical in all cases. What matters is the construction (S, T, pure), not the library.

Where is the causal forest really better?

Excluding the “w/o mediator” variants, which solve the problem through domain knowledge. AIPW does not count as an alternative because it builds on the causal forest.

Case	Best method (CATE-RMSE)	Causal forest	Best alternative to the causal forest
1 Small n	AIPW (causal forest) (0.136)	0.21	LM/GLM (T + X) (0.141)
2 Confounding linear	LM/GLM (T + X) (0.041)	0.152	LM/GLM (T + X) (0.041)
3 Confounding nonlinear	Double ML (xgboost) (0.071)	0.177	Double ML (xgboost) (0.071)
4 Mediator	Naive (diff. in means) (0.123)	1.00	Naive (diff. in means) (0.123)
5 Unobs. confounding	2SLS (uses Z) (0.149)	1.76	2SLS (uses Z) (0.149)
6 Linear heterogeneity	LM/GLM (T x X) (0.127)	0.24	LM/GLM (T x X) (0.127)
7 Risk ≠ uplift	GBM S-learner (0.043)	0.059	GBM S-learner (0.043)
8 Many variables	Causal forest (0.28)	0.28	R-learner (xgboost) (0.36)
9 Observational + heterogeneity	Causal forest (0.26)	0.26	R-learner (xgboost) (0.34)

The causal forest is clearly the best method in Case 8 (unknown interaction among many variables) and in Case 9 (observational data with nonlinear confounding and a varying effect; there it is also nearly unbiased for the ATE). There, however, the S-learner and the R-learner deliver the same top-30% selection; the advantage lies in the more accurate magnitude of the effect per customer (e.g., for cost-benefit calculations), not in the ranking. In Case 3 (nonlinear confounding), the causal forest is nearly unbiased, but Double ML with ordinary xgboost estimates the average effect somewhat more accurately still. What wins there is the orthogonalized construction, not the forest.

Evaluation without knowing the truth

In real data, only the tools from the Methodology section “Evaluation methods” would be available. Here we know the truth and can check whether these tools would have found the right winner. To this end, they are computed on the same test data as the quality measures above, but without using the true effect.

R-loss and Qini: Do they find the best method?

In each simulation, the methods are ranked once by the metric and once by the truth (CATE-RMSE or true top-30% uplift, respectively). Rank correlation: 1 = same order. “Best found”: share of simulations (in %) in which the metric selects the same method as the truth as the best. Regret: how much worse, according to the truth, the selected method is than the best one. Qini only for randomized cases with a varying effect.

Case	R-loss: rank correlation	Best found	Regret (CATE-RMSE)	Qini: rank correlation	Best found	Regret (top-30% uplift)
1 Small n	0.94	50	0.03	–	–	–
2 Confounding linear	0.93	35	0.03	–	–	–
3 Confounding nonlinear	0.96	72	0.03	–	–	–
4 Mediator	-0.27	0	0.94	–	–	–
5 Unobs. confounding	-0.18	0	1.55	–	–	–
6 Linear heterogeneity	0.98	95	0.01	0.96	88	0.00
7 Risk ≠ uplift	0.90	48	0.00	0.86	48	0.00
8 Many variables	0.97	95	0.00	0.95	30	0.00
9 Observational + heterogeneity	0.97	75	0.04	–	–	–

Interpretation

• Identification holds (Cases 1–3, 6–9): The R-loss ranks the methods almost like the truth (rank correlation 0.98 in Case 6, 0.97 in Case 8), and anyone choosing by it loses little (regret at most 0.04). It does not always pick the single best method (Case 2: 35 %, Case 7: 48 %), because several methods are almost equally good there. The Qini value likewise ranks almost like the truth (Case 8: 0.95) but rarely names the best method exactly (30 %): causal forest, R-learner and S-learner are too close together. By contrast, pure ATE estimators (Qini around 0) and the pure GBM (Qini below 0, i.e., worse than chance) are reliably screened out.
• Identification fails (Cases 4, 5): The R-loss ranks the methods against the truth (rank correlation -0.27 and -0.18, respectively) and never selects the right method (regret 0.94 and 1.55, respectively). It rewards whatever fits the observed residuals, including the biased effect. As with predictive accuracy: a quality measure on observed data does not detect a wrong choice of variables.

Calibration (Case 8)

If a point lies on the dashed line, the prediction agrees with the experiment. The calibration slope (red line) summarizes this: 1 = calibrated, > 1 = predictions compressed too much, < 1 = spread too much. Averaged over 40 simulations in Case 8: causal forest 1.20, R-learner 1.23, S-learner 1.66, LM T x X 0.77. Calibration thus shows not only whether the effect is right but also in which direction a model is off. Constant estimators (Double ML, AIPW) have no spread and cannot be checked this way.

Overlap diagnostic

In Case 3, 6.6 % of customers lie outside 0.1–0.9 (Case 2: 2.8 %, randomized Case 6: 0.0 %). This is the warning signal that leads to failures in AIPW. If anything, the diagnostic underestimates the problem: the smoothed propensity from the forest does not reach the extremes, while the true propensity lies outside this range for 10.4 % of customers.

Placebo test and E-value (Case 5)

The placebo outcome has the same confounder (U) as Y but is not influenced by T. In real data, it must first exist.

Check	Estimate [95% CI]	Interpretation
Effect of T on Y, adjusted for \(X_1\) (LM)	2.65 [2.47; 2.83]	Measured “effect” (true effect: 1)
Placebo outcome on T, adjusted for \(X_1\)	1.58 [1.40; 1.76]	Should be 0, since T does not act on the placebo. Deviation: there is unobserved confounding.
Placebo outcome on the instrument Z, adjusted for \(X_1\)	-0.07 [-0.26; 0.13]	Should be 0 if Z acts only through T and has nothing to do with the confounder. Consistent with the assumption for the instrument.

The placebo outcome reveals the error: although \(T\) has no effect on it, an “effect” of 1.58 results. This almost equals the bias of the effect on \(Y\) (1.65). The placebo test thus detects the unobserved confounding where R-loss and quality measures fail. The instrument passes the same test (-0.07). The E-value of the measured effect is 4.4: an unobserved confounder would need a risk ratio of about 4.4 with both \(T\) and \(Y\) to explain it away completely. That sounds like a lot but is in fact the case here (\(U\) shifts \(T\) and \(Y\) strongly). The E-value says how strong a confounder would have to be, not whether it is that strong.

Agreement of methods

Excluding naive estimators, pure GBMs, variants using domain knowledge (“w/o mediator”) and instrument methods. Only methods with an effect per customer or an ATE.

Case	Methods	Range of the ATE estimates	Mean |ATE bias|
1 Small n	10	0.39	0.078
2 Confounding linear	10	0.23	0.085
3 Confounding nonlinear	10	1.74	0.49
4 Mediator	10	0.179	1.00
5 Unobs. confounding	10	0.150	1.68
6 Linear heterogeneity	10	0.089	0.023
7 Risk ≠ uplift	10	0.016	0.007
8 Many variables	11	0.051	0.016
9 Observational + heterogeneity	10	1.75	0.47

In Cases 4 and 5, the methods agree with one another (range 0.179 and 0.150, respectively) and are all off by about 1.00 and 1.68, respectively. In Case 3, by contrast, the estimates diverge widely (range 1.74), and here that is a legitimate warning signal. Disagreement therefore points to a problem; agreement proves nothing.

Lesson: The tools from the Methodology section distinguish good from bad methods as long as identification holds. Whether identification holds (right variables, no missing confounder) is not shown by R-loss, Qini, calibration or agreement of methods. There, placebo tests (for a missing confounder), checks of the instrument and above all domain knowledge help (for the mediator in Case 4 there is no data test).

Do the confidence intervals hold?

For the methods with a standard formula (see Methodology, “Confidence intervals”), the 95% interval for the average effect is computed in every run. The table shows in how many of the 40 runs it contains the true effect. A correct interval should do so in about 95% of runs.

Share of runs in which the 95% interval contains the true average effect. Bold = below 80%. With 40 runs, the share fluctuates by about ±7 percentage points even for a correct interval; values between 88 and 100% are compatible with 95%. – = no standard error (LM with binary Y in Case 7) or method not in the case.

Method	Case 1	Case 2	Case 3	Case 4	Case 5	Case 6	Case 7	Case 8	Case 9
Difference in means	95 %	0 %	0 %	92 %	0 %	95 %	95 %	98 %	0 %
LM (T + X)	95 %	95 %	0 %	0 %	0 %	98 %	–	100 %	0 %
Double ML	100 %	95 %	92 %	0 %	0 %	98 %	95 %	100 %	92 %
AIPW	98 %	98 %	97 %	0 %	0 %	98 %	90 %	100 %	95 %
2SLS	–	–	–	–	92 %	–	–	–	–

Lesson: Where identification holds, the intervals deliver what they promise (between 90 and 100%), also for Double ML and AIPW with ML nuisance models. The estimated standard errors match the actual spread across runs (a median of 1.10 times it, i.e., if anything slightly conservative). Where identification fails, not a single interval contains the true effect: the difference in means without random assignment (Cases 2, 3, 9), the LM with nonlinear predictors (Cases 3, 9), all adjusted methods with a mediator (Case 4), and all methods without an instrument when a variable is missing (Case 5). In Case 4, it is the simple difference in means that is correct, precisely because it does not use the mediator. A narrow interval is therefore no sign that the estimate is right.

Conclusion

When does CausalML pay off?

CausalML methods pay off when predictors act nonlinearly (Cases 3, 9), when the effect depends on many variables in an unknown form and its magnitude per unit matters (Cases 8, 9), and, for selection, only when the model includes the treatment (Case 7). They bring no advantage with small samples and in linear worlds (Cases 1, 2, 6), with a wrong choice of variables (Case 4), and with a missing variable without an instrument (Case 5). How to choose a method from this is explained in the guide; what has not been tested is listed in the box at the end of the guide.

Continuous treatment

Example: discount size instead of discount yes/no. In all cases, \(T\) is binary; what follows is methodological knowledge, not a simulation. The rules on confounder, mediator and collider apply unchanged. The question changes: the target is a dose-response curve or the slope \(\partial Y / \partial t\), and “whom to treat?” becomes “how much?”. LM/GLM, S-learner, Double ML (residuals \(T - \hat m(X)\) instead of the propensity), causal forest and 2SLS can be carried over to continuous \(T\); the T-learner and the difference in means cannot, and AIPW only via a generalized propensity density (unstable with thin overlap). Double ML and causal forest assume the effect to be linear in \(T\). If the effect saturates, they return a mean slope that is correct at no dose; the model must then learn the shape of the curve (S-learner, splines). Overlap here means: every relevant dose must occur among comparable customers; outside the observed range, every estimate is extrapolation.

Limitations

• The results apply to these data-generating processes. The ranking of the methods depends on the setup, not on a fixed hierarchy.
• The ATE of the row “Causal forest” is the mean of its CATE predictions on the test data set; the doubly robust estimator is listed separately as “AIPW (causal forest)”. Double ML (without an effect model) and AIPW deliver only an average effect; their CATE-RMSE is therefore poor by construction when the effect is heterogeneous, and their top-30% uplift corresponds to a random selection.
• In Case 3, overlap is weak at the extremes (for \(|X_1| > 2\), the propensity is close to 1). This makes the case harder for all methods, not easier.
• In real data, the true effect is never known; only proxy validation is possible there (e.g., uplift curves on randomized data).
• A much larger study of the same kind, with dozens of methods and teams, is the ACIC competition [23]. It reaches the same basic finding: flexible methods help with estimation, but no method wins everywhere.

References

The numbers in the text refer to this list.

Introductory reading

[1] Hernán, M. A., Robins, J. M. (2020). Causal Inference: What If. Chapman & Hall/CRC. Free: miguelhernan.org/whatifbook

[2] Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., Syrgkanis, V. (2024). Applied Causal Inference Powered by ML and AI. arXiv:2403.02467. Free: causalml-book.org

[3] Facure, M. (2022). Causal Inference for the Brave and True. Free: matheusfacure.github.io/python-causality-handbook

[4] Pearl, J., Glymour, M., Jewell, N. P. (2016). Causal Inference in Statistics: A Primer. Wiley.

[5] Angrist, J. D., Pischke, J.-S. (2009). Mostly Harmless Econometrics. Princeton University Press.

[6] Molnar, C. (2022). Modeling Mindsets: The Many Cultures of Learning From Data. christophmolnar.com/books/modeling-mindsets

Methods

[7] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21(1), C1–C68. doi:10.1111/ectj.12097

[8] Nie, X., Wager, S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika 108(2), 299–319. doi:10.1093/biomet/asaa076

[9] Künzel, S. R., Sekhon, J. S., Bickel, P. J., Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. PNAS 116(10), 4156–4165. doi:10.1073/pnas.1804597116

[10] Wager, S., Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. JASA 113(523), 1228–1242. doi:10.1080/01621459.2017.1319839

[11] Athey, S., Tibshirani, J., Wager, S. (2019). Generalized random forests. Annals of Statistics 47(2), 1148–1178. doi:10.1214/18-AOS1709

[12] Athey, S., Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. PNAS 113(27), 7353–7360. doi:10.1073/pnas.1510489113

[13] Robins, J. M., Rotnitzky, A., Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. JASA 89(427), 846–866. doi:10.1080/01621459.1994.10476818

[14] Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics 17(2), 3008–3049. doi:10.1214/23-EJS2157

[15] Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics 20(1), 217–240. doi:10.1198/jcgs.2010.08162

[16] Hahn, P. R., Murray, J. S., Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects. Bayesian Analysis 15(3), 965–1056. doi:10.1214/19-BA1195

[17] van der Laan, M. J., Rose, S. (2011). Targeted Learning: Causal Inference for Observational and Experimental Data. Springer.

[18] Athey, S., Wager, S. (2021). Policy learning with observational data. Econometrica 89(1), 133–161. doi:10.3982/ECTA15732

Pitfalls and evaluation

[19] Cinelli, C., Forney, A., Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research 53(3), 1071–1104. doi:10.1177/00491241221099552

[20] VanderWeele, T. J., Ding, P. (2017). Sensitivity analysis in observational research: Introducing the E-value. Annals of Internal Medicine 167(4), 268–274. doi:10.7326/M16-2607

[21] Gutierrez, P., Gérardy, J.-Y. (2017). Causal inference and uplift modelling: A review of the literature. Proceedings of Machine Learning Research 67, 1–13.

[22] Schuler, A., Baiocchi, M., Tibshirani, R., Shah, N. (2018). A comparison of methods for model selection when estimating individual treatment effects. arXiv:1804.05146.

[23] Dorie, V., Hill, J., Shalit, U., Scott, M., Cervone, D. (2019). Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science 34(1), 43–68. doi:10.1214/18-STS667

Process models and tools

[24] Petersen, M. L., van der Laan, M. J. (2014). Causal models and learning from data: Integrating causal modeling and statistical estimation. Epidemiology 25(3), 418–426. doi:10.1097/EDE.0000000000000078

[25] Hernán, M. A., Robins, J. M. (2016). Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology 183(8), 758–764. doi:10.1093/aje/kwv254

[26] Textor, J., van der Zander, B., Gilthorpe, M. S., Liśkiewicz, M., Ellison, G. T. H. (2016). Robust causal inference using directed acyclic graphs: the R package dagitty. International Journal of Epidemiology 45(6), 1887–1894. doi:10.1093/ije/dyw341. Online: dagitty.net

[27] Sharma, A., Kiciman, E. (2020). DoWhy: An end-to-end library for causal inference. arXiv:2011.04216. pywhy.org/dowhy

[28] PyWhy EconML: Library Flow Chart (decision tree for choosing an estimator). pywhy.org/EconML/spec/flowchart.html

Appendix: Software

Software	Version
R	4.6.1
grf	2.6.1
xgboost	3.2.1.1
lightgbm	4.7.0
ggplot2	4.0.3
ggrepel	0.9.8
rmarkdown	2.31
knitr	1.51

40 simulations per case, seeds deterministic (1000 · case number + repetition). Code and data: https://github.com/PhilippPro/blog_philipppro/tree/main/posts/causalml-toy-cases. As of: 2026-10-26.