LearnCausalLearnCausal
  • Modules

    Randomized experiments1 Matching and weighting2 Instrumental variables3 Regression discontinuity4 Difference-in-differences5 Synthetic control6 Machine learning7

    Choose and compare

    Choosing a design Method summaries
  • Interactive labs

    Plan your study Power calculatorM1 Overlap labM2 Complier explorerM3 Bandwidth explorerM4 Parallel trends explorerM5 Pre-test labM5 Estimator comparison labM5 Synthetic control explorerM6 Overfitting explorerM7 Replication labs
  • Practice

    Review Find the flaw Design projects

    Reference

    Glossary Bibliography Reporting checklists Code library Software packages
  • News

    Article reviews Portal updates
  • About

    About LearnCausal Editorial standards License and reuse Privacy
Start learning
Foundations

Module 5 · Difference-in-differences

Level
Code

Search

    Learn causal inference one design at a time

    Short lessons built on published studies, simulations you can change, and worked code in R, Python and Stata. Seven designs, each taught from foundations to expert level.

    Start learning →Explore methods

    Identification lens Lottery effect? U D Y Confounder Program Outcome
    Methods

    Browse methods

    Each method is a module with four levels. Start with any of them.

    Module 1

    Randomized experiments

    Assigning treatment by chance so that groups are comparable, and what can still go wrong.

    Module 2

    Matching and weighting

    Comparing like with like when treatment depends on observed characteristics.

    Module 3

    Instrumental variables

    Using a source of variation that affects treatment but not the outcome directly.

    Module 4

    Regression discontinuity

    Comparing units just above and just below an eligibility cutoff.

    Module 5

    Difference-in-differences

    Comparing the change in outcomes for a group that received a program with the change for a group that did not.

    Module 6

    Synthetic control

    Building a weighted comparison for a single treated unit.

    Module 7

    Machine learning for causal inference

    Using flexible prediction to adjust for many controls and to find who benefits most.

    Levels

    Four levels in every module

    Pick a method, then choose your level inside it with the Level switch at the top of every module. The levels build on each other, but you can start wherever your training fits.

    Foundations

    Intuition before notation, published examples, and a first estimate in R, Python or Stata. No prior course in causal inference is needed.

    Intermediate

    Estimation and inference for each design, with standard errors and the assumptions behind them.

    Advanced

    Validity checks, common threats, and the estimators to use when the simple version fails.

    Expert

    Recent methods from the research literature, what each one assumes, and how to report it.

    About

    What LearnCausal is for

    Mission

    To make causal inference accessible through free education that helps learners examine assumptions, choose methods, and evaluate evidence.

    Vision

    A world in which people in every discipline investigate causal questions rigorously and use credible evidence to inform decisions.

    Principles

    • Intuition firstEach idea starts with a concrete example, often from schools. Notation comes after.
    • Assumptions in the openEvery estimate is stated with the assumptions it needs and the ways they can fail.
    • Checked sourcesCited works are checked against their records or full texts. The editorial standards say what was not checked.
    • Reproducible codeEvery example runs in R, Python and Stata, so you can work in any of the three.

    Free to use, with credit

    Every lesson, simulation, and code example is open to anyone, with no account or subscription. Lessons are licensed under CC BY 4.0 and code under the MIT license. See license and reuse.

    For any field

    Most examples come from education and the social sciences, but the methods apply in any field that asks causal questions, from public health to economics and public policy.

    Sources and updates

    Each module lists its sources in an annotated reading list, with working papers labeled as such. All seven modules are complete at four levels, and lessons are revised as new research appears. See Portal updates.

    LearnCausal

    Free causal inference lessons, from foundations to expert level.

    Methods

    • Randomized experiments
    • Matching and weighting
    • Instrumental variables
    • Regression discontinuity
    • Difference-in-differences
    • Synthetic control
    • Machine learning
    • Choosing a design
    • Method summaries

    Tools

    • Plan your study
    • Power calculator
    • Overlap lab
    • Complier explorer
    • Bandwidth explorer
    • Parallel trends explorer
    • Synthetic control explorer
    • Overfitting explorer
    • Replication labs

    Resources

    • Review
    • Find the flaw
    • Design projects
    • Glossary
    • Bibliography
    • Reporting checklists
    • Code library
    • Software packages

    News

    • Article reviews
    • Portal updates

    About

    • About LearnCausal
    • Editorial standards
    • License and reuse
    • Privacy
    © 2026 Bakhyt ZhumatayVersion 1.0.0
    Tool · Module 1 · Randomized experiments

    Power and sample size calculator

    How small an effect could your trial detect? Set the design, and the calculator returns the minimum detectable effect size (MDES) in standard deviation units, using the formula from the Advanced level of Module 1 (Raudenbush, 1997).

    What is randomized?
    Averages across grades K to 12 reported by Hedges and Hedberg (2007).
    Uses about 0.75 of the between-school variance and 0.50 of the within-school variance explained, near the typical values in Hedges and Hedberg (2007).
    More settings
    Minimum detectable effectstandard deviations
    Design effect
    Multiplier
    Minimum detectable effect by sample size

    What to notice. With schools randomized, adding students within schools soon stops helping, because the between-school term shrinks only with the number of schools. A pretest often buys as much precision as doubling the number of schools. The calculation assumes a two-level design with treatment at the school level, equal school sizes, and the t multiplier with J − 2 degrees of freedom used in the lesson. Plans for real trials should also allow for attrition and for covariates' degrees of freedom.

    How this tool computes its numbers
    What is computed
    No simulation. The minimum detectable effect, in standard deviations of the outcome, comes from the closed-form formula MDES = M × √V, where M is the sum of two t quantiles, t(1 − α/2, df) + t(power, df).
    Students randomized
    V = (1 − R²1) / [P(1 − P)N], with df = N − 2. N is the number of students, P the share treated, and R²1 the share of outcome variance explained by student covariates such as a pretest.
    Schools randomized
    V = ρ(1 − R²2) / [P(1 − P)J] + (1 − ρ)(1 − R²1) / [P(1 − P)JM], with df = J − 2. J is the number of schools, M the students per school, ρ the intraclass correlation, and R²2 the share of school-level variance explained by school covariates. The design effect shown is 1 + (M − 1)ρ.
    Not included
    Attrition, unequal school sizes, blocking, and the degrees of freedom used by covariates.
    Checked against known values
    Compared with an independent computation of the same formula in Python for four designs, it agreed to three decimals. For example, 1,000 students, half treated, no covariates, 5% significance and 80% power give 0.177 standard deviations. Checked on 14 November 2026.

    The editorial standards describe how the site is checked.

    Read the lesson on minimum detectable effectsThe intraclass correlation

    Tool · Module 3 · Instrumental variables

    Complier explorer

    A tutoring offer is randomized between two equal groups of students. Some offered students never attend, and some students without an offer attend anyway. Change who complies and watch three estimates respond. The numbers are exact population values, not a random sample.

    Without tutoring, always-takers score this many points above compliers and never-takers this many points below.
    More settings
    For example, offered families also receive free reading books, worth this many points to every offered student.
    Shares of always-takers, compliers, and never-takers
    True effect for compliers
    Wald estimate
    Attended vs did notas-treated comparison

    What to notice. The motivation gap biases the attended-versus-not comparison but never the Wald estimate, because the offer was random. A direct effect of the offer biases the Wald estimate by that effect divided by the share of compliers, so the same small violation matters far more when few students comply. The always-takers' effect never enters the Wald estimate, which describes compliers only. The approximate standard error assumes 1,000 students and an outcome standard deviation of 10.

    How this tool computes its numbers
    What is computed
    No sampling. The tool computes population averages exactly from the shares and effects you set, so the same settings always give the same numbers.
    The population
    Always-takers (share a) attend whatever the offer, never-takers (share n) never attend, and compliers (share 1 − a − n) attend only if offered. Without tutoring, always-takers score s points above compliers and never-takers s points below. Tutoring raises compliers' scores by τ and always-takers' scores by τa. The offer itself can add d points to everyone, which breaks the exclusion restriction.
    Estimators
    The Wald ratio divides the difference in average scores between offered and not offered (the intention-to-treat effect) by the difference in attendance (the first stage). It equals τ + d / (1 − a − n), so it equals the compliers' effect when d = 0. The attended-versus-not comparison assumes half the students are offered.
    Standard error
    Approximate only, 10 × √(4 / 1,000) / first stage, which assumes an outcome standard deviation of 10 and 1,000 students split evenly.
    Checked against known values
    With no direct effect of the offer, the Wald estimate equalled the compliers’ effect exactly (4.00 and 6.00 in two settings). With a direct effect of 0.5 and 50% compliers it gave 5.00, the effect plus 0.5 / 0.5, as the formula predicts. Checked on 14 November 2026.

    The editorial standards describe how the site is checked.

    Noncompliance in plain termsThe three conditions

    Tool · Module 4 · Regression discontinuity

    Bandwidth explorer

    Students scoring at or above zero on a placement test receive a program. Choose how close to the cutoff to look and how flexible a line to fit on each side, and see how the estimated jump trades bias against noise. The data are simulated, 600 students.

    Only students within this distance of the cutoff are used.
    Fit on each side
    Kernel
    More settings
    True jump
    Estimated jump
    Students used
    Outcome by test score with fitted lines on each side of the cutoff
    Dots are students. The shaded band is the bandwidth. Solid lines are the fits, and the dashed line is the true relationship without the jump.

    What to notice. With curvature, a wide bandwidth and a straight line miss the shape near the cutoff, and the estimate is biased. A narrow bandwidth removes most of that bias but uses fewer students, so the interval widens. A flat fit is biased whenever the outcome rises with the score. The interval shown is the conventional one with robust standard errors. The Expert level of Module 4 explains why it can be too narrow when the bandwidth is chosen to minimize error, and how robust bias-corrected intervals fix this.

    How this tool computes its numbers
    Data
    600 students with a running score x drawn uniformly between −1 and 1. The score is y = 50 + 6x + 12k·x³ + τ·[x ≥ 0] + e, where k is the curvature setting, τ the true jump, and e normal with standard deviation 3.
    Estimator
    A separate weighted polynomial (flat, linear, or the order you choose) on each side of the cutoff, using only students within the bandwidth, with triangular or uniform kernel weights. The estimate is the difference between the two fitted values at the cutoff.
    Interval
    A conventional 95% interval from heteroskedasticity-robust standard errors. It is not bias-corrected, which is why it can miss the truth when the bandwidth is wide and the curve bends. The Expert lesson covers robust bias-corrected intervals.
    Randomness
    Pseudo-random numbers from a seeded generator (Mulberry32, with Box–Muller for normal draws). The tool starts at seed 11, and each new sample adds 1 to the seed, so every sample can be reproduced.
    Checked against known values
    Over 500 samples with a straight-line truth, a local linear fit, bandwidth 0.5 and a true jump of 5, the mean error was −0.06 (Monte Carlo standard error 0.03), and the 95% interval contained the true jump in 95.2% of samples. With a curved truth fitted by flat lines over the whole range, the mean error was +12.0, the failure the tool is meant to show. Checked on 14 November 2026.

    The editorial standards describe how the site is checked.

    Local linear estimation and bandwidthsThe idea of a cutoff

    Tool · Module 2 · Matching and weighting

    Overlap and balance lab

    Students with higher prior scores are more likely to join an after-school program, and prior scores also raise later scores. Change how strongly prior scores drive participation and compare four estimates with the truth. The data are simulated, 1,000 students.

    More settings
    Points of extra effect per standard deviation of prior score. Above zero, the effect on participants differs from the effect on everyone.
    Distribution of estimated propensity scores for participants and non-participants
    Estimated propensity scores. Participants above the line, non-participants below. Where one side is empty, there is no one to compare with.

    What to notice. The raw difference mixes the program's effect with prior scores. Stratifying, weighting, and matching remove that bias when overlap is good. As participation depends more strongly on prior scores, the two groups stop overlapping, weights become extreme, and the adjusted estimates become unstable. No adjustment can create comparisons the data do not contain. Here prior score is the only confounder and it is measured, which real studies cannot guarantee.

    How this tool computes its numbers
    Data
    1,000 students with a standardized prior score x. A student joins with probability 1 / (1 + exp(0.5 − γx)), where γ is the selection strength. The effect for a student is τ + h·x, and the score is y = 50 + b·x + (τ + h·x)·D + e, with e normal and standard deviation 3.
    Targets
    The average effect (ATE) is the mean of the individual effects in the sample, and the effect on participants (ATT) is their mean among those who joined. Weighting and stratification aim at the ATE, matching at the ATT.
    Estimators
    Stratification on fifths of the prior score, weighted by stratum size, dropping fifths without both groups. Inverse probability weighting with normalized weights and a propensity score from a logistic regression on the prior score, which matches how participation was generated. Nearest-neighbor matching on the prior score, with replacement.
    Randomness
    Seeded generator as in the other tools, starting at seed 5.
    Checked against known values
    Over 500 samples at the default settings with a constant effect of 3, the raw difference was off by +5.30, stratification on fifths by +0.58 (bias left within each fifth), weighting by +0.06 and matching by +0.07 (small finite-sample biases). Checked on 14 November 2026.

    The editorial standards describe how the site is checked.

    The two assumptions behind matchingEffect for whom?

    Tool · Module 6 · Synthetic control

    Synthetic control explorer

    One district adopts a reading program in 2016. Eight similar districts do not. The tool finds the weighted average of those districts that best tracks the adopting district before 2016 and uses it as the counterfactual afterward. The data are simulated.

    Raises the treated district's level. Once it lies outside the donors' range, no weighted average can match it.
    True effect
    Average gap after 2016
    Pre-period fit (RMSPE)root mean squared gap before 2016
    Reading scores of the treated district, its synthetic control, and the donor districts
    Solid teal is the adopting district, dashed gold its synthetic control, gray lines the donors.

    What to notice. With a long pre-period and a district inside the donors' range, the synthetic control tracks the treated district closely and the post-2016 gap recovers the effect. Shortening the pre-period lets the weights fit noise, and pushing the treated district above every donor leaves a gap that no weights can close, which shows up first as poor pre-period fit. Dropping the heaviest donor is the leave-one-out check from the Advanced level.

    How this tool computes its numbers
    Data
    Nine districts from 2005 to 2022, with the program starting in 2016. Each district's score is a district level plus two shared year patterns, each multiplied by a district-specific loading, plus noise with standard deviation 0.8. The treated district is built as 0.5 × D1 + 0.3 × D2 + 0.2 × D3 plus noise with standard deviation 0.5, so a perfect set of weights exists. The level-shift setting moves the treated district away from the donors. The true effect is added in every year from 2016.
    Weights
    Non-negative weights that sum to one, chosen to minimize the mean squared gap between the treated district and the weighted donors over the pre-period years you choose. They are found by projected gradient descent (4,000 steps). Only past outcomes are matched, with no other predictors.
    Estimate
    The average gap between the treated district and its synthetic control over 2016 to 2022. The fit statistic is the root mean squared prediction error over the pre-period years used.
    Randomness
    Seeded generator, starting at seed 3.
    Checked against known values
    Over 500 samples with the treated district inside the donors’ range, 11 pre-period years and a true effect of 4, the mean error was +0.01 (0.02). With the treated district shifted up by 8 points, the mean error was +2.77, the bias a poor pre-period fit warns about. Checked on 14 November 2026.

    The editorial standards describe how the site is checked.

    The idea of a synthetic controlModule 6

    Tool · Module 7 · Machine learning

    Overfitting explorer

    A model is fit to 30 training students and judged on 300 new ones. Make the model more flexible and watch the training error keep falling while the error on new students turns back up. The data are simulated.

    Training errormean squared error
    Error on new datamean squared error
    Best degree on new data
    Training data with the fitted curve and the true relationship
    Dots are training students. The teal curve is the fitted model, the dashed gold curve the true relationship.
    Training and test error by model flexibility
    Error by degree. Teal is training error, gold is error on new data. The dot marks your choice.

    What to notice. Training error always falls as flexibility grows, so it cannot tell you when to stop. Error on new data is lowest at a moderate degree, the bias-variance trade-off. A larger training sample moves the best degree upward. Causal methods that use machine learning judge models on held-out data for the same reason, through cross-validation and cross-fitting.

    How this tool computes its numbers
    Data
    A true relationship f(x) = 3 sin(2.2x) + 1.5x, with x uniform between −1 and 1. Observed scores add normal noise with the standard deviation you set. Training students are the first n of a fixed pool of 200, and the error on new data is measured on 300 further students.
    Model
    Ordinary least squares on Legendre polynomials up to the chosen degree. Errors are mean squared errors. On average, error on new data cannot fall below the noise variance, the square of the noise setting.
    Randomness
    Seeded generator, starting at seed 7.
    Checked against known values
    Over 500 samples with degree 3, 200 students and noise standard deviation 2, the average error on new data was 4.08, close to the floor of 4. With degree 14 and 30 students the training error averaged 1.97, while the median error on new data was about 1,060. Checked on 14 November 2026.

    The editorial standards describe how the site is checked.

    Fit versus overfitModule 7

    Resource · Software

    Software packages

    Widely used packages for each method in R, Python, and Stata. Packages marked "used here" appear in this site's code examples. Most other examples are written by hand so that each step is visible. Code examples on this site are in R, Python and Stata. The Stata column lists the matching Stata packages.

    Packages change over time. Check each package's documentation for current functions and defaults, and cite the package and its methods paper when you use it. Stata packages marked SSC install with ssc install followed by the package name.

    News

    Article reviews

    Nothing matches this filter yet.

    Article reviews

    New reviews are added every week. Each one summarizes what the authors report and states the limits of the evidence. A review is not an endorsement.

    How the reviews are made

    Each Friday, a search of scholarly databases and preprint servers lists new articles on causal inference methods. The reviews are drafted with the help of an AI assistant from the abstract and, where it is available, the full text. The references and links are checked against the arXiv or journal record when the list is compiled. The reviews have not been through peer review or editorial review, and most items are preprints, which can change or be withdrawn. Read the original article before you rely on a result.

    Portal updates

    The current release is listed below. The first number changes with a new module or redesign, the second with new lessons or tools, and the third with corrections.

    1. 15 November 2026ReleaseVersion 1.0.0

      LearnCausal 1.0.0

      The first public release of LearnCausal, with seven methods at four levels each and tools for practicing and reviewing what you learn.

      • Lessons. Randomized experiments, matching and weighting, instrumental variables, regression discontinuity, difference-in-differences, synthetic control, and machine learning. Each runs from Foundations to Expert, with worked code in R, Python and Stata. The Stata examples were run in Stata 18 and their output compared with R. Every level opens with an “About this level” box that states what to know beforehand, its review status, sources, packages, and linked article reviews. The editorial standards explain what has and has not been checked and how to report an error.
      • Assessment. Seven design projects, one per module, give you a simulated dataset with a hidden true effect; the site scores your design choice, estimate and check, and you rate your design memo against a model. The review page also schedules short delayed checks 7 and 30 days after you finish a level quiz, and ends with a printable summary of your results.
      • Practice. Each lesson has a quiz and recall questions. Your answers join the review page, which brings a question back the next day if you missed it and after 3, 7, 14, and 30 days if you got it right. Find the flaw gives seventeen invented study write-ups, each with one serious problem for you to name before you read the model answer.
      • Labs and calculators. The power, overlap, complier, bandwidth, synthetic control, and overfitting tools and the difference-in-differences labs ask for a prediction before they show a result. Each level ends with a five-question self-test, and the method summaries compare the seven designs. Each tool explains the model behind its numbers and how it performed against known true values. The study planner turns your choices into a design memo with a self-check.
      • Reference. A design guide, glossary, bibliography, reporting checklists, a code library, and a guide to software packages. Search covers all of them.
      • Your data. Quiz answers, the review queue, assessment results and the planner memo stay in your browser. Nothing you enter is sent to a server. Lessons and figures are licensed under CC BY 4.0 and code under the MIT license (license and reuse).
    Methods · Choose and compare

    Choosing a research design

    The design depends on how treatment was assigned and what data you have. Answer a few questions and the guide suggests a module to start with. More than one design may fit the same study, and the guide shows the one that usually needs the weakest assumptions first. Treat the result as a first draft of your design.

    Resource · Reference

    Glossary

    Short definitions of the terms used across the seven modules. Each entry links to the lesson that explains the term.

    Resource · Reference

    Bibliography

    Every work in the site's reading lists, in APA 7, with the modules that cite it. Annotations stay in each lesson's reading list.

      Resource · Practice

      Reporting checklists

      The "What to report" lists from every lesson, gathered by design. Use them when writing up a study or reviewing one.

      Methods · Choose and compare

      Method summaries

      One summary per module, built around the comparison that identifies the effect, so the designs can be compared side by side.

      Each summary answers the same four questions in order. When does the design fit, what does it estimate, what must be true, and what should you check? Estimators and technical notes are in the expandable part of each card. The notes carry one of four labels.

      Formal result A proven property that holds under stated conditions.

      Design guidance Recommended practice for choosing or reporting an analysis, which still depends on the study.

      Mixed empirical evidence Studies disagree, often for identifiable reasons.

      Active methods research Questions on which methods and advice are still developing.

      The labels reflect the site author's reading of the sources cited in each module, checked as the editorial standards describe.

      Module 1 Randomized experiments

      When to use it
      Use it when you control who receives the program and can assign it by chance, for example a lottery for tutoring places or a trial that randomizes schools.
      What it estimates
      The effect of being assigned to the program, compared with not being assigned, for the students or schools in the study. When some assigned students do not take part, it also estimates the effect for compliers, the students who take part only if assigned.
      What must be true
      • Assignment is random and carried out as planned.
      • One student's assignment does not change another student's outcome.
      • Outcomes are measured the same way, and as completely, in both groups.

      Chance makes the groups comparable in expectation, including on characteristics nobody measured, so the control group shows what would have happened to the treated group. A single trial can still be unbalanced by chance.

      What to check
      Compare response rates between groups, because attrition that differs can reintroduce bias. Check that the analysis follows the design, for example by school when schools were randomized. Baseline balance tables describe chance imbalance but do not test whether randomization worked.
      Estimators and technical notes
      Estimators taught
      Difference in means with Neyman standard errors, regression adjustment and randomization tests (Intermediate). Cluster-level analysis and power (Advanced). Intention to treat, complier effects and Lee bounds (Expert).
      Notes
      • Formal result Under complete randomization, regression with a centered covariate interacted with treatment is, in large samples, at least as precise as the difference in means (Lin, 2013).
      • Design guidance When schools are randomized, inference must be at the school level, and with few schools it needs small-sample corrections.
      • Active methods research Whether a result applies to your target population. Moving from the study sample to other schools needs information about who differs and assumptions about how effects vary.

      Module 2 Matching and weighting

      When to use it
      Use it when participation was chosen rather than assigned, but the factors that drove it were measured before the program, for example prior scores, attendance and family background for students who joined a summer program.
      What it estimates
      The effect for a population you specify, usually the participants (ATT) or everyone (ATE), built by comparing participants with non-participants who had similar measured characteristics. After trimming, the effect for the students with overlap.
      What must be true
      • No unmeasured confounding. The measured baseline characteristics capture everything that drives both participation and outcomes.
      • Overlap. Comparable participants and non-participants exist across those characteristics.
      • The characteristics are measured before the program.

      If participation depends only on measured characteristics, students who look alike on them differ in participation by something unrelated to their outcomes, so the reweighted comparison group stands in for the counterfactual.

      What to check
      Check balance with standardized differences and variance ratios, and look at the full distributions, not only means. Inspect overlap and how concentrated the weights are. None of these checks can detect a confounder that was not measured.
      Estimators and technical notes
      Estimators taught
      Stratification (Foundations). Propensity scores, nearest-neighbor matching and balance checks (Intermediate). Inverse probability weighting, entropy balancing, doubly robust and overlap weights (Advanced). Sensitivity analysis (Expert).
      Notes
      • Formal result A doubly robust estimator is consistent if either the outcome model or the propensity model is correct, provided no unmeasured confounding and overlap hold.
      • Design guidance Sensitivity analysis states how strong an unmeasured confounder would need to be to change the conclusion, for example with partial R² values and robustness values (Cinelli & Hazlett, 2020).
      • Mixed empirical evidence Comparisons with randomized benchmarks give mixed results. Agreement depends on which covariates were measured, how well, and how the comparison group was built.

      Module 3 Instrumental variables

      When to use it
      Use it when something outside the student's control shifts who receives the treatment without affecting outcomes in any other way, for example a lottery for charter school seats that not every winner accepts.
      What it estimates
      The average effect for compliers, the students whose treatment the instrument changes. It is not the effect for everyone. With covariates or several instruments, a weighted average of complier effects.
      What must be true
      • Relevance. The instrument shifts the treatment.
      • Independence. The instrument is as good as randomly assigned.
      • Exclusion. The instrument affects the outcome only through the treatment.
      • Monotonicity. Nobody takes the treatment because the instrument discourages it.

      The instrument creates groups that are comparable but differ in treatment rates, so the change in outcomes divided by the change in treatment isolates the effect for the students it moved.

      What to check
      Report the first stage and its strength, the reduced form, and the estimate together. Argue for exclusion from the setting, because no statistic tests it. Describe who the compliers are with the data.
      Estimators and technical notes
      Estimators taught
      The Wald ratio (Foundations). Two-stage least squares (Intermediate). Complier shares and characteristics (Advanced). Anderson–Rubin tests, tF and the effective F statistic (Expert).
      Notes
      • Formal result Under the four assumptions, with a binary instrument and treatment, the Wald ratio equals the average effect for compliers (Imbens & Angrist, 1994).
      • Formal result With one instrument, a rule that uses the usual t-test only when the first-stage F exceeds 10 does not keep a 5% error rate in the worst case. Lee et al. (2022) show the threshold for that rule is about 104.7.
      • Design guidance Weak-instrument-robust inference, such as the Anderson–Rubin test, is the safer default to report. Screening on the sign of the first stage (Angrist & Kolesár, 2024) suits settings where theory justifies the sign.

      Module 4 Regression discontinuity

      When to use it
      Use it when a rule assigns the program by whether a score crosses a cutoff, for example summer school for students who score below a line on a placement test.
      What it estimates
      The effect for students at the eligibility cutoff. When the cutoff only changes the chance of treatment, the effect for compliers at the cutoff.
      What must be true
      • Nothing other than the program changes abruptly at the cutoff.
      • Students cannot precisely control which side of the cutoff they land on.

      Students just above and just below the cutoff are alike except for the program, so the jump in outcomes at the cutoff shows its effect there.

      What to check
      Plot the data around the cutoff. Check the density of the score for bunching, check that characteristics fixed before the test do not jump, and run placebo cutoffs. Report estimates across a range of bandwidths.
      Estimators and technical notes
      Estimators taught
      The comparison at the cutoff (Foundations). Local linear regression and bandwidth choice (Intermediate). Density, covariate and placebo checks, and fuzzy designs (Advanced). Robust bias-corrected inference and local randomization (Expert).
      Notes
      • Design guidance Local linear estimation near the cutoff is preferred to global high-order polynomials (Gelman & Imbens, 2019).
      • Formal result Conventional intervals can under-cover at a bandwidth chosen for point estimation, because smoothing bias matters. Robust bias-corrected intervals address this (Calonico et al., 2014).
      • Design guidance The continuity and local randomization frameworks rest on different assumptions. One targets a limit at the cutoff, the other treats a narrow window as if randomized.

      Module 5 Difference-in-differences

      When to use it
      Use it when a program starts in some schools and not others, and outcomes are observed for both groups before and after, for example tutoring introduced in one set of schools in 2024.
      What it estimates
      The effect of the program on the schools that adopted it, over the period studied, found by comparing their change with the change in comparison schools. With staggered adoption, effects for each adoption cohort and period, then averaged.
      What must be true
      • Parallel trends. Without the program, both groups would have changed by the same amount.
      • No anticipation before the program starts.
      • No spillovers between groups and no other change hitting one group at the same time.

      If the groups would have moved in parallel, the comparison group's change shows what would have happened to the program schools without the program, and level differences cancel.

      What to check
      Plot both groups before adoption and look for concurrent changes. Similar trends before adoption support the argument but cannot prove it after adoption, so ask how large a violation would change the conclusion. With staggered adoption, state which units serve as controls at each date.
      Estimators and technical notes
      Estimators taught
      The two-by-two comparison (Foundations). Two-way fixed effects, event studies and covariates (Intermediate). Estimators for staggered adoption and sensitivity analysis (Advanced). Doubly robust and machine learning versions, changes-in-changes, triple differences and few treated units (Expert).
      Notes
      • Formal result With staggered adoption, two-way fixed effects can compare newly treated with already-treated units and put negative weights on some underlying treatment effects.
      • Formal result Passing a pre-trend test does not establish parallel trends, and reporting only results that pass distorts inference (Roth, 2022).
      • Formal result Parallel trends can hold for the outcome in levels and fail in logs, so the choice of scale matters (Roth & Sant'Anna, 2023).
      • Design guidance Choose among staggered-adoption estimators by the target effect, the timing of adoption, which units are valid controls, and how effects are averaged.
      • Active methods research Designs with a continuous dose. The effect at the doses observed and the effect of changing the dose need different arguments.

      Module 6 Synthetic control

      When to use it
      Use it when one district, state or school adopts a policy and a long series of outcomes exists for it and for comparable units that did not, for example a single district's new reading program.
      What it estimates
      The effect on the one adopting unit in each year after adoption, found by comparing it with a weighted average of untreated units.
      What must be true
      • The weighted untreated units would have followed the adopting unit's path without the policy.
      • The donors are unaffected by the policy and by shocks of their own.
      • No anticipation.

      Weights chosen to reproduce the adopting unit's path before adoption build a comparison that should keep tracking it afterwards, if nothing else changes.

      What to check
      Screen donors for credibility before estimating. Assess fit over an informative pre-period, and where feasible hold out some pre-period years to check it. Run placebo tests in space and time, and drop each donor in turn.
      Estimators and technical notes
      Estimators taught
      The weighted comparison (Foundations). Weights and placebo inference (Intermediate). Feasibility checks, leave-one-out and the demeaned estimator (Advanced). Augmented synthetic control and conformal inference (Expert).
      Notes
      • Design guidance Use the method only when the pre-period is informative and the fit is good, and judge the setting before seeing any result (Abadie, 2021).
      • Formal result With imperfect fit and adoption related to unobserved factors that drive outcomes, the bias can persist however many pre-periods are available (Ferman & Pinto, 2021).
      • Active methods research Choosing among augmented, demeaned and synthetic difference-in-differences estimators.
      • Design guidance Placebo tests ask whether the gap is unusual among untreated units, conformal inference whether post-period residuals look like pre-period ones, and prediction intervals how uncertain one unit's effect is. They answer different questions.

      Module 7 Machine learning for causal inference

      When to use it
      Use it inside a design that already identifies the effect, when there are many possible controls or when you want to know how effects vary, for example a randomized tutoring trial with hundreds of student records.
      What it estimates
      Usually the same average effect as the underlying design, estimated with flexible models for the outcome and the treatment. Also how effects vary across students and the value of a rule for whom to treat.
      What must be true
      • The design identifies the effect, through randomization or no unmeasured confounding with overlap.
      • Flexible models are fitted on data separate from where they are used, by cross-fitting or sample splitting.
      • Subgroups are found and tested on different data.

      Machine learning estimates the parts of the analysis that are prediction problems, such as outcome models and treatment probabilities. The identifying comparison still comes from the design.

      What to check
      Check the estimate with more than one learner. Validate claims about who benefits on held-out data, and judge a model by the effect or policy decision it serves rather than by its prediction accuracy alone.
      Estimators and technical notes
      Estimators taught
      Prediction versus causal questions (Foundations). Lasso, post-double-selection and double machine learning (Intermediate). Meta-learners, honest estimation and causal forests (Advanced). Inference on heterogeneity and policy learning (Expert).
      Notes
      • Formal result Double machine learning with orthogonal scores and cross-fitting gives valid inference for average effects when identification holds and the nuisance models are estimated accurately enough (Chernozhukov et al., 2018).
      • Design guidance Searching for subgroups and estimating their effects on the same observations can exaggerate apparent differences. Honest estimation, best linear predictors and sorted group effects validate heterogeneity (Chernozhukov et al., 2025).
      • Active methods research Which learner estimates conditional effects best. No single method wins across settings.
      About · How this site is made

      Editorial standards and corrections

      How LearnCausal is written and checked, what the checks do not cover, and how to report a mistake.

      How LearnCausal is made

      LearnCausal is written and maintained by one author, Bakhyt Zhumatay. The lessons, examples, exercises, quiz and self-test questions, code, and weekly article reviews were drafted with AI assistance. The author chose the topics and sources, edited the drafts, and decided what to publish, and remains responsible for the content. No outside expert has yet reviewed the lessons. Each level’s “About this level” box states its review status, and News explains how the weekly reviews are made. Corrections and suggestions are welcome, and the “Report an error” section below explains how to send them.

      What has been checked, and what has not

      • Sources. Each cited work was checked against its bibliographic record or full text when the lesson was written, and working papers are labeled.
      • Code. Every worked example runs in R, Python and Stata. The Stata versions were run in Stata 18 and checked against the R output.
      • Data. All data in examples, exercises and tools are simulated for teaching and report nothing about real schools or students.
      • Tools. Each interactive tool was run where the right answer is known, and its “How this tool computes its numbers” box gives the result.
      • The site. Each release is tested for page errors, broken links and accessibility (WCAG 2.2 AA) with automated tools, which do not find every barrier.

      The author’s check of cited sources is not independent verification. It does not show that every explanation, worked example, exercise, or quiz answer is correct, and no second person has reviewed them. If you rely on an explanation or a number, confirm it against the original source and your own computation.

      Report an error

      Write to the address that appears when this page loads with the page address, the sentence or answer in question, what you think is correct, and a source if you have one. Reports go to the author, and the site does not promise a reply time. Corrections are made in the next release, and each one that changes what a lesson says is listed below.

      Corrections

      • 14 November 2026, Synthetic control, Advanced. The summary of Ferman and Pinto (2021) said the method could remain biased “even with an imperfect fit”. It now says the bias arises when the fit is imperfect and adoption is related to the unobserved common factors that drive outcomes, which is what the paper shows.
      About · Reuse

      License and reuse

      How you may reuse LearnCausal material.

      Lessons and figures

      Copyright © 2026 Bakhyt Zhumatay. The lessons, figures, exercises and questions are licensed under CC BY 4.0. You may copy, share, translate and adapt them, including for commercial use, if you give credit, link to the license and note any changes. A suggested credit line is “Adapted from LearnCausal (Zhumatay, 2026), https://learncausal.org, CC BY 4.0.”

      Code

      The R, Python and Stata code is licensed under the MIT license.

      MIT license text

      Copyright (c) 2026 Bakhyt Zhumatay

      Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

      The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

      THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

      Not covered

      Cited works, datasets that the labs download, and the software packages the examples call keep their own terms. The licenses do not cover the LearnCausal name or logo. Parts of the site were drafted with AI assistance (see the editorial standards), so the licenses apply to the extent the material is protected by copyright.

      Third-party software

      MathJax (Apache License 2.0), Prism (MIT license), and the Merriweather, Source Sans 3 and JetBrains Mono fonts (SIL Open Font License 1.1). They are served from this site, and their license files are in the assets folder.

      For other uses, write to the address on the editorial standards page.

      LearnCausal

      Page not found

      The address you followed does not match a page on this site.

      The address may have been mistyped, or the page may have moved. Lessons, tools and resources can be reached from the menus at the top of the page.

      Go to the home page Browse the methods

      If a link on this site brought you here, please report it as described on the Editorial standards and corrections page.

      About · Privacy

      Privacy

      What this site stores and what it sends.

      Summary

      LearnCausal has no accounts, no advertising and no sign-up forms. It does not ask you for personal information.

      Stored in your browser

      To remember your place, the site saves some settings and entries in your browser's local storage, on your own device. These include your chosen level and language, your last position, your answers to self-tests and prediction prompts, and anything you enter in the tools and the study planner. This information is not sent to a server, and the author cannot see it. Clearing your browser's site data removes it.

      Counting visits

      The site uses Cloudflare Web Analytics to count visits. It reports page views, referring sites, countries, browser and device types and loading times, in aggregate. Cloudflare states that this service does not use cookies, localStorage or other client-side state to collect usage metrics, and that it does not fingerprint visitors by IP address, user agent or similar data. The author sees only the aggregate reports.

      Hosting

      Cloudflare hosts the site. Like any web host, it processes technical connection data, such as IP addresses, when a page is requested. See Cloudflare's privacy policy (opens in a new tab).

      Fonts and scripts

      The fonts, equation display and code highlighting are served from this site. The site does not load code from advertising or social networks and has no tracking pixels.

      Links, data and email

      Links to articles, datasets and software lead to other websites, which have their own privacy practices. Some code examples read public datasets from those sites when you run them on your own computer. Links for reporting an error open your own email program with a prepared message, and nothing is sent unless you send it. Messages are used to answer you and to correct the site.

      For questions, write to the address on the editorial standards page. This statement was last updated on 15 November 2026.

      Resource · Practice

      Review

      Questions you have already answered are brought back after a gap, and two short checks follow each level quiz. Waiting before you retrieve something tends to strengthen memory more than rereading it. Everything on this page is stored in this browser only.

      Questions to review

      A question you miss returns the next day. One you get right returns after 3, 7, 14, and then 30 days.

      Delayed checks

      When you finish a level quiz, two short checks are scheduled for that level, one after 7 days and one after 30 days. Each has three questions that appear nowhere else on the site, so the result shows what you remember rather than what you recognise.

      Your progress

      A dated summary of your design projects, delayed checks and level self-tests, built from the results saved in this browser. Print it or save it as a PDF to share with a teacher or supervisor.

      Resource · Practice

      Find the flaw

      Short write-ups of studies, each with one serious problem in how it reaches its conclusion. Choose the problem, then read the model answer, which also explains why the other choices are weaker. The cases are invented for this site, and your choices are saved in this browser so that you can see a running score.

      Resource · Practice

      Design projects

      One project per module. Each gives you a research question and a simulated dataset with a true effect that stays hidden until you submit. You choose the design, estimate the effect in R, Python or Stata, run the key check and write a short design memo. The site scores the design choice, the estimate and the check against the known answer (10 points) and then shows the true effect and a reference analysis. You rate your memo against a rubric and a model memo, and that rating is reported separately.

      Scores stay in this browser. Nothing is sent anywhere. See your printable summary of results on the Review page.

      Tools · Real data

      Replication labs

      Each lab takes the data from a published study and walks through the analysis step by step, in R, Python and Stata, with the output you should see. You check the design, estimate the effect, test it, compare your numbers with the published ones, and finish with questions to ask of your own study.

      Module 1 · Randomized experimentsClass size and achievement in Project STARKrueger (1999)EducationTennessee5 stepsModule 2 · Matching and weightingJob training and the experimental benchmarkLaLonde (1986); Dehejia and Wahba (1999)LaborUnited States6 stepsModule 3 · Instrumental variablesCollege proximity and the return to schoolingCard (1995)EducationUnited States5 stepsModule 4 · Regression discontinuityClose Senate races and the incumbent party's advantageCattaneo, Frandsen, and Titiunik (2015)PoliticsUnited States5 stepsModule 5 · Difference-in-differencesCastle doctrine laws and homicideCheng and Hoekstra (2013)CrimeUnited States5 stepsModule 6 · Synthetic controlThe economic cost of conflict in the Basque CountryAbadie and Gardeazabal (2003)EconomicsSpain5 stepsModule 7 · Machine learning for causal inference401(k) eligibility and household savingsChernozhukov et al. (2018)EconomicsUnited States6 steps

      How the labs work

      • The data come from the original authors where they share it, or from widely used packages for R, Python and Stata. Each lab names its source, and every file is free to download.
      • Every estimator is written out in base R, in NumPy and, where Stata has no built-in command that gives the same numbers, in Mata, so you can see each calculation. R, Python and Stata print the same numbers. In your own work, the packages listed under software packages do the same jobs with more options.
      • Teaching files sometimes differ from the authors' data. Each lab says where its numbers match the article and where they do not.
      • Each lab assumes the Foundations and Intermediate levels of its module.
      Replication lab · Module 1 · Randomized experiments

      Class size and achievement in Project STAR

      In Project STAR, Tennessee kindergartners and their teachers were randomly assigned, within each school, to a small class, a regular class, or a regular class with a full-time aide. Do smaller classes raise achievement? You will check the randomization, put test scores on Krueger's percentile scale, estimate the effect within schools, and look at who benefits and who stays in the study.

      Study
      Krueger (1999)
      Data
      Project STAR data as prepared for Stock and Watson's textbook and distributed with the AER package for R (Kleiber & Zeileis, 2008). R, Python and Stata read the same CSV from Rdatasets. 5,786 kindergartners with reading and math scores in 79 schools.
      Before you start
      Foundations and Intermediate of Module 1
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data

        Load the file and keep kindergartners with both a reading and a math score. Count the students in each type of class.

        What to notice. The three arms are similar in size. Every school had at least two types of class, and 78 of the 79 had all three. That matters because assignment happened within schools, so the cleanest comparisons are within schools too.

      2. Step 2 Check balance

        Randomization should make small and regular classes alike before assignment. Compare characteristics that the class could not have changed, including the experience of the teacher, who was also assigned at random.

        What to notice. Every difference is small, and no t statistic exceeds 0.4 in absolute value. This is what random assignment should produce. Balance on observed characteristics does not prove balance on everything, but it is the evidence a reader will look for first.

      3. Step 3 Compare test scores

        Krueger turned each test score into a percentile of the scores in regular classes, with and without an aide, and averaged reading and math. On this scale, 50 is typical for a regular class.

        What to notice. Small classes average the 56th percentile, against about the 51st in both kinds of regular class. The raw gap is 4.9 percentile points.

      4. Step 4 Compare within schools

        Because students were randomized within schools, compare them within schools by adding a fixed effect for each school. Students in the same school share many influences, so cluster the standard errors by school.

        What to notice. Within schools, small classes score 5.6 percentile points higher (SE 1.45). An aide in a regular class adds 0.25 points, too small to distinguish from zero. Krueger reports a gain of four percentile points in the first year students attend a small class, and little effect of aides.

        Your turn. Repeat the comparison for reading and math separately. Is the gain similar in both?

      5. Step 5 Who benefits, and who stays in the study

        Estimate the small-class effect separately for students who did and did not receive free lunch, then check how many kindergartners still have a test score a year later.

        What to notice. The estimate is 6.9 points for students on free lunch and 4.5 for the others, a pattern Krueger also reports. The two estimates have wide standard errors, so treat the difference with care. A first-grade score exists for 73.7 percent of students from small classes but 69.3 percent from regular classes. When attrition differs by arm, later comparisons can be biased, which is why Krueger adjusts for attrition and moves between classes.

      Compare with the published study

      QuantityPublishedThis lab
      Gain in the first year in a small classFour percentile points4.9 raw, 5.6 within schools (kindergarten)
      Teacher aidesLittle effect0.25 (SE 1.28)
      Larger effect for students on free lunchYes6.9 against 4.5

      Published values are from the abstract of Krueger (1999). His figure is for the first year in a small class, in whichever grade it came, while this lab uses kindergarten only.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. What exactly was randomized, the offer or the treatment itself? If people could decline, you are estimating the effect of the offer. Module 3 shows how to estimate the effect of taking it up.
      2. At what level was assignment made, student, classroom, or school, and within which blocks? Compare within blocks and cluster standard errors at the level where outcomes are related.
      3. Check balance on characteristics measured before assignment, and decide in advance which ones you will adjust for.
      4. Record missing outcomes and moves between arms by arm. If one arm loses more people, the arms may no longer be comparable.
      5. Put the outcome on a scale readers understand, such as percentiles or standard deviations, and report the comparison-group mean.
      6. Decide in advance which subgroups you will examine, so that a large effect in one is not a chance find.
      7. Register the main outcome and analysis before you see the data.

      When you write up, the reporting checklists list what readers will look for.

      References

      • Krueger, A. B. (1999). Experimental estimates of education production functions. The Quarterly Journal of Economics, 114(2), 497–532. https://doi.org/10.1162/003355399556052
      • Kleiber, C., & Zeileis, A. (2008). Applied econometrics with R. Springer. https://doi.org/10.1007/978-0-387-77318-6
      All labsReplication labsNext labJob training and the experimental benchmark
      Replication lab · Module 2 · Matching and weighting

      Job training and the experimental benchmark

      The National Supported Work program was a randomized experiment, so it gives a trustworthy estimate of the effect of job training on earnings. Replace its randomized controls with adults from a national survey, and the question becomes whether matching or weighting on observed characteristics can recover the experimental answer. Few settings let you check an observational method against the truth. This is one.

      Study
      LaLonde (1986); Dehejia and Wahba (1999)
      Data
      nsw_dw.dta (185 trainees, 260 randomized controls) and cps_controls.dta (15,992 survey adults), posted on Rajeev Dehejia's website with Robert LaLonde's permission.
      Before you start
      Foundations and Intermediate of Module 2
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data and the experimental benchmark

        Compute the experimental estimate first, then build the observational sample of trainees and survey adults.

        What to notice. The experiment gives $1,794 with a standard error of about $671. That is the target. The rest of the lab acts as if the randomized controls did not exist.

      2. Step 2 The naive comparison

        Compare trainees with survey adults directly.

        What to notice. Trainees earned $8,498 less in 1978 than the survey adults. Nobody believes the program cost participants that much. Trainees were drawn from people with weak work histories, and the raw comparison mixes that selection with the program.

      3. Step 3 How different are the groups?

        Measure how different the groups are with standardized mean differences.

        What to notice. Almost every variable is far out of balance. The largest differences are in race (2.43 standard deviations), earnings in 1974 and 1975 (1.57 and 1.75), and marital status (1.23). A common guideline treats values below 0.1 as adequate. These groups are nowhere near it.

      4. Step 4 Estimate the propensity score

        Estimate each person's probability of being a trainee from observed characteristics, with the terms Dehejia and Wahba used, and look at the overlap.

        What to notice. Of the 15,992 survey adults, 12,136 have a score below the lowest trainee's. They have no counterpart among the trainees and cannot inform the comparison. Only 34 have a score above 0.5, so trainees with high scores will share a few matches.

      5. Step 5 Match each trainee to the nearest survey adult

        Match each trainee to the survey adult with the closest score, with replacement, and check balance again.

        What to notice. Matching moves most differences close to zero, yet the estimate, $952, is far below the benchmark. Only 106 distinct survey adults serve as matches, so a few people carry much of the weight, and education and degree status remain imbalanced (0.18 and 0.25).

        Your turn. Match each trainee to its five nearest survey adults instead of one. How do the estimate and the balance change?

      6. Step 6 Weight instead of match

        Weight the survey adults by p / (1 − p) so that, together, they resemble the trainees, after dropping those outside the trainees' range of scores.

        What to notice. Weighting gives $1,860, close to the benchmark. Do not read too much into one close number. The same propensity score gave a matching estimate about $900 lower. In a real study there is no benchmark, so report several reasonable estimators together with the balance and overlap behind them.

      Compare with the published study

      QuantityPublishedThis lab
      Experimental estimate$1,794$1,794
      Trainees minus survey adults−$8,498−$8,498

      Published values are from Dehejia and Wahba (1999). Their propensity score estimates use other specifications and matching rules, so the adjusted estimates in this lab are not expected to equal theirs.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. List the characteristics that drive both who gets treated and the outcome, measured before treatment. If an important one is missing, no matching method can fix it.
      2. Look at overlap before estimating. Report how many comparison units fall outside the treated range, and drop or flag them.
      3. Check balance after matching or weighting with standardized differences, not t-tests.
      4. Decide whether you want the effect on the treated or on everyone, and choose the matching rule or weights to fit.
      5. Use at least two estimators and report them side by side.
      6. Run a sensitivity analysis for an unmeasured confounder (Module 2, Expert).

      When you write up, the reporting checklists list what readers will look for.

      References

      • LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604–620.
      • Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062. https://doi.org/10.1080/01621459.1999.10473858
      Previous labClass size and achievement in Project STARNext labCollege proximity and the return to schooling
      Replication lab · Module 3 · Instrumental variables

      College proximity and the return to schooling

      Do more years of school raise wages, or do people who stay in school differ in ways that would raise their wages anyway? Card used whether a man grew up near a four-year college as an instrument, because a nearby college makes more schooling cheaper. You will estimate the first stage, the reduced form, and two-stage least squares, and then ask whether the instrument is believable.

      Study
      Card (1995)
      Data
      card from the wooldridge package for R and Python (in Stata, bcuse card), which distributes Card's data with Wooldridge's econometrics textbook. 3,010 men from the National Longitudinal Survey of Young Men, wages measured in 1976.
      Before you start
      Foundations and Intermediate of Module 3
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata, after ssc install bcuse once.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data

        Load the data and compare schooling by whether a four-year college was nearby. The helper function computes least squares with robust standard errors and is used in every later step.

        What to notice. Men who grew up near a four-year college completed 0.83 more years of schooling on average. That raw gap mixes any effect of the college with differences between the places where colleges are located.

      2. Step 2 Ordinary least squares

        Start with ordinary least squares, controlling for experience and its square, race, Southern residence, and urban residence.

        What to notice. Each extra year of schooling is associated with log wages 0.074 higher, about 7.7 percent. Ability and family background could push this estimate either up or down, which is why an instrument is worth trying.

      3. Step 3 Does the instrument move schooling?

        The first stage asks whether the instrument moves schooling once the controls are included.

        What to notice. With controls, a nearby college adds 0.34 years of schooling. The F statistic, 17.5, passes the common rule of thumb of 10 but is not large. Module 3, Expert explains why weak instruments can mislead even above that line.

      4. Step 4 Reduced form and two-stage least squares

        The reduced form is the instrument's effect on wages. Dividing it by the first stage gives the instrumental variables estimate, and two-stage least squares gives the same number with a standard error.

        What to notice. The estimate, 0.132, is almost twice the OLS estimate, but its 95% confidence interval (0.037 to 0.228) is wide and includes the OLS value. If the instrument is valid, it applies to men whose schooling depended on having a college nearby. Card reported that the gains from proximity were concentrated among men from less-educated families, which is one reason the instrumental variables estimate can exceed OLS.

        Your turn. Compute the 2SLS estimate separately for men who live in the South and men who do not. Are the estimates and first stages similar?

      5. Step 5 Is the instrument as good as random?

        An instrument must be as good as randomly assigned. Compare places with and without a college on characteristics that schooling cannot change.

        What to notice. Places near a college are much more urban (82 against 48 percent) and less often in the South. Without controls the instrumental variables estimate is 0.188, and with them it is 0.132. An instrument that needs controls to look random depends on those controls being enough. A nearby college could also raise wages directly through local labor markets, which would violate the exclusion restriction.

      Compare with the published study

      QuantityPublishedThis lab
      2SLS, basic controls0.132 (SE 0.049)0.132 (SE 0.049)
      First-stage F16.717.5

      Published values are from Kang et al. (2021), who reproduce Card's basic specification on this file with conventional standard errors. This lab uses robust standard errors, so the F statistic differs slightly. Card's own tables add regional and family-background controls that this file does not contain.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. Name the instrument, the treatment, and the outcome, and draw the diagram from Module 3.
      2. Relevance. Report the first stage and its F statistic.
      3. Independence. Explain why the instrument is as good as random, and show balance on characteristics measured before treatment.
      4. Exclusion. List every path from the instrument to the outcome that does not pass through the treatment. Each one is a threat to defend against.
      5. Monotonicity. Could the instrument push some people away from treatment?
      6. Say who the compliers are. The estimate is their average effect, not everyone's.
      7. Report the reduced form, which needs fewer assumptions than the ratio.

      When you write up, the reporting checklists list what readers will look for.

      References

      • Card, D. (1995). Using geographic variation in college proximity to estimate the return to schooling. In L. N. Christofides, E. K. Grant, & R. Swidinsky (Eds.), Aspects of labour market behaviour: Essays in honour of John Vanderkamp (pp. 201–222). University of Toronto Press.
      • Kang, H., Jiang, Y., Zhao, Q., & Small, D. S. (2021). ivmodel: An R package for inference and sensitivity analysis of instrumental variables models with one endogenous variable. Observational Studies, 7(2), 1–24. https://doi.org/10.1353/obs.2021.0029
      • Kling, J. R. (2001). Interpreting instrumental variables estimates of the returns to schooling. Journal of Business & Economic Statistics, 19(3), 358–364. https://doi.org/10.1198/073500101681020006
      • Shea, J. M. (n.d.). wooldridge: 115 data sets from "Introductory econometrics: A modern approach" by Jeffrey M. Wooldridge [R package]. Retrieved November 10, 2026, from https://CRAN.R-project.org/package=wooldridge
      Previous labJob training and the experimental benchmarkNext labClose Senate races and the incumbent party's advantage
      Replication lab · Module 4 · Regression discontinuity

      Close Senate races and the incumbent party's advantage

      When a party narrowly wins a U.S. Senate seat, does it do better the next time that seat is contested, six years later? A race decided by a fraction of a point is close to a coin flip, so states on either side of a zero margin should be alike. You will plot the jump, estimate it at several bandwidths, treat the closest races as a small randomized experiment, and test the design.

      Study
      Cattaneo, Frandsen, and Titiunik (2015)
      Data
      rdrobust_RDsenate, distributed with the rdrobust package for R, Python and Stata. 1,390 Senate elections, 1914 to 2010, constructed by Cattaneo, Frandsen, and Titiunik (2015).
      Before you start
      Foundations and Intermediate of Module 4
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data

        Load the file. The running variable is the Democratic margin of victory at election t, and the outcome is the Democratic vote share when the same seat is next contested.

        What to notice. Of 1,390 elections, 1,297 have an outcome six years later, and the analysis uses those. Report how many rows you use and why the others are missing.

      2. Step 2 Look at the jump

        Average the outcome in 5-point bins of the margin on each side of zero.

        What to notice. The outcome rises gently with the margin on each side, then jumps from 45.0 to 52.8 between the bins just below and just above zero. A plot of these bin means is the first figure readers expect.

        Your turn. Plot the bin means with a vertical line at zero, using 2.5-point bins from −50 to 50.

      3. Step 3 Local linear estimates

        Fit a separate line on each side of zero with a triangular kernel, at several bandwidths.

        What to notice. From a bandwidth of 10 points upward the estimate is stable, between 7.3 and 8.0 percentage points. With a bandwidth of 5 it jumps to 12.3, but it rests on only 245 elections and its standard error nearly doubles. Cattaneo, Frandsen, and Titiunik report 7.43 with a local linear estimator.

      4. Step 4 Treat the closest races as a randomized experiment

        Within a very narrow window the winner may be as good as randomly assigned. Compare means in the window and judge the difference by reshuffling who won, the logic of a randomization test.

        What to notice. Within 0.75 points of a tie there are 22 Democratic wins and 15 losses, and the Democratic vote share six years later differs by 9.69 points. None of 2,000 reshuffles produced a difference that large. The article reports 9.32 in this window with 23 wins and 15 losses, so this copy of the data differs by one election near the edge of the window. Small windows are sensitive to single cases, which is one reason to report both approaches.

      5. Step 5 Check the design

        Two checks. First, narrow wins and losses should be about equally common. Second, characteristics settled before election t should not jump at zero.

        What to notice. Wins and losses are balanced within 2.5 points (65 and 63), and three of four characteristics show no clear jump. Open seats are the exception. Narrow Democratic wins were 17 percentage points less likely to be open-seat races (SE 0.064). With several checks, one significant result can arise by chance, but this one deserves a closer look, for example by adding the open-seat indicator as a covariate or estimating separately by seat type.

      Compare with the published study

      QuantityPublishedThis lab
      Local linear estimate7.437.27 to 7.98 at bandwidths 10 to 20
      Difference in means, window of 0.759.32 (23 wins, 15 losses)9.69 (22 wins, 15 losses)

      Published values are from Table 3, Panel A, of Cattaneo, Frandsen, and Titiunik (2015). Their local linear estimate uses a bandwidth chosen from the data, so this lab shows a range of fixed bandwidths instead.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. Name the running variable, the cutoff, and the rule. Is treatment fully decided at the cutoff (sharp) or only made more likely (fuzzy)?
      2. Could anyone control which side of the cutoff they land on? Look at the density of the running variable near the cutoff.
      3. Plot binned means before you estimate anything.
      4. Use local linear regression, report several bandwidths, and add a data-driven choice with robust confidence intervals (Module 4, Expert).
      5. If a narrow window is plausible as an experiment, add a randomization test (Module 4, Expert).
      6. Test characteristics fixed before treatment, and say what you will do if one of them jumps.
      7. State that the estimate describes units near the cutoff.

      When you write up, the reporting checklists list what readers will look for.

      References

      • Cattaneo, M. D., Frandsen, B. R., & Titiunik, R. (2015). Randomization inference in the regression discontinuity design: An application to party advantages in the U.S. Senate. Journal of Causal Inference, 3(1), 1–24. https://doi.org/10.1515/jci-2013-0010
      • Calonico, S., Cattaneo, M. D., & Titiunik, R. (2015). rdrobust: An R package for robust nonparametric inference in regression-discontinuity designs. The R Journal, 7(1), 38–51. https://doi.org/10.32614/RJ-2015-004
      Previous labCollege proximity and the return to schoolingNext labCastle doctrine laws and homicide
      Replication lab · Module 5 · Difference-in-differences

      Castle doctrine laws and homicide

      Between 2000 and 2010, 21 states expanded the right to use lethal force in self-defense outside the home. Did homicides rise? States adopted in different years, so this lab moves from one clean two-by-two comparison to the article's two-way fixed effects model and then to comparisons by adoption group that avoid using early adopters as controls.

      Study
      Cheng and Hoekstra (2013)
      Data
      castle from the causaldata package. 50 states, 2000 to 2010, with homicide rates per 100,000 and state populations. Scott Cunningham prepared this file for Causal Inference: The Mixtape (Cunningham, 2021), whose difference-in-differences chapter also works through this study. The code and text in this lab were written independently.
      Before you start
      Foundations and Intermediate of Module 5
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data

        Load the panel and find the first year each state had the law.

        What to notice. The adopting states fall into five groups by year, with 13 states in 2007, and 29 states never adopted. Because adoption is staggered, the comparisons in Step 4 matter. This file switches the law on for whole years. The article codes the share of the year in which the law was in effect.

      2. Step 2 One clean comparison

        Start with one two-by-two comparison. The 2007 adopters against never adopters, 2006 against 2007 to 2010, with means weighted by population.

        What to notice. Log homicide fell in both groups, by 0.093 in the adopters and by 0.121 in the never adopters. The difference, 0.027, suggests about 3 percent more homicides than the adopters' trend would predict. One comparison is noisy. The next steps use every state.

      3. Step 3 Two-way fixed effects, as in the article

        Estimate the article's main model by weighted least squares with state and year effects, clustering standard errors by state.

        What to notice. The estimate is 0.0755 (SE 0.0348), about 7.8 percent more homicides. Table 5 of the article reports 0.0801 (SE 0.0342) for this specification. The difference in how the law is coded is the likely reason for the small gap.

      4. Step 4 Effects by adoption group and year

        Compare each adoption group with never adopters, measuring both from the year before adoption, and average across adopting state-years.

        What to notice. The 2006 group, a single state, shows effects of 0.23 to 0.35 in every year, while the 13 states that adopted in 2007 show effects near zero on average. The average over adopting state-years, 0.042, is about half the fixed-effects estimate. With staggered adoption, two-way fixed effects weights groups unevenly and uses already-treated states as controls, so the two can disagree. Module 5, Advanced explains why.

        Your turn. Use states that have not yet adopted, not only never adopters, as the comparison group. Does the average change?

      5. Step 5 Check the years before adoption

        Check the years before adoption, where the law cannot have had an effect, and estimate the same model for crimes the law should not affect.

        What to notice. The pre-adoption comparisons are negative (−0.022 to −0.076), so homicide in adopting states was rising relative to never adopters just before the law. With few states in some groups these comparisons are imprecise, and the article's own pre-adoption test, a different specification, finds little divergence. Treat them as a warning. Larceny shows no effect (0.0012). Motor vehicle theft shows 0.041, and the article notes that this estimate is sensitive to region-by-year effects.

      Compare with the published study

      QuantityPublishedThis lab
      Two-way fixed effects, weighted0.0801 (SE 0.0342)0.0755 (SE 0.0348)
      Headline effect on homicideAbout 8 percentAbout 7.8 percent
      Larceny (falsification)Close to zero0.0012

      Published values are from Cheng and Hoekstra (2013), Table 5, Panel A, column 1, and from the abstract, which reports a statistically significant 8 percent net increase in homicides.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. When did each unit start treatment? If adoption is staggered, use a group-time estimator, not only two-way fixed effects.
      2. Choose the comparison group, never treated or not yet treated, and say why.
      3. Plot outcomes and effects by year relative to adoption, including the years before.
      4. Decide whether to weight by population or enrollment, and state the reason.
      5. Cluster standard errors at the level where the policy was assigned.
      6. Add an outcome the policy should not affect.
      7. Ask how large a violation of parallel trends would overturn the result (Module 5, Advanced).

      When you write up, the reporting checklists list what readers will look for.

      References

      • Cheng, C., & Hoekstra, M. (2013). Does strengthening self-defense law deter crime or escalate violence? Evidence from expansions to castle doctrine. Journal of Human Resources, 48(3), 821–854. https://doi.org/10.3368/jhr.48.3.821
      • Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. https://doi.org/10.12987/9780300255881
      • Huntington-Klein, N., & Barrett, M. (n.d.). causaldata: Example data sets for causal inference textbooks [R, Python and Stata packages]. Retrieved November 10, 2026, from https://github.com/NickCH-K/causaldata
      Previous labClose Senate races and the incumbent party's advantageNext labThe economic cost of conflict in the Basque Country
      Replication lab · Module 6 · Synthetic control

      The economic cost of conflict in the Basque Country

      Terrorism in the Basque Country began in the late 1960s. What would its income have been without it? No other Spanish region looks like the Basque Country, so this lab builds a weighted blend of regions that tracks its income before 1970, measures the gap that follows, and then asks how unusual that gap is.

      Study
      Abadie and Gardeazabal (2003)
      Data
      basque.csv, real GDP per capita for 17 Spanish regions, 1955 to 1997. Download the file. It is taken from the pysyncon package (MIT license), which distributes the data of Abadie and Gardeazabal (2003).
      Before you start
      Foundations and Intermediate of Module 6
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data

        Load the file, drop the national total, and arrange GDP per capita with one column per region.

        What to notice. In 1969 the Basque Country's GDP per capita was 6.08 against a donor average of 4.33. No single region resembles it, which is the situation synthetic control was built for.

      2. Step 2 Build the synthetic Basque Country

        Find weights, zero or positive and summing to one, that make the weighted donors track the Basque path from 1955 to 1969 as closely as possible. The loop is a projected gradient method, written out so that R, Python and Stata give the same answer.

        What to notice. Six regions receive weight, led by Madrid (0.394), Baleares (0.255), and Cataluña (0.198). The article's weights were Cataluña 0.85 and Madrid 0.15. They came from matching 1960s characteristics such as investment, sector shares, and schooling, with the importance of each chosen to reproduce the GDP path. This lab matches the GDP path directly. Different choices give different weights, which is why the weights belong in any report.

      3. Step 3 Measure the gap

        Compare the Basque Country with its synthetic control year by year.

        What to notice. The two paths stay close until the mid-1970s. The Basque Country then falls 11 to 14 percent below its synthetic control in the 1980s, and the gap narrows to 8 percent by 1997. The average gap from 1975 to 1997 is 10.8 percent, close to the decline of about 10 percentage points the article reports.

      4. Step 4 Placebos across regions

        Pretend each donor was treated instead. Fit its own synthetic control from the other donors and compare its gap after 1975 with its fit before 1970.

        What to notice. The Basque gap after 1975 is the second largest in the table, behind only Baleares, whose fit before 1970 is poor. Ranked by the ratio of the two, though, it is sixth of 17, a placebo p-value of 0.35. Several regions fit very closely before 1970 also fell well below their synthetic controls, among them the northern regions of Asturias and Cantabria. Part of the Basque gap may reflect forces it shared with them, a possibility any write-up should discuss.

        Your turn. Drop placebo regions whose fit before 1970 is more than twice as poor as the Basque fit. Where does the Basque Country rank now?

      5. Step 5 Leave one donor out

        Refit without each donor that received weight, one at a time.

        What to notice. The average gap stays between 7.2 and 11.1 percent. It moves most without Baleares or Madrid, the two largest weights, but stays negative and sizable.

      Compare with the published study

      QuantityPublishedThis lab
      Donor weightsCataluña 0.85, Madrid 0.15Madrid 0.39, Baleares 0.26, Cataluña 0.20, three others
      Decline in GDP per capitaAbout 10 percentage points10.8 percent on average, 1975 to 1997

      Published values are from Abadie and Gardeazabal (2003). They chose weights by matching economic characteristics, while this lab matches the GDP path, so the weights differ even though the size of the gap is similar.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. Do you have one or a few treated units and a long period before treatment? Synthetic control needs both.
      2. Choose donors that could plausibly have followed the treated unit's path and were not affected by the treatment or a similar shock.
      3. Report the weights and the fit before treatment.
      4. Run placebos across units and across time, and show the full distribution, not only the rank.
      5. Drop each important donor in turn and report how much the estimate moves.
      6. Date the start of treatment carefully. Effects may begin before the official date if people anticipate it.

      When you write up, the reporting checklists list what readers will look for.

      References

      • Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the Basque Country. American Economic Review, 93(1), 113–132. https://doi.org/10.1257/000282803321455188
      • Huntington-Klein, N., & Barrett, M. (n.d.). causaldata: Example data sets for causal inference textbooks [R, Python and Stata packages]. Retrieved November 10, 2026, from https://github.com/NickCH-K/causaldata
      Previous labCastle doctrine laws and homicideNext lab401(k) eligibility and household savings
      Replication lab · Module 7 · Machine learning for causal inference

      401(k) eligibility and household savings

      Does being eligible for a 401(k) retirement plan raise a household's savings? Eligibility depends on the employer, and employers that offer plans may hire people who already save more, so the answer depends on controlling well for income, age, and other characteristics. This is one of the applications in the article that introduced double machine learning. You will reproduce its starting point, add flexible controls with a lasso, and check how stable and how well supported the answer is.

      Study
      Chernozhukov et al. (2018)
      Data
      sipp1991.dta, 9,915 households from the 1991 Survey of Income and Program Participation, from the authors' replication repository (MIT license). The same file loads with the DoubleML package in R and Python. The same data were used by Chernozhukov and Hansen (2004).
      Before you start
      Foundations and Intermediate of Module 7
      Software
      R, Python or Stata. Install the data once with install.packages("causaldata") in R or pip install causaldata in Python. Everything else is base R or NumPy and pandas. The Stata version reads the data from the web and uses Stata's own commands and Mata.

      Run the steps in order in one session, since each builds on the one before. The output under each step is what you should see, in R, Python and Stata alike.

      1. Step 1 Load the data

        Load the file and look at the outcome, net financial assets, and the treatment, eligibility for a 401(k).

        What to notice. About 37 percent of households are eligible, and the average household holds about $18,000 in net financial assets.

      2. Step 2 Compare eligible and ineligible households

        Compare eligible and ineligible households without any controls.

        What to notice. Eligible households hold $19,559 more, exactly the unadjusted figure in the article (SE 1,413). Eligible households also earn more and differ in other ways, so this comparison mixes the plan with who has access to it.

      3. Step 3 Add the controls by least squares

        Add the nine controls the article uses by least squares, each entering in a straight line.

        What to notice. The estimate falls to $5,896. Least squares assumes, for example, that assets rise in a straight line with income, which is unlikely, and a wrong functional form can push the estimate in either direction.

      4. Step 4 Double machine learning

        Build 51 candidate controls from the nine variables, their squares and cubes, and their pairwise products. A lasso predicts assets and eligibility from them, always on folds it did not see, and the effect comes from regressing one prediction error on the other. The lasso and its penalty are the ones used in Module 7.

        What to notice. Double machine learning gives $8,869 (SE 1,379), with a 95% confidence interval from $6,165 to $11,573. The article's partially linear estimates with five folds run from $8,187 with the lasso to $9,247 with random forests. Flexible controls give an estimate well below the unadjusted $19,559 and above the straight-line $5,896.

      5. Step 5 Repeat with different splits

        Cross-fitting depends on a random split. Repeat it with other splits and report the median, as the article does with 100 splits.

        What to notice. Other splits give $8,309 to $9,084, and the median of all six is $8,853. Report the median together with this spread.

        Your turn. Use 2 folds instead of 5, as in the article's first row. Do the estimates move?

      6. Step 6 Check overlap

        Look at the cross-fitted predictions of eligibility, drop households whose predicted eligibility is below 0.01 or above 0.99, as the article does for its interactive model, and estimate again.

        What to notice. Because the lasso fits a straight line, some predictions exceed 1, a reminder that these are tools for removing confounding rather than probabilities. Only 34 households fall outside the range, and dropping them moves the estimate to $8,577. Overlap is good here, unlike in the matching lab. Remember that the estimate is the effect of being eligible, not of participating, since not every eligible household joins the plan.

      Compare with the published study

      QuantityPublishedThis lab
      Eligible minus ineligible, no controls$19,559 (SE 1,413)$19,559 (SE 1,413)
      Partially linear model, lasso, 5 folds$8,187$8,869 (median $8,853)
      Partially linear model, range across learners$8,187 to $9,247Lasso only

      Published values are from Chernozhukov et al. (2018), Section 6.2 and Table 2, Panel B. Their lasso uses a different set of candidate controls and 100 sample splits, so the estimates are close but not identical.

      Apply it to your study

      Work through these questions with your own design before you collect or analyze data. Most of them correspond to a step above.

      1. Settle the causal question and the identifying assumption first. Machine learning helps estimate the parts of the model that are not of interest. It does not make unconfoundedness more plausible.
      2. List candidate controls measured before treatment, and leave out anything the treatment could have changed.
      3. Start from the unadjusted and the simple regression estimates, so readers can see what the flexible controls change.
      4. Use cross-fitting and report results over several splits.
      5. Check overlap with the cross-fitted predictions of treatment.
      6. Try more than one learner, such as the lasso and random forests, and report them all.
      7. For effects that differ across people, see Module 7, Advanced and Expert.

      When you write up, the reporting checklists list what readers will look for.

      References

      • Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097
      • Chernozhukov, V., & Hansen, C. (2004). The effects of 401(k) participation on the wealth distribution: An instrumental quantile regression analysis. The Review of Economics and Statistics, 86(3), 735–751. https://doi.org/10.1162/0034653041811734
      • Belloni, A., Chernozhukov, V., & Hansen, C. (2014). Inference on treatment effects after selection among high-dimensional controls. The Review of Economic Studies, 81(2), 608–650. https://doi.org/10.1093/restud/rdt044
      Previous labThe economic cost of conflict in the Basque CountryAll labsReplication labs
      Tools · Your own study

      Plan your study

      Describe your study and how treatment was assigned. The planner suggests a design, draws the causal diagram with your own labels, lists the assumptions you will need to defend and the checks to run, and drafts starter code in R and Python with your variable names. Your answers stay in this browser.

      The planner gives a first draft of a design. A design is only as credible as the argument for its assumptions, and the starter code needs checking against your data. The R code uses fixest, rdrobust, rddensity, did, Synth, MatchIt, sensemakr, and DoubleML. The Python code uses pyfixest, rdrobust, rddensity, csdid, pysyncon, statsmodels, scikit-learn, and DoubleML.

      Resource · R, Python and Stata

      Code library

      Every worked example on the site, by module and level. Each example simulates its own data, so it runs on its own in R, Python or Stata and gives the same results in all three. The difference-in-differences examples share one simulated school panel, built by the shared data generator.

      LearnCausal

      Free causal inference lessons, from foundations to expert level.

      Methods

      • Randomized experiments
      • Matching and weighting
      • Instrumental variables
      • Regression discontinuity
      • Difference-in-differences
      • Synthetic control
      • Machine learning
      • Choosing a design
      • Method summaries

      Tools

      • Plan your study
      • Power calculator
      • Overlap lab
      • Complier explorer
      • Bandwidth explorer
      • Parallel trends explorer
      • Synthetic control explorer
      • Overfitting explorer
      • Replication labs

      Resources

      • Review
      • Find the flaw
      • Design projects
      • Glossary
      • Bibliography
      • Reporting checklists
      • Code library
      • Software packages

      News

      • Article reviews
      • Portal updates

      About

      • About LearnCausal
      • Editorial standards
      • License and reuse
      • Privacy
      © 2026 Bakhyt ZhumatayVersion 1.0.0
        Your progress
        LevelFoundations
        Quiz0 / 6
        Exercises answered0 / 3
        Questions to review0
        Saved in this browser only.
        Module 5 · Foundations

        Compare the change, not the level.

        Difference-in-differences estimates the effect of a program or policy by comparing how much an outcome changed for the group that got it with how much it changed for a group that did not. Start with the slider, then work through the six predictions beside the chart.

        Tutoring program simulator

        Program schools start tutoring in 2024. Comparison schools never do. You control the true effect, how the groups would have trended without the program, and what else happens along the way.

        What DiD assumes What would truly have happened Your previous setting

        0.0
        Points per year, before any program. At 0 the two groups move in parallel.
        Predict, then see
          More controls
          +4.0
          Points of math score the program actually adds in 2024.
          5.0
          How far below the comparison schools the program schools start.
          Each option breaks one of the assumptions listed further down the page.
          True effect+4.0
          DiD estimate+4.0
          Bias0.0

          Three ways to estimate the same effect

          What to notice. The starting gap never changes the DiD estimate, but it ruins the after-only comparison. Pre-existing growth ruins the before-after comparison. DiD survives both, and fails only when the two groups would not have moved in parallel. With noise on, the same break in parallel trends can hide inside ordinary year-to-year variation.

          How this tool computes its numbers
          Data
          Four years, 2021 to 2024, with the program in 2024. Comparison schools average 61 + 2k points in year k (k = 0 for 2021). Without the program, program schools would average 61 − gap + (2 + g)k, where g is the extra yearly trend. The program adds the effect in 2024. Scenarios add half the effect in 2023 (teachers start early), add 3 points to program schools in 2024 (another change at the same time), or lower program schools by 3 points in 2023 (schools chosen after an unusual drop).
          Estimate
          The change from 2023 to 2024 for program schools minus the same change for comparison schools. Without noise it equals the true effect plus whatever breaks parallel trends, for example the effect plus g.
          Noise option
          20 schools per group, each with a school effect (standard deviation 5) and a year-to-year shock (standard deviation 3), from a seeded generator starting at seed 7. The standard error treats schools as independent.
          Checked against known values
          In four settings without noise, the estimate equalled the true effect plus the extra yearly trend every time (4.0, 5.5, 1.0 and 2.0). Checked on 14 November 2026.

          The editorial standards describe how the site is checked.

          Before thisModule 1 · Randomized experimentsCounterfactuals and why random assignment makes groups comparable.
          After thisIntermediate level of this moduleMany groups, many periods, and adoption at different times.
          Learning objectives

          By the end of this module you can

          1. Explain in plain words why a simple before-after comparison can mislead.
          2. Compute a difference-in-differences estimate from four group averages.
          3. State the parallel trends assumption and describe one way it can fail.
          4. Recognize the DiD estimate as the interaction term in a simple regression, in R, Python or Stata.
          5. Report a DiD result in cautious language that names its assumptions.
          Core idea

          Two subtractions

          Suppose a group of schools starts a tutoring program, and their average math score rises from 60 to 66. It is tempting to credit the program with 6 points. But scores may have risen anyway, through a new curriculum, an easier test form, or ordinary year-to-year growth. A before-after comparison cannot separate the program from everything else that changed over the same year.

          Now add a second group of similar schools that did not start the program. Their scores rose from 65 to 67, a change of 2 points. If the program schools would also have gained 2 points without tutoring, the extra gain, 6 minus 2, is 4 points. That second subtraction is the whole design.

          Before (2023)
          After (2024)
          Change
          Program
          60
          66
          +6
          Comparison
          65
          67
          +2
          Difference
          +4

          Written as a formula, with each bar meaning a group average:

          \[\text{DiD} = \big(\bar{Y}_{\text{program, after}} - \bar{Y}_{\text{program, before}}\big) - \big(\bar{Y}_{\text{comparison, after}} - \bar{Y}_{\text{comparison, before}}\big)\]

          The program schools started 5 points lower than the comparison schools. That gap does not matter, because each group is compared only with itself. What matters is whether the two groups would have moved by the same amount. The comparison group's change is our best guess at the program group's missing counterfactual: what would have happened to the program schools without the program.

          This page says program and comparison groups. The code and most readings use the research terms: the treated group and the control group, with treatment meaning the program.

          Because the comparison group stands in only for the program schools, the result is the average effect of the program on the schools that actually received it. Researchers call this the average treatment effect on the treated, or ATT, one of the estimands defined in Module 2, Matching and weighting. It may differ from what the program would do in other schools.

          Identifying assumptions

          What has to be true

          The arithmetic always works. Whether the answer is a causal effect depends on these conditions.

          The key assumption

          Parallel trends

          Without the program, both groups' averages would have changed by the same amount.

          Fails when the program went to schools that were already improving, or already declining, faster than others.

          Timing

          No anticipation

          Schools did not change what they did before the program started in ways that moved the outcome.

          Fails when teachers start preparing for tutoring the spring before it launches.

          Separation

          No spillovers

          The program affects only the program schools.

          Fails when students transfer in from comparison schools, or comparison schools copy the idea.

          Coincidence

          Nothing else changed for one group

          No other policy or event hit one group but not the other at the same time.

          Fails when program schools also got new principals or a new reading curriculum that year.

          Composition

          Stable groups

          The students or schools in each group are comparable before and after.

          Fails when lower-scoring families leave program schools because of the program.

          You can never observe the program schools' path without the program, so parallel trends cannot be tested directly. You can check whether the groups moved together in earlier years, and you can argue from what you know about how the program was assigned. Both are supporting evidence, not proof.

          Worked example · Card and Krueger (1994)

          Did a higher minimum wage cost jobs?

          On April 1, 1992, New Jersey raised its minimum wage from $4.25 to $5.05 an hour. Neighboring Pennsylvania did not. Card and Krueger surveyed 410 fast-food restaurants in both states, in February and March 1992 and again in November and December 1992.

          1. The question

            Standard economic reasoning predicted that a higher minimum wage would reduce employment at low-wage employers such as fast-food restaurants. Looking only at New Jersey before and after cannot answer this, because the regional economy was also changing in 1992.

          2. The four averages

            Average full-time-equivalent (FTE) employment per store, from Table 3 of the article.

            StateBeforeAfterChange
            New Jersey (wage rose)20.4421.03?
            Pennsylvania (no change)23.3321.17?
          3. First subtraction, New Jersey

            21.03 − 20.44 = +0.59 FTE per store

            Employment in New Jersey stores rose slightly.

          4. First subtraction, Pennsylvania

            21.17 − 23.33 = −2.16 FTE per store

            Employment in Pennsylvania stores fell. Under parallel trends, this is roughly what would have happened in New Jersey without the wage increase.

          5. Second subtraction

            0.59 − (−2.16) = 2.75 FTE per store

            The article reports 2.76 (standard error 1.36), computed from unrounded averages. Relative to Pennsylvania, New Jersey employment rose by about 2.8 FTE per store, roughly 13 percent.

          6. Interpreting it cautiously

            The estimate is not consistent with the predicted job losses. Read causally, it says that the wage increase did not reduce fast-food employment in New Jersey during 1992. That reading holds only if Pennsylvania's stores track what New Jersey's stores would have done, if other regional changes affected both states alike, and if the late-1992 survey captured the full response. The standard error is large relative to the estimate, so the data are compatible with a range of effects.

          Your turn. Before the next step, compute the difference-in-differences from the two changes above.

          FTE per store

          Step 1 of 6

          What to notice. New Jersey's line barely moves, but the dashed line, New Jersey shifted by Pennsylvania's change, ends well below it. The estimate is the vertical gap at the right edge, not the slope of New Jersey's own line.

          Reproducible code

          Do it in R

          Two short examples. The first repeats the Card and Krueger arithmetic. The second shows that a regression with an interaction term gives the same number as the two subtractions.

          R needs only base R. Python needs pandas and statsmodels. Both examples were run before publication and the outputs above are copied from those runs. In-browser execution is planned for a later version.

          Common mistakes

          How DiD goes wrong in practice

          • Comparing only the after period.Program schools score 66 and comparison schools 67 after 2024, so the program looks harmful. This ignores that program schools started 5 points behind.
          • Comparing only the program group over time.A 6-point gain includes everything else that changed that year. Without a comparison group there is no estimate of the counterfactual change.
          • Reading the wrong regression coefficient.In math ~ treated + post + treated:post, the coefficient on treated is the starting gap. The DiD estimate is the interaction.
          • Calling parallel trends "tested."Pre-trend plots can raise or lower confidence. They cannot confirm what would have happened after the program began.
          • Choosing the comparison group after seeing the results.Pick and justify the comparison group before estimating effects, and say why it should share the program group's trend.
          • Trusting the default standard errors.When the same schools appear in both years, ordinary standard errors are usually too small. The Intermediate level covers clustered standard errors.
          Spot the flaw

          A district evaluation

          After reading scores dropped sharply in 2022 at 12 elementary schools, the district placed those schools in a new literacy coaching program for 2023. An evaluation compares the 12 schools with 20 other schools in the district from 2022 to 2023. Coached schools gained 9 points and the other schools gained 3, so the report concludes that coaching raised reading scores by 6 points.

          What is the main threat to this conclusion? Decide before you open the answer.

          Show the answer

          Schools entered the program because their scores had just dropped. A sharp one-year drop is often partly temporary, from an unusual cohort, a disrupted year, or chance. Such scores tend to bounce back the next year even without help. The coached schools would likely have gained more than the other schools anyway, so parallel trends is doubtful and the 6-point estimate probably overstates the effect.

          What would help: plot several years before 2022 to see whether the drop was unusual, compare against schools that had similar drops but were not coached, or use a pre-period that ends before the drop.

          Self-check

          Quiz

          Each answer explains why it is right or wrong. Your score is saved in this browser.

          Exercises with solutions

          Practice

          One page

          Cheat sheet

          The estimate

          • Change in program group minus change in comparison group
          • Regression: coefficient on treated × post
          • Under parallel trends, it estimates the effect on the treated group (ATT)

          The assumptions

          • Parallel trends (key, not directly testable)
          • No anticipation
          • No spillovers
          • No coinciding change in one group
          • Stable group composition

          Before you report

          • Plot several pre-periods for both groups
          • Explain why the comparison group shares the trend
          • Name what else changed at the same time
          • Use clustered standard errors (Intermediate)
          • Write "is associated with … assuming parallel trends"
          Annotated reading list

          Where to read more

          • Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.The worked example on this page. Section III.A and Table 3 present the difference-in-differences comparison.
          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.Chapter 8 works through a college financial-aid example in education, first by hand and then by regression, the same two steps used on this page.
          • Huntington-Klein, N. (2022). The effect: An introduction to research design and causality. CRC Press. https://doi.org/10.1201/9781003226055Chapter 18, "Difference-in-Differences." An accessible treatment with diagrams and code in R, Stata and Python.
          • Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. https://doi.org/10.12987/9780300255881Chapter "Difference-in-Differences." Covers the history of the design and discusses Card and Krueger.
          • Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828Section 5.2 presents DiD as a regression and uses the Card and Krueger study. Best read after this level.
          • Wing, C., Simon, K., & Bello-Gomez, R. A. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39, 453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507A practical design checklist. The advice on choosing comparison groups transfers directly to education studies.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008Next levelA map of recent methods. Read its first sections after this module to see where DiD goes next.
          • Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014AdvancedShows what goes wrong when groups adopt a program at different times. The starting point for module 3.2.
          Module 5, Foundations level · Last revised November 2026 · Content checked against the full texts of all cited sources on November 9, 2026. Every work this level draws on is listed above. Card and Krueger figures are taken from Table 3 of the published article and cited there; all other explanations and examples were written for this module.
          Module 5 · Intermediate

          From two subtractions to a regression.

          This level states difference-in-differences formally, estimates it with two-way fixed effects, adds the event-study plot that most published DiD studies report, and shows how covariates change the assumption you rely on. Every example uses one simulated school panel, so you can check each estimate against the true effect.

          Before thisFoundations of Module 5The two subtractions, and the notation for counterfactual outcomes.
          After thisAdvanced of Module 5What changes when schools adopt the program in different years.
          Learning objectives

          By the end of this level you can

          1. Write the ATT and the parallel trends assumption in potential-outcomes notation, and show why the 2×2 comparison identifies the ATT.
          2. Estimate DiD as a two-way fixed effects (TWFE) regression in R, Python or Stata, and cluster standard errors at the right level.
          3. Specify, estimate, and read an event study, including what a pre-trend test can and cannot tell you.
          4. Relax parallel trends to conditional parallel trends with covariates, and recognize a bad control.
          Identification

          What the 2×2 comparison identifies

          Each school i has two potential outcomes in year t. \(Y_{it}(1)\) is its score with the program and \(Y_{it}(0)\) its score without it. \(D_i = 1\) marks program schools. Period 0 is before adoption and period 1 is after. The target is the average treatment effect on the treated:

          \[\text{ATT} = E\big[\,Y_{i1}(1) - Y_{i1}(0) \mid D_i = 1\,\big]\]

          The second term is never observed for program schools. Three assumptions let the comparison group fill the gap.

          A1 · Parallel trends

          \[\begin{aligned}&E[Y_{i1}(0) - Y_{i0}(0) \mid D=1]\\&\quad= E[Y_{i1}(0) - Y_{i0}(0) \mid D=0]\end{aligned}\]

          Without the program, both groups’ average outcomes would have changed by the same amount.

          A2 · No anticipation

          \[Y_{i0}(1) = Y_{i0}(0) \ \text{ for } D=1\]

          The program does not affect outcomes before it starts.

          A3 · Consistency (SUTVA)

          \[Y_{it} = D_{it}Y_{it}(1) + (1 - D_{it})Y_{it}(0)\]

          Observed scores equal the potential outcome for the condition received, with no spillovers between schools.

          Under these assumptions the observed difference in changes equals the ATT:

          1. \(E[\Delta Y \mid D=1] - E[\Delta Y \mid D=0]\)observed changes
          2. \(= E[Y_1(1) - Y_0(0) \mid D=1] - E[Y_1(0) - Y_0(0) \mid D=0]\)A2 and A3
          3. \(= \text{ATT} + \big\{ E[Y_1(0) - Y_0(0) \mid D=1] - E[Y_1(0) - Y_0(0) \mid D=0] \big\}\)add and subtract \(E[Y_1(0) \mid D=1]\)
          4. \(= \text{ATT}\)A1 sets the braces to zero

          The bracketed term is the bias whenever parallel trends fails. Two consequences follow. The assumption concerns averages, not individual schools. It also concerns the scale on which outcomes are measured, so parallel trends in scores does not imply parallel trends in log scores. The Expert level returns to this point.

          Estimation

          From the 2×2 table to TWFE

          With two groups and two periods, the interaction regression reproduces the hand calculation exactly:

          \[Y_{it} = \alpha + \beta D_i + \gamma\,\text{Post}_t + \delta\,(D_i \times \text{Post}_t) + \varepsilon_{it}\]

          With many schools and years, researchers replace the group and period dummies with school and year fixed effects. This is the two-way fixed effects (TWFE) model:

          \[Y_{it} = \alpha_i + \lambda_t + \delta D_{it} + \varepsilon_{it}, \qquad D_{it} = 1 \text{ once school } i \text{ has adopted}\]

          \(\alpha_i\) absorbs every difference between schools that is constant over time, such as neighborhood or baseline achievement. \(\lambda_t\) absorbs shocks common to all schools in a year, such as a new state test. When every program school adopts in the same year, δ compares the average post-adoption gap with the average pre-adoption gap. In the shared dataset the true ATT is 3.30 and TWFE estimates 3.34.

          This works because all program schools share one adoption date and the comparison schools are never treated. When adoption is staggered across years, the same regression can give badly wrong answers. That is the subject of the Advanced level.

          Inference

          Cluster where treatment is assigned

          A school’s scores are correlated from one year to the next, and the program is assigned to whole schools. Default standard errors treat each school-year as independent information, so they are too small. Bertrand et al. (2004) showed that DiD studies ignoring this serial correlation reject true null hypotheses far too often.

          The usual remedy is to cluster standard errors at the level at which treatment is assigned. Cluster by school when schools choose the program, by district when districts do, and by state for state policies. In the shared dataset, which has realistic year-to-year correlation, the default standard error is 0.24 and the school-clustered one is 0.41, about 1.7 times larger.

          Clustered standard errors need many clusters. When there are few clusters in total, Cameron et al. (2008) show that the wild cluster bootstrap can work with as few as five; it is available in Python through pyfixest and in R through packages such as fwildclusterboot. Few treated clusters is a different problem. With only a handful of treated states or districts, even the wild cluster bootstrap can perform poorly, and with one treated cluster it fails (Ferman & Pinto, 2019; Roth et al., 2023). Methods built for that case appear at the Expert level, under few treated units.

          Dynamics and pre-trends

          The event study

          An event study replaces the single treatment dummy with one dummy for each year relative to adoption, leaving out the year just before adoption as the reference:

          \[Y_{it} = \alpha_i + \lambda_t + \sum_{k \neq -1} \beta_k \, \mathbf{1}[t - G_i = k] + \varepsilon_{it}\]

          Each \(\beta_k\) is a DiD comparing year k with the reference year. The lags (k ≥ 0) trace how the effect evolves after adoption. The leads (k < −1) are placebo estimates: if parallel trends held before adoption, they should be near zero. A coefficient that drifts away from zero before adoption is a warning.

          Pre-trend tests are useful but weak evidence. Roth (2022) shows they often lack power to detect violations large enough to matter. He also shows that reporting results only after a passed pre-test can make the remaining bias worse. Try it below.

          Event-study explorer

          Program schools adopt in 2020. The true effects are 0 before adoption and 2, 3, 3.5, 4, 4 afterwards (gray line). Each dot is an estimated coefficient with its 95 percent confidence interval; the horizontal axis counts years since adoption.

          0.0
          A violation of parallel trends that starts before adoption and continues after it.
          0.0
          Points gained in the year before adoption, which is also the reference year.
          More schools means narrower intervals and a more powerful pre-test.
          Try this
          1. Set the extra trend to 0.3 and schools to 20. Draw several samples. How often do the pre-period intervals all include zero, even though the post estimates are biased?

          2. Keep the trend at 0.3 and switch to 200 schools. What happens to the pre-test?

          3. Reset the trend and add 1.5 points of anticipation. Why does every coefficient shift?

          Pre-trend checkPasses
          Average post estimate+3.3
          Bias in the average0.0

          Conditional parallel trends

          When trends depend on who you serve

          Suppose scores grow more slowly in schools that serve more students eligible for free or reduced-price lunch (FRL), and program schools serve more of them. Unconditional parallel trends then fails, even if the program has no special link to growth. A weaker assumption may still hold: parallel trends among schools with the same FRL share.

          Adding FRL share as a regressor does nothing in TWFE, because a time-invariant covariate is absorbed by the school fixed effect. To relax the assumption, let the year effects differ by FRL, for example with i(year, frl) in fixest or pyfixest. In the shared covariate dataset (true ATT = 3.0), the unconditional estimate is 0.70 and the FRL-specific-trends estimate is 3.03. Doubly robust estimators that model the covariate more flexibly appear at the Expert level.

          Use only covariates measured before treatment, or covariates the program cannot affect. Controlling for an outcome of the program, such as teacher turnover after a school-improvement grant, can bias the estimate in either direction.

          Reproducible code

          Estimate it yourself

          All three examples build on make_schools(), a short generator that produces the same data in R, Python and Stata. Run it first, then any example.

          R uses fixest (Bergé). Python uses pyfixest, which mirrors fixest’s syntax and gives identical results. Every example was run before publication, and the outputs shown come from those runs.

          Common mistakes

          What reviewers catch

          • Clustering at the wrong level.If districts chose the program, clustering by school or by student understates uncertainty.
          • Treating insignificant leads as proof.A pre-test that fails to reject is weak evidence when intervals are wide. Report the leads with their intervals, and consider the sensitivity analysis at the Advanced level.
          • Adding a time-invariant covariate to TWFE.The school fixed effect already absorbs it. To relax parallel trends, interact the covariate with time.
          • Controlling for post-treatment variables.Covariates the program can change are outcomes, not controls.
          • Hiding the reference period.State which period is omitted. Anticipation in that period shifts every coefficient.
          • Using TWFE when adoption is staggered.With different adoption years and changing effects, the static and dynamic TWFE estimates can mislead. Use the estimators at the Advanced level.
          Spot the flaw

          A state policy evaluation

          A study evaluates a state-mandated early-literacy policy adopted by 3 states in 2019, using 4 other states as comparisons. It analyzes 2 million student test scores from 2015 to 2023 with a TWFE regression, clusters standard errors by student, and reports a 0.04 SD effect with p < 0.001.

          What is wrong with the inference? Decide before you open the answer.

          Show the answer

          Treatment is assigned at the state level, so the effective number of independent units is 7 states, not 2 million students. Clustering by student ignores the correlation of outcomes within states over time, and the p-value is far too small.

          What would help: cluster at the state level, but recognize that 3 treated states are too few for cluster-robust standard errors or even a wild cluster bootstrap to be reliable. Use inference designed for few treated units, such as the Conley–Taber or Ferman–Pinto procedures or randomization inference (Expert level). Report the result with appropriate caution, and consider whether 3 treated states can support a general conclusion.

          Self-check

          Quiz

          Exercises with solutions

          Practice

          One page

          Cheat sheet

          Identification

          • \(\text{ATT} = E[Y_1(1) - Y_1(0) \mid D=1]\)
          • Parallel trends in untreated outcomes
          • No anticipation; consistency, no spillovers
          • Bias = difference in counterfactual trends

          Estimation

          • R feols(y ~ treated | id + year); Stata reghdfe y treated, absorb(id year)
          • Event study in R i(rel, ref = -1); in Stata shift event time to nonnegative values and set the base, as in ib4.rel5
          • Covariate trends in R i(year, x); in Stata i.year#c.x
          • Single adoption date only; staggered → Advanced

          Inference and reporting

          • Cluster at the level of assignment
          • Few clusters → wild cluster bootstrap
          • Few treated clusters → Conley–Taber, Ferman–Pinto, or randomization inference
          • Show leads and lags with intervals
          • State the reference period
          • Pre-tests support, never establish
          Annotated reading list

          Where to read more

          • Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828Section 5.2 derives DiD as a regression and discusses serial correlation.
          • Huntington-Klein, N. (2022). The effect: An introduction to research design and causality. CRC Press. https://doi.org/10.1201/9781003226055Chapter 18 covers TWFE, event studies, and placebo tests with code in R, Stata and Python.
          • Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. https://doi.org/10.12987/9780300255881The chapter “Difference-in-Differences” discusses inference, event studies, and pre-trends.
          • Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249–275. https://doi.org/10.1162/003355304772839588The classic demonstration that ignoring serial correlation makes DiD standard errors too small.
          • Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3), 414–427. https://doi.org/10.1162/rest.90.3.414The wild cluster bootstrap for designs with few clusters in total.
          • Ferman, B., & Pinto, C. (2019). Inference in differences-in-differences with few treated groups and heteroskedasticity. Review of Economics and Statistics, 101(3), 452–467. https://doi.org/10.1162/rest_a_00759Why standard methods, including the wild cluster bootstrap, break down with few treated clusters, and an alternative.
          • Roth, J. (2022). Pretest with caution: Event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305–322. https://doi.org/10.1257/aeri.20210236Why passing a pre-trend test is weak evidence, and how pre-testing can distort results.
          • Wing, C., Simon, K., & Bello-Gomez, R. A. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39, 453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507A design checklist covering comparison groups, covariates, and event studies.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008Next levelThe survey that frames the Advanced and Expert levels.
          Module 5, Intermediate level · Last revised November 2026 · Content checked against the full texts of all cited sources on November 9, 2026. Every work this level draws on is listed above. Explanations, examples, simulated data, and code were written for this module; no text, figures, or tables are reproduced from the sources apart from short cited figures.
          Module 5 · Advanced

          When schools adopt at different times.

          Most real programs roll out in waves. This level shows why the familiar TWFE regression can badly misstate effects under staggered adoption, then teaches eight estimators built to avoid the problem and a sensitivity analysis for parallel trends. The level ends in a lab where you run every method on the same data.

          True ATT4.57known because the data are simulated
          TWFE estimate2.8039 percent too small
          Robust estimators4.64–4.80Callaway–Sant’Anna, Sun–Abraham, imputation
          Before thisIntermediate of Module 5TWFE, event studies, clustering, and conditional parallel trends.
          After thisExpert of Module 5Covariates, machine learning, and the open questions.
          Learning objectives

          By the end of this level you can

          1. Define group-time average treatment effects and the ways they can be aggregated.
          2. Explain, using the Goodman-Bacon decomposition and negative weights, why TWFE fails when effects vary across cohorts or over time.
          3. Choose among the Callaway–Sant’Anna, Sun–Abraham, imputation, two-stage, de Chaisemartin–D’Haultfœuille, stacked, local projections, and extended TWFE estimators, and say what each assumes.
          4. Run each estimator in R, Python or Stata and compare event-study estimates.
          5. Report how sensitive a conclusion is to violations of parallel trends.
          Setting

          Cohorts and group-time effects

          In the shared staggered dataset, 30 schools start tutoring in 2018, 30 in 2020, 30 in 2022, and 30 never do. Call a school’s adoption year its cohort, \(G_i\), with \(G_i = \infty\) for schools that never adopt. The effects differ by cohort and grow with time. Early adopters gain 4 points in their first year plus 1 point a year after that, while the 2020 and 2022 cohorts gain less.

          The natural building block is the group-time average treatment effect (Callaway & Sant’Anna, 2021):

          \[\text{ATT}(g,t) = E\big[\,Y_t(g) - Y_t(\infty) \mid G = g\,\big], \qquad t \geq g\]

          That is the effect in year t for schools that adopted in year g. One study can contain dozens of these, so researchers aggregate them. The simple aggregate averages over all treated school-years. The dynamic aggregate averages by years since adoption and gives an event study. The group aggregate averages by cohort. Each answers a different question, so state which one you report and why.

          The problem

          Forbidden comparisons and negative weights

          Goodman-Bacon (2021) proved that the TWFE estimate is a weighted average of every possible 2×2 DiD in the data. Some of these compare a cohort with never-treated schools, and some compare earlier with later adopters before the later ones start. A third kind uses schools that adopted earlier as the comparison group for schools that adopt later. Those already-treated schools are still gaining from the program, so their growth is subtracted as if it were an ordinary trend.

          Type of 2×2 comparisonWeightAverage estimate
          Treated vs. never treated0.5453.88
          Earlier vs. later treated (later not yet started)0.2284.32
          Later vs. earlier treated (earlier already treated)0.228−1.30
          TWFE estimate (weighted sum)1.0002.80

          The forbidden comparisons average −1.30 even though every true effect in the data is positive. They pull the TWFE estimate down to 2.80 against a truth of 4.57.

          de Chaisemartin and D’Haultfœuille (2020) describe the same problem differently. TWFE equals a weighted sum of the individual school-year effects, but some weights can be negative. In this dataset 90 of 450 treated school-years get a negative weight. All 90 belong to the 2018 cohort in 2022 to 2024, which are exactly the school-years with the largest effects. With heterogeneous effects, TWFE can even get the sign wrong.

          The problem disappears if effects are the same for every cohort and every year since adoption. It also disappears with a single adoption date and a never-treated comparison group. It grows with wider variation in timing, effects that differ by cohort, and effects that build over time. Baker et al. (2022) use simulations and reexaminations of published studies to show that the problem can matter in finance research.

          The fixes

          Five heterogeneity-robust estimators

          All five avoid forbidden comparisons. They differ in which comparison group they use, which version of parallel trends they need, and how efficiently they use the data.

          Callaway & Sant’Anna (2021)

          Estimates each ATT(g, t) with its own clean 2×2 DiD against never-treated or not-yet-treated schools, using year g − 1 as the baseline, then aggregates. It handles covariates with doubly robust methods.

          R did · Python csdid · Stata csdid

          Sun & Abraham (2021)

          An event-study regression with separate coefficients for every cohort and event time, averaged with cohort-share weights. With never-treated controls, a balanced panel, and no covariates, it matches Callaway–Sant’Anna exactly.

          R fixest::sunab · Python pyfixest (saturated) · Stata eventstudyinteract

          Borusyak, Jaravel & Spiess (2024)

          Imputation. Fits school and year effects using only untreated school-years, predicts each treated school-year’s untreated outcome, and averages the differences. Most efficient when parallel trends holds in every period and the errors have equal variance and no correlation over time (Borusyak et al., 2024). With strongly autocorrelated errors, a g − 1 baseline can be more precise (Dube et al., 2025).

          R didimputation · Stata did_imputation

          Gardner two-stage DiD

          The same idea run as two regressions. The first removes school and year effects estimated on untreated observations, and the second regresses the residualized outcome on treatment. Its point estimates equal the imputation estimator’s (Butts & Gardner, 2022).

          R did2s · Python pyfixest.did2s · Stata did2s

          de Chaisemartin & D’Haultfœuille

          Compares schools that switch into treatment with schools whose treatment has not yet changed, period by period (2020, 2026). It also handles treatments that switch on and off or vary in intensity.

          R DIDmultiplegtDYN · Python py-did-multiplegt-dyn · Stata did_multiplegt_dyn

          EstimatorComparison groupParallel trends neededCovariatesNon-binary treatment
          Callaway–Sant’AnnaNever or not-yet treatedFrom g − 1 onwardYes (DR, IPW, OR)No
          Sun–AbrahamNever or last-treatedAll periodsLimitedNo
          Imputation / two-stageAll untreated observationsAll periodsYes (in the first stage)No
          de Chaisemartin–D’HaultfœuilleNot yet switchedAround each switchYesYes
          Stacked DiDNot treated within the event windowWithin each windowYesNo
          Local projections DiDNot yet treated at each horizonFrom t − 1 onwardYesLimited
          Extended TWFENot-yet or never treatedAll periodsYes (interacted)No

          On the shared dataset the estimates are 4.64 (Callaway–Sant’Anna and Sun–Abraham) and 4.80 (imputation), against a truth of 4.57 and a TWFE estimate of 2.80. The Monte Carlo study at the Expert level shows that these differences between robust estimators are sampling noise, while TWFE’s error is systematic.

          Three more fixes

          Stacked DiD, local projections, and extended TWFE

          These three stay close to familiar regressions, which is much of their appeal. Each avoids forbidden comparisons in its own way.

          Stacked DiD (Cengiz et al., 2019; Wing et al., 2024)

          Builds one clean dataset per cohort, with its treated schools plus schools not treated within an event window. It stacks the datasets and runs one regression with dataset-specific school and year effects. Wing et al. show that unweighted stacking targets an unclear mix of effects and propose weights that recover a well-defined average. Their paper is still a working paper.

          R, Python and Stata: a few lines with fixest, pyfixest or reghdfe

          Local projections DiD (Dube et al., 2025)

          For each horizon h, regresses the change from year t − 1 to year t + h on an indicator for newly treated schools. It keeps only newly treated schools and schools still untreated at t + h. The regression weights adoption years by the variance of treatment, and a reweighted version matches Callaway–Sant’Anna.

          R: fixest · Python: pyfixest.lpdid · Stata: reghdfe in a loop over horizons

          Extended TWFE (Wooldridge, 2025)

          Keeps TWFE but replaces the single treatment dummy with a separate dummy for every treated cohort and year. The saturated regression no longer imposes a common effect. Its estimates equal the imputation estimator’s exactly: 4.804 in the shared dataset. The equality also holds with time-constant covariates, provided they are interacted with the cohort and year dummies in both estimators. Without covariates, Gardner’s two-stage estimator is the same estimator again (Wooldridge, 2025).

          R etwfe · Python: pyfixest with cohort × year terms · Stata: jwdid

          Years since adoptionTruthStackedLP-DiDExtended TWFE
          02.332.132.522.34
          13.002.963.313.08
          23.673.723.983.88
          35.25(outside window)5.555.21
          46.00(outside window)6.346.32

          All three avoid the contamination that affects dynamic TWFE. LP-DiD runs slightly high because its variance weights give the 2018 cohort, which has the largest effects, more weight than its share of treated schools. At adoption it targets 2.52 rather than the cohort-share average of 2.33. It estimates a differently weighted average, not a biased one, and the reweighted version in Dube et al. (2025) targets the cohort-share average. All three now appear in the comparison lab.

          Judgment

          Choosing among them

          No single estimator is best in every design. These questions settle most cases:

          • Do you trust parallel trends in every pre-period, or only near adoption?Imputation uses all untreated periods and is more precise when the stronger assumption holds. Callaway–Sant’Anna with a g − 1 baseline relies on less and is more robust if early periods drift.
          • Do you have never-treated units, and are they comparable?If never-treated schools differ (for example, they never applied), not-yet-treated schools may be the better comparison. Without any never-treated units, use not-yet-treated controls.
          • Do you need covariates?Callaway–Sant’Anna offers doubly robust adjustment for conditional parallel trends.
          • Does treatment switch off, vary in dose, or repeat?Use de Chaisemartin–D’Haultfœuille, which was designed for these cases.
          • Could units anticipate the program?Most packages let you move the baseline earlier, for example anticipation = 1 in did.

          In practice, report your main estimator and show that at least one other robust estimator gives a similar answer. Disagreement between them is informative: it can signal that parallel trends holds in some periods but not others.

          Robustness

          How wrong could parallel trends be?

          Instead of asking whether parallel trends holds, Rambachan and Roth (2023) ask how much it would have to fail to change the conclusion. Their relative-magnitudes approach bounds the post-adoption violation by M̄ times the largest violation seen between consecutive pre-periods. It then reports confidence intervals that remain valid under that bound.

          For the adoption-year effect in the single-adoption dataset (estimate 1.91), the robust interval excludes zero as long as M̄ ≤ 1.5. It first includes zero at M̄ = 2. The conclusion therefore survives a post-adoption violation up to 1.5 times the largest pre-period wobble, but not twice as large. Reporting this breakdown value complements a pre-test, which shows only whether pre-period estimates differ from zero.

          M̄Lower boundUpper boundIncludes zero?
          01.112.70No
          0.50.932.84No
          10.603.14No
          1.50.213.52No
          2−0.243.95Yes
          Reproducible code

          Run every estimator

          The examples use make_schools(“staggered”). Where a method has no maintained implementation in one language, the example says so and shows the closest alternative.

          Package versions used: R 4.5 with fixest 0.14, did 2.5, didimputation 0.5, bacondecomp 0.1, HonestDiD 0.2.8, and DIDmultiplegtDYN 2.2. Python 3.13 with pyfixest 0.70, csdid 0.4.5, and py-did-multiplegt-dyn 0.1.9, which needs polars 1.x (tested with 1.20).

          Common mistakes

          What still goes wrong

          • Reporting static TWFE with staggered adoption.Even when it happens to be close, reviewers will ask for a robust estimator. Report one.
          • Reading a dynamic TWFE event study as clean.Sun and Abraham (2021) show that each lead and lag can be contaminated by effects from other event times. Use an interacted or imputation event study.
          • Mixing aggregation schemes.The simple, dynamic, and group aggregates weight cells differently. Compare like with like across estimators.
          • Ignoring how pre-periods are defined.Callaway–Sant’Anna’s default pre-period estimates are period-to-period changes, unlike the “universal” baseline of regression event studies. They can look different even when nothing is wrong.
          • Running a sensitivity analysis on a contaminated event study.Feed HonestDiD estimates from a robust estimator, or a design with a single adoption date.
          Spot the flaw

          “The effect fades over time”

          A paper studies a principal-mentoring program that districts adopted between 2012 and 2020. A dynamic TWFE event study shows positive effects in the first two years that shrink toward zero by year six. The authors conclude that the program’s benefits fade.

          Why might this conclusion be an artifact? Decide before you open the answer.

          Show the answer

          Late event times are estimated only from the earliest adopters, and dynamic TWFE compares them partly with already-treated districts. If effects grow over time or differ by cohort, contamination from other event times can pull late coefficients down. The apparent fade may come from the estimator rather than from the program. The earliest adopters may also differ from later ones, so late event times describe a different set of districts.

          What would help: re-estimate with Sun–Abraham, Callaway–Sant’Anna, or imputation. Report a balanced event study restricted to cohorts observed at every event time, so changes over time are not changes in composition.

          Self-check

          Quiz

          Exercises with solutions

          Practice

          One page

          Cheat sheet

          Diagnose

          • Staggered timing + heterogeneous effects → TWFE unreliable
          • bacondecomp::bacon() in R or bacondecomp in Stata shows forbidden comparisons
          • Count negative weights

          Estimate

          • R did::att_gt() + aggte(); Stata csdid + estat event
          • R feols(y ~ sunab(g, t) | id + t); Stata eventstudyinteract
          • did_imputation or did2s, with the same names in R and Stata
          • did_multiplegt_dyn, with the same name in R and Stata

          Report

          • Name the aggregation (simple, dynamic, group)
          • Name the comparison group
          • Show a second robust estimator
          • HonestDiD breakdown value M̄
          Annotated reading list

          Where to read more

          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008Start here: a map of every method on this level and how they relate.
          • Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014The decomposition of TWFE into 2×2 comparisons.
          • de Chaisemartin, C., & D'Haultfœuille, X. (2020). Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9), 2964–2996. https://doi.org/10.1257/aer.20181169Negative weights in TWFE and the DIDM estimator.
          • Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001Group-time ATTs, aggregation, and doubly robust estimation.
          • Sun, L., & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006Contamination in dynamic TWFE and the interaction-weighted estimator.
          • Borusyak, K., Jaravel, X., & Spiess, J. (2024). Revisiting event-study designs: Robust and efficient estimation. Review of Economic Studies, 91(6), 3253–3285. https://doi.org/10.1093/restud/rdae007The imputation estimator and its efficiency properties.
          • Butts, K., & Gardner, J. (2022). did2s: Two-stage difference-in-differences. The R Journal, 14(3), 162–173. https://doi.org/10.32614/rj-2022-048The two-stage DiD estimator and its R package.
          • de Chaisemartin, C., & D'Haultfœuille, X. (2026). Difference-in-differences estimators of intertemporal treatment effects. Review of Economics and Statistics, 108(4), 863–880. https://doi.org/10.1162/rest_a_01414Event-study estimators for treatments that change more than once or vary in intensity.
          • Cengiz, D., Dube, A., Lindner, A., & Zipperer, B. (2019). The effect of minimum wages on low-wage jobs. Quarterly Journal of Economics, 134(3), 1405–1454. https://doi.org/10.1093/qje/qjz014The minimum-wage study that popularized stacked event studies.
          • Wing, C., Freedman, S. M., & Hollingsworth, A. (2024). Stacked difference-in-differences (NBER Working Paper No. 32054). National Bureau of Economic Research. https://doi.org/10.3386/w32054Working paperThe weighting that makes stacked DiD target a well-defined average effect.
          • Dube, A., Girardi, D., Jordà, Ò., & Taylor, A. M. (2025). A local projections approach to difference-in-differences. Journal of Applied Econometrics, 40(7), 741–758. https://doi.org/10.1002/jae.70000Local projections DiD and its relation to other robust estimators.
          • Wooldridge, J. M. (2025). Two-way fixed effects, the two-way Mundlak regression, and difference-in-differences estimators. Empirical Economics, 69(5), 2545–2587. https://doi.org/10.1007/s00181-025-02807-zExtended TWFE, the Mundlak regression, and their equivalence to imputation.
          • Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591. https://doi.org/10.1093/restud/rdad018Sensitivity analysis for violations of parallel trends.
          • Baker, A. C., Larcker, D. F., & Wang, C. C. Y. (2022). How much should we trust staggered difference-in-differences estimates? Journal of Financial Economics, 144(2), 370–395. https://doi.org/10.1016/j.jfineco.2022.01.004How much staggered-DiD problems change published conclusions.
          • de Chaisemartin, C., & D'Haultfœuille, X. (2023). Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: A survey. The Econometrics Journal, 26(3), C1–C30. https://doi.org/10.1093/ectj/utac017A survey of the heterogeneous-effects literature by two of its main contributors.
          Module 5, Advanced level · Last revised November 2026 · Content checked against the full texts of all cited sources on November 9, 2026. Every work this level draws on is listed above. Explanations, examples, simulated data, and code were written for this module; no text, figures, or tables are reproduced from the sources apart from short cited figures.
          Module 5 · Expert

          Identification, efficiency, and the open questions.

          This level covers semiparametric and machine-learning DiD under conditional parallel trends, the dependence of parallel trends on how outcomes are measured, changes-in-changes, triple differences, designs with few treated units, design-based inference, continuous treatments, and a Monte Carlo study of the estimators from the Advanced level. It closes with the debates that are still unsettled.

          Before thisAdvanced of Module 5Staggered adoption and the heterogeneity-robust estimators.
          Next moduleSynthetic controlBuilding a comparison for one treated unit. Module 7, Machine learning for causal inference, covers the double machine learning tools used here.
          Learning objectives

          By the end of this level you can

          1. Derive the outcome-regression, inverse-probability-weighting, and doubly robust DiD estimators, and state when each is consistent.
          2. Explain how double/debiased machine learning extends doubly robust DiD.
          3. Determine whether a parallel-trends assumption is sensitive to the outcome’s scale, and estimate effects across the outcome distribution with changes-in-changes.
          4. Analyze designs with few treated units using synthetic DiD, factor models, and matrix completion, with inference suited to few treated units.
          5. Set up a triple-difference design and state its identifying assumption.
          6. Design and interpret a Monte Carlo comparison of DiD estimators.
          7. Summarize the main open debates fairly and take a defensible position in your own work.
          Conditional parallel trends

          Three estimators and one that combines them

          Assume parallel trends holds conditional on pre-treatment covariates X, and that every program school has comparable comparison schools (overlap: \(P(D = 1 \mid X) < 1\)). Write \(\Delta Y\) for a school’s change in outcome, \(p(X) = P(D = 1 \mid X)\) for the propensity score, and \(\mu_0(X) = E[\Delta Y \mid X, D = 0]\) for the comparison schools’ expected change.

          Outcome regression

          \[\tau_{\text{OR}} = E\big[\,\Delta Y - \mu_0(X) \mid D = 1\,\big]\]

          Consistent if the model for \(\mu_0\) is correct.

          Inverse probability weighting (Abadie, 2005)

          \[\tau_{\text{IPW}} = \frac{E\!\left[\dfrac{(D - p(X))\,\Delta Y}{1 - p(X)}\right]}{P(D = 1)}\]

          Consistent if the model for \(p(X)\) is correct.

          Doubly robust (Sant’Anna & Zhao, 2020)

          \[\begin{aligned}\tau_{\text{DR}} = E\big[&\,(w_1 - w_0(X))\\&\times(\Delta Y - \mu_0(X))\,\big]\end{aligned}\]

          \(w_1 = D / E[D]\); \(w_0 \propto p(X)(1 - D)/(1 - p(X))\), normalized to mean one. Consistent if either model is correct, and locally efficient if both are.

          The doubly robust estimator is built on the efficient influence function for the ATT. That gives it two properties beyond double robustness. Errors in the two nuisance models enter only as a product, and the estimator attains the semiparametric efficiency bound when both models are correct. On the covariate dataset (true ATT = 3.0), the no-covariate estimate is 1.75, outcome regression gives 2.92, IPW 2.70, and the doubly robust estimator 3.03.

          Flexible nuisance models

          Double/debiased machine learning for DiD

          The doubly robust score is Neyman-orthogonal: small errors in \(\mu_0\) and \(p(X)\) do not change the estimate to first order. Chernozhukov et al. (2018) show that this property, combined with cross-fitting, lets researchers estimate the nuisance functions with machine learning and still get \(\sqrt{n}\)-consistent, asymptotically normal estimates of the target. Chang (2020) applies this to DiD.

          Machine learning helps when many covariates may shape trends in unknown ways, for example dozens of school characteristics. It does not fix a failure of conditional parallel trends or of overlap. With one covariate, as in the example, a random forest adds noise rather than flexibility. The estimate is 3.47 with a standard error of 1.33, against 3.27 (SE 0.66) for the parametric doubly robust estimator.

          Scale matters

          Parallel in points or parallel in percent?

          Parallel trends in levels and parallel trends in logs generally cannot both hold when the groups start at different levels. Roth and Sant’Anna (2023) show that parallel trends is insensitive to the outcome’s scale only under strong conditions. Roughly, the population must split into one part for which treatment is effectively random and another part whose untreated outcomes have a stable distribution over time. Choosing the scale is therefore part of the identifying assumption, and it deserves a substantive justification.

          Levels versus logs

          The program has no effect. Choose how untreated outcomes grow, then compare the DiD estimate on each scale.

          10
          Percent (or points) between before and after.
          60
          The comparison group starts at 40.
          DiD in levels+2.0
          DiD in logs (≈ %)0.0 percent
          True effect0

          Beyond the mean

          Changes-in-changes

          Athey and Imbens (2006) replace parallel trends with a different assumption. Untreated outcomes are a monotone function of an unobserved school-level characteristic, and the distribution of that characteristic within each group does not change over time. Under this assumption, a program school’s rank in the comparison group’s before-period distribution tells you where it would have landed in the comparison group’s after-period distribution.

          The method maps each program school’s pre-period score to its percentile among comparison schools before the program. It then reads off the comparison group’s score at that percentile after the program, which gives the counterfactual. Averaging gives an ATT, and comparing quantiles gives quantile treatment effects on the treated. Because the assumption concerns ranks, not levels, changes-in-changes gives the same answer for any monotone transformation of the outcome, which sidesteps the scale problem above. It needs continuous outcomes, enough units to estimate distributions, and the absence of the population shifts that would move ranks.

          In the single-adoption dataset, changes-in-changes gives 3.06 for the second program year, against 2.97 from DiD and a truth of 3. The quantile effects (3.73, 3.51, and 3.14 at the 25th, 50th, and 75th percentiles) vary only by sampling noise, because the simulated effect is the same for every school.

          A third difference

          Triple differences

          Suppose a district adopts tutoring for grades 3 to 5 only, and a comparison district does not. A DiD across districts may be biased by district-wide shocks, and a DiD across grades within the program district may be biased by grade-specific trends. The triple difference subtracts the comparison district’s grade gap from the program district’s:

          \[\text{DDD} = \big[\Delta\bar{Y}_{\text{program, grades 3–5}} - \Delta\bar{Y}_{\text{program, other}}\big] - \big[\Delta\bar{Y}_{\text{comparison, grades 3–5}} - \Delta\bar{Y}_{\text{comparison, other}}\big]\]

          Olden and Møen (2022) show that the triple difference does not need two separate parallel-trends assumptions. It needs only one: the bias in the two DiDs is the same. In other words, the gap in trends between target and other grades would have been the same in both districts. This is weaker than either DiD’s assumption alone, but it is still an assumption and needs its own defense.

          Where does randomness come from?

          Design-based inference

          Standard errors in DiD usually assume schools are sampled from a large population. Athey and Imbens (2022) instead treat potential outcomes as fixed and locate the randomness in which schools adopt when. When adoption dates are as good as randomly assigned, they characterize the causal estimand that the standard DiD estimator targets, a weighted average of causal effects. They also show that conventional clustered variance estimators are generally conservative from this perspective. The framework fits education settings where the full population of schools in a state is observed, and where sampling from a larger population is hard to justify.

          Doses, not switches

          Continuous and multi-valued treatments

          Many education programs vary in intensity: hours of tutoring, dollars per pupil, share of teachers trained. Callaway et al. (2024, working paper) show that standard parallel trends identifies the average effect of each dose among units that received it. It does not identify how outcomes change as the dose changes. Comparing doses requires a stronger assumption that rules out selection into doses based on gains. TWFE regressions on a continuous treatment mix these quantities with weights that can be hard to interpret. The paper is not yet peer reviewed, so treat its specifics as provisional. For non-binary treatments that change over time, the de Chaisemartin–D’Haultfœuille estimators from the Advanced level also apply.

          One district, one state

          When only a few units are treated

          Education policy often changes in one state or a handful of districts. Two things go wrong at once. The comparison group’s average may not resemble the few treated units, and standard errors that rely on many treated clusters stop working.

          The few-treated dataset has 6 program schools that adopt in 2021 and 114 comparison schools. A hidden factor, such as enrollment growth, shifts schools’ paths over time, and the program schools are more exposed to it. Parallel trends therefore fails for the average comparison school, but some comparison schools are as exposed as the program schools. The true effect is 3.

          Synthetic DiD (Arkhangelsky et al., 2021)

          Chooses weights for comparison schools so their weighted pre-period path parallels the program schools’. It also weights pre-periods so they resemble the post-period, then runs a weighted DiD. It nests DiD (equal weights) and synthetic control (no time weights, levels matched).

          R synthdid · Python causaltensor · Stata sdid

          Factor models and matrix completion

          Model untreated outcomes as school effects, year effects, and a low-rank interaction, then predict the missing untreated outcomes of treated school-years. Interactive fixed effects fix the number of factors. Matrix completion (Athey et al., 2021) instead penalizes the nuclear norm and chooses the penalty by cross-validation.

          R fect · Python causaltensor · Stata fect

          Inference with few treated units

          Cluster-robust standard errors need many treated clusters. Conley and Taber (2011) use the comparison units’ residuals to build the distribution of the estimator under the null. Ferman and Pinto (2019) extend this to heteroskedastic errors, the leading case being treated and comparison clusters of different sizes. Synthetic DiD’s placebo method reassigns treatment to comparison units many times.

          Placebo loops in R, Python and Stata (Example 4)

          EstimatorEstimate (R)Estimate (Python)
          DiD6.756.75
          Synthetic control4.14see note below
          Synthetic DiD4.254.24
          Interactive fixed effects (one factor)4.25no maintained package
          Matrix completion (cross-validated)6.405.50
          Truth3.003.00

          Synthetic DiD and the one-factor model remove most of DiD’s bias, and the placebo 95 percent interval for synthetic DiD (2.91 to 5.58) contains the truth. Matrix completion removes little here. With only 6 pre-periods and 6 treated schools, cross-validation chose heavy regularization, and the two packages’ different tuning grids give different answers. None of these methods is a free lunch. They rely on a stable factor structure, enough pre-periods to learn it, and comparison units that span the treated units’ exposure. Module 6 develops synthetic control in full. In Python, the pysyncon package fits the classic synthetic control estimator, which differs from the regularized variant in R’s synthdid.

          Simulation evidence

          How the estimators behave across 200 samples

          A single dataset can flatter or embarrass any estimator. Example 3 below draws 200 staggered datasets with the same design and effects but different random noise, and runs three estimators on each. Bias is the average error, RMSE the root mean squared error, and coverage the share of 95 percent confidence intervals that contain the truth.

          EstimatorBiasRMSE95 percent CI coverage
          TWFE−1.771.800.00
          Callaway–Sant’Anna0.080.460.96
          Imputation (BJS, R)0.050.450.92
          Two-stage (Gardner, Python)0.050.450.96

          TWFE is wrong in every one of the 200 samples. Its intervals are centered on the wrong value, so collecting more data makes it more confidently wrong. The robust estimators are approximately unbiased with similar precision. The imputation and two-stage estimators give identical point estimates but use different standard errors, which explains their different coverage. Simulation results hold for the design simulated, so build a simulation that resembles your own data before trusting any estimator’s small-sample behavior.

          Unsettled questions

          Where researchers still disagree

          • Efficiency versus robustness.Imputation uses every pre-period and is the most efficient estimator if parallel trends holds throughout and the errors are homoskedastic and serially uncorrelated (Borusyak et al., 2024). Callaway–Sant’Anna with a g − 1 baseline assumes less. It usually gives up precision, but with strongly autocorrelated errors the g − 1 baseline can be the more precise choice (Dube et al., 2025). Which trade-off is right depends on how credible early pre-periods are, which is a judgment about the setting.
          • Testing versus bounding parallel trends.Pre-trend tests remain standard, but Roth (2022) shows they can mislead, and Rambachan and Roth (2023) argue for reporting sensitivity instead. Choosing the restriction on violations, such as M̄, still involves judgment.
          • Which scale.Roth and Sant’Anna (2023) show the choice between levels and logs is substantive. Applied work rarely justifies it.
          • Which estimand.Simple, dynamic, and group aggregates answer different policy questions. There is no consensus on a default, and papers often report whichever their software prints.
          • Inference with few treated clusters.Wild cluster bootstrap, randomization inference, and design-based approaches can disagree when only a few states or districts are treated, and the wild cluster bootstrap can perform poorly in that setting (Roth et al., 2023).
          • How much the new estimators change conclusions.Some reanalyses find large changes (Baker et al., 2022). Others find modest ones when effects are fairly homogeneous. The size of the problem depends on the setting, so check it rather than assume it.
          Reproducible code

          Estimators and simulation

          Example 3 takes about 30 to 40 seconds in R and in Python, and Example 4 about a minute in R. Example 4 in Python uses causaltensor, which needs NumPy 1.x, so install it in a separate environment.

          R’s DoubleML package (version 1.0.2) has no DiD model yet, so the R version of Example 2 uses the parametric doubly robust estimator from DRDID. Python uses doubleml 0.11.

          Self-check

          Quiz

          Exercises with solutions

          Practice

          One page

          Cheat sheet

          Conditional parallel trends

          • OR: model \(\mu_0(X)\)
          • IPW: model p(X)
          • DR: either suffices; efficient if both
          • DML: ML nuisances + cross-fitting
          • Needs overlap and pre-treatment X

          Design choices

          • Justify the outcome’s scale
          • DDD: equal bias across two DiDs
          • Few treated units → synthetic DiD
          • Doses → strong parallel trends

          Evidence standards

          • Simulate a design like yours
          • Report bias, RMSE, coverage
          • Report sensitivity, not only pre-tests
          • Name the estimand and aggregation
          Annotated reading list

          Where to read more

          • Sant'Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101–122. https://doi.org/10.1016/j.jeconom.2020.06.003The doubly robust DiD estimators and their efficiency properties.
          • Abadie, A. (2005). Semiparametric difference-in-differences estimators. Review of Economic Studies, 72(1), 1–19. https://doi.org/10.1111/0034-6527.00321The semiparametric IPW DiD estimator.
          • Chang, N.-C. (2020). Double/debiased machine learning for difference-in-differences models. The Econometrics Journal, 23(2), 177–191. https://doi.org/10.1093/ectj/utaa001Double/debiased machine learning for DiD.
          • Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097The general theory of orthogonal scores and cross-fitting.
          • Roth, J., & Sant'Anna, P. H. C. (2023). When is parallel trends sensitive to functional form? Econometrica, 91(2), 737–747. https://doi.org/10.3982/ecta19402When parallel trends depends on the outcome’s scale.
          • Olden, A., & Møen, J. (2022). The triple difference estimator. The Econometrics Journal, 25(3), 531–553. https://doi.org/10.1093/ectj/utac010The identifying assumption behind triple differences.
          • Athey, S., & Imbens, G. W. (2022). Design-based analysis in difference-in-differences settings with staggered adoption. Journal of Econometrics, 226(1), 62–79. https://doi.org/10.1016/j.jeconom.2020.10.012Design-based estimands and variances under staggered adoption.
          • Callaway, B., Goodman-Bacon, A., & Sant'Anna, P. H. C. (2024). Difference-in-differences with a continuous treatment (NBER Working Paper No. 32117). National Bureau of Economic Research. https://doi.org/10.3386/w32117Working paperDiD with continuous treatments and the strong parallel trends assumption. A revised draft (December 2025) remains unpublished.
          • Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118. https://doi.org/10.1257/aer.20190159Synthetic difference-in-differences.
          • Athey, S., Bayati, M., Doudchenko, N., Imbens, G., & Khosravi, K. (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association, 116(536), 1716–1730. https://doi.org/10.1080/01621459.2021.1891924Matrix completion for causal panel data.
          • Athey, S., & Imbens, G. W. (2006). Identification and inference in nonlinear difference-in-differences models. Econometrica, 74(2), 431–497. https://doi.org/10.1111/j.1468-0262.2006.00668.xChanges-in-changes and nonlinear DiD.
          • Conley, T. G., & Taber, C. R. (2011). Inference with “difference in differences” with a small number of policy changes. Review of Economics and Statistics, 93(1), 113–125. https://doi.org/10.1162/rest_a_00049Inference with a small number of policy changes.
          • Ferman, B., & Pinto, C. (2019). Inference in differences-in-differences with few treated groups and heteroskedasticity. Review of Economics and Statistics, 101(3), 452–467. https://doi.org/10.1162/rest_a_00759Inference with few treated groups under heteroskedasticity, such as unequal cluster sizes.
          • Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591. https://doi.org/10.1093/restud/rdad018Sensitivity analysis as an alternative to pre-testing.
          • Borusyak, K., Jaravel, X., & Spiess, J. (2024). Revisiting event-study designs: Robust and efficient estimation. Review of Economic Studies, 91(6), 3253–3285. https://doi.org/10.1093/restud/rdae007The efficiency result behind the imputation estimator, and its conditions.
          • Dube, A., Girardi, D., Jordà, Ò., & Taylor, A. M. (2025). A local projections approach to difference-in-differences. Journal of Applied Econometrics, 40(7), 741–758. https://doi.org/10.1002/jae.70000How the choice of baseline period affects precision when errors are autocorrelated.
          • Baker, A. C., Larcker, D. F., & Wang, C. C. Y. (2022). How much should we trust staggered difference-in-differences estimates? Journal of Financial Economics, 144(2), 370–395. https://doi.org/10.1016/j.jfineco.2022.01.004Reanalyses showing how much staggered-DiD problems can change published results.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008The survey that ties these threads together.
          Module 5, Expert level · Last revised November 2026 · Content checked against the full texts of all cited sources on November 9, 2026. Every work this level draws on is listed above. Explanations, examples, simulated data, and code were written for this module; no text, figures, or tables are reproduced from the sources apart from short cited figures.
          Capstone · Advanced and Expert

          Estimator comparison lab

          Every estimator from this module, run live on the same simulated schools. Thirty schools adopt in each of 2018, 2020, and 2022, and the rest never do. Change how the effects behave and how the data were generated, then see which estimators recover the truth.

          Lab controls and results

          Scenarios
          2.0
          1.0
          The 2020 cohort gains this much more than 2022, and 2018 twice as much more.
          0.5
          0.0
          Breaks parallel trends for every estimator.
          0.0

          Overall effect on treated school-years

          Event study: effect by years since adoption

          TWFE negative weights0
          True overall effect0average over treated school-years
          Largest robust-estimator error0Callaway–Sant'Anna, Sun–Abraham, imputation

          Reading the lab. Callaway–Sant'Anna with never-treated controls and Sun–Abraham give identical answers here, because the panel is balanced and there are no covariates. The de Chaisemartin–D'Haultfœuille event study equals Callaway–Sant'Anna with not-yet-treated controls for a binary treatment that stays on. Imputation, two-stage DiD and extended TWFE share point estimates. Stacked DiD covers its event window (−3 to 2), and LP-DiD weights adoption years by treatment variance, so both can differ slightly from the cohort-share average. Every estimator here was checked against the R packages on the shared dataset before publication.

          Capstone

          Estimator comparison lab

          At the Advanced and Expert levels, this module ends with a lab that runs TWFE, Callaway–Sant'Anna, Sun–Abraham, imputation, two-stage DiD, extended TWFE, stacked DiD, local projections DiD, and de Chaisemartin–D'Haultfœuille on the same data, side by side.

          Learn · Module 5 · Foundations · Lesson F1

          The idea behind difference-in-differences.

          This lesson is for readers meeting difference-in-differences for the first time. It explains, without heavy notation, what question the method answers, why simple comparisons mislead, how two subtractions fix them, what has to be true for the answer to be believable, and how to read a published study that uses it.

          Before thisThe simulator at the top of this levelMove the slider and watch the two subtractions at work.
          After thisIntermediate: identification and the TWFE regressionThe same ideas stated formally.
          By the end

          What you will be able to do

          1. Explain what a counterfactual is and why every causal question needs one.
          2. Say why before-after and after-only comparisons can mislead.
          3. Compute a DiD estimate from four averages.
          4. State parallel trends in plain words and give examples of how it can fail in schools.
          5. Read a published DiD study and identify its comparison group, its assumption, and its main threats.
          The question

          What would have happened anyway?

          A principal starts an after-school tutoring program, and by the end of the year her students’ math scores are 6 points higher than the year before. Did the program work? The answer is that we cannot tell yet. Scores might have risen anyway. The state may have adopted an easier test, a new curriculum may have arrived, or this year’s students may simply have been stronger.

          To say the program caused a gain, we need to know what scores would have been without the program in the same year. Researchers call that the counterfactual, the outcome in the world that did not happen. The effect of the program is the difference between what happened and what would have happened. The trouble is that the counterfactual is never observed. The school ran the program, so we will never see its scores in the same year without it.

          Every method for causal inference is a strategy for estimating that missing counterfactual. Randomized experiments do it by assigning the program by chance, so that the untreated group shows what would have happened. Difference-in-differences, often shortened to DiD, does it with a comparison group and the passage of time. Murnane and Willett (2011) describe such designs as natural experiments, situations in which a policy change creates groups exposed to different treatments without a researcher assigning them.

          Why simple comparisons fail

          Two tempting comparisons

          Suppose the program schools averaged 60 points before the program and 66 after. A group of similar schools without the program averaged 65 before and 67 after. Two comparisons come to mind.

          • Before and after, in program schools only. Scores rose by 6 points. But the comparison schools rose by 2 points without any program, so at least some of the 6 points would probably have happened anyway. This comparison confuses the program with everything else that changed over time.
          • Program against comparison schools, after the program only. Program schools score 66 and comparison schools 67, so the program looks harmful. But program schools started 5 points behind. This comparison confuses the program with differences that existed before it began.

          Each comparison removes one source of confusion and keeps the other. Before-after removes fixed differences between schools but keeps changes over time. After-only removes changes over time but keeps differences between schools. Difference-in-differences removes both.

          The method

          Two subtractions

          The first subtraction takes each group’s change over time. Program schools changed by 66 − 60 = 6 points. Comparison schools changed by 67 − 65 = 2 points. The second subtraction takes the difference between those changes, 6 − 2 = 4 points. That is the DiD estimate.

          GroupBeforeAfterChange
          Program schools6066+6
          Comparison schools6567+2
          Difference in changes+4

          The logic is easiest to see as a picture. Draw the comparison schools’ line from 65 to 67. Now draw a dashed line for the program schools that starts at their own value, 60, and rises by the same 2 points, to 62. That dashed line is the estimated counterfactual, where program schools would have been without the program. They actually reached 66, so the estimated effect is 66 − 62 = 4 points.

          Notice what the method does not require. The two groups do not need to start at the same level. Program schools began 5 points lower, and that gap is removed by the first subtraction. Schools did not need to be assigned to the program at random either. What the method does require is that the comparison group’s change is a good stand-in for the change the program group would have experienced. The simulator at the top of this level lets you see what happens when that is not true.

          The method has a long history. Goodman-Bacon (2021) notes that it dates back to John Snow’s analysis of a London cholera outbreak in 1855.

          In the vocabulary of Module 1, the estimand here is the average effect on the schools that adopted the program, measured as a change over time. The next section lists what has to be true for the data to identify it.

          Assumptions

          What has to be true

          The arithmetic always produces a number. Whether that number is the program’s effect depends on four conditions.

          • Parallel trends. Without the program, both groups’ averages would have changed by the same amount. The whole estimate rests on this assumption. It fails if the program went to schools that were already improving faster, or that had just had an unusually bad year and would have bounced back anyway.
          • No anticipation. Schools did not change what they were doing before the program started in ways that moved scores. It fails if teachers began preparing for tutoring in the spring before it launched, so that the “before” scores already include part of the effect.
          • No spillovers. The program affects only the program schools. It fails if students move from comparison schools to program schools to get tutoring, or if comparison schools copy the idea.
          • Nothing else changed for one group only. No other policy or event hit one group but not the other at the same time. It fails if program schools also received new principals or a new curriculum that year.

          Parallel trends concerns what would have happened, so it can never be checked directly after the program starts. Researchers look for indirect evidence. Did the two groups move together in the years before the program? Is there a reason, rooted in how the program was assigned, to expect them to keep moving together? Kahn-Lang and Lang (2020) argue that any DiD study should explain why the groups differed in level before the program, and why the same reason would not make their trends differ. Parallel trends before the program is encouraging, but it does not prove they would have stayed parallel.

          Worked example · published study

          Did a higher minimum wage cost jobs?

          The most famous DiD study is Card and Krueger (1994). In April 1992, New Jersey raised its minimum wage from $4.25 to $5.05 an hour. Neighboring Pennsylvania did not. Standard reasoning predicted that fast-food restaurants in New Jersey would employ fewer workers. Looking at New Jersey alone could not answer the question, because the regional economy was also changing in 1992.

          Card and Krueger surveyed 410 fast-food restaurants in both states, shortly before and several months after the increase. Average full-time-equivalent employment per store in New Jersey went from 20.44 to 21.03, a change of +0.59. In Pennsylvania it went from 23.33 to 21.17, a change of −2.16. The difference in changes is +2.75 workers per store from these rounded averages. The article reports 2.76, which is about 13 percent.

          Read the result through the lens of the assumptions. Pennsylvania is the comparison group, and the estimate is credible only if New Jersey’s restaurants would have followed Pennsylvania’s path without the wage increase. The two states share a regional economy, which supports this. The surprising result, employment rising rather than falling relative to Pennsylvania, made the study famous and much debated, which is a reminder that a DiD estimate is only as strong as the case for its comparison group.

          Run it yourself in R: code and output
          Example · education

          Does student aid get students to college?

          From 1965 to 1982, Social Security paid college benefits to the children of deceased, disabled, or retired beneficiaries. In 1981, Congress voted to end the program. Dynarski (2003) used this change to study whether financial aid affects college attendance.

          The program group is high school seniors whose fathers had died, because they were eligible for the benefit. Among seniors in 1979 to 1981, who could still receive it, 56.0 percent attended college by age 23. Among seniors in 1982 and 1983, after the benefit ended, 35.2 percent did. That 20.8-point drop is the first difference. But college attendance might have been falling for everyone. The comparison group is seniors whose fathers were alive and who were never eligible. Their attendance fell from 50.2 percent to 47.6 percent, a drop of 2.6 points. The difference in differences is 18.2 percentage points. Ending the aid appears to have reduced college attendance among eligible students by that much, which also means the aid had been raising it.

          This example shows two things. A DiD can study a program that is taken away as well as one that is introduced. And the comparison group does not have to be a different place. Here it is a different kind of student observed in the same years.

          Practice

          How to read a DiD study

          1. What changed, for whom, and when? Identify the program or policy, the group it reached, and the date it started.
          2. Who is the comparison group, and why? Look for a reason the comparison group’s change should mirror the program group’s, not just a statement that it does.
          3. What do the years before show? Look for a graph of both groups over several years before the change. Similar movement is reassuring but not proof.
          4. What else happened? Ask whether another event could have affected only one group at the same time.
          5. How precise is the estimate? Check the standard error or confidence interval. The Intermediate level explains why standard errors in DiD need special care.
          6. Whose effect is it? A DiD estimates the effect for the group that received the program. It may differ for other groups.
          Cautions

          Common misreadings

          • “The groups were similar, so the result is causal.”Similar levels are not required, and similar trends before the program do not guarantee similar trends after it.
          • “The pre-trends test passed.”A test that fails to find a difference may simply lack the data to find one. The Intermediate level returns to this.
          • “The regression controls for everything.”A regression with many controls still rests on parallel trends. Controls can even introduce bias if they are affected by the program.
          • “The effect applies everywhere.”The estimate is for the schools, students, and years studied.
          • “Bigger samples fix the problem.”More data make the estimate more precise, not less biased. If parallel trends fails, a large sample gives a precise wrong answer.
          Next steps

          Where the method goes from here

          The two subtractions are the core of every DiD study, but real studies add layers. The Intermediate level writes the method as a regression with many schools and years, explains how to measure uncertainty when scores are correlated over time, shows how to use the years before a program to check the comparison group, and covers what to do when trends differ by student population. The Advanced level handles programs adopted in different years, which is the usual case in education and which breaks the familiar regression. The Expert level covers flexible methods, the scale of the outcome, and settings with only one or a few treated schools or states.

          The shared simulated school data used throughout the module make it possible to compare every estimate with the true effect, something no real study can do. That is the best way to build intuition about when each method works.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.Natural experiments and DiD, written for education researchers. Start here.
          • Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.The minimum-wage study in this lesson.
          • Dynarski, S. M. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279–288. https://doi.org/10.1257/000282803321455287The student-aid study in this lesson.
          • Huntington-Klein, N. (2022). The effect: An introduction to research design and causality. CRC Press. https://doi.org/10.1201/9781003226055A friendly textbook chapter on DiD.
          • Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. https://doi.org/10.12987/9780300255881Another accessible textbook treatment, with code.
          • Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014Notes the method’s origin in Snow’s 1855 cholera study. The rest is Advanced material.
          • Kahn-Lang, A., & Lang, K. (2020). The promise and pitfalls of differences-in-differences: Reflections on 16 and Pregnant and other applications. Journal of Business & Economic Statistics, 38(3), 613–620. https://doi.org/10.1080/07350015.2018.1546591Why a DiD study must explain its comparison group.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Intermediate · Lesson I1

          From two subtractions to a regression.

          This lesson states difference-in-differences formally. It defines the effect being estimated, derives exactly when the comparison group recovers it, shows what goes wrong otherwise, and connects the hand calculation to the two-way fixed effects regression used in most published studies.

          Before thisFoundations: the idea and the assumptionsThe two subtractions and why the comparison group matters.
          After thisIntermediate: standard errors and clusteringHow precise the estimate is, and why default standard errors mislead.
          By the end

          What you will be able to do

          1. Define the ATT with potential outcomes and explain which part of it is never observed.
          2. State parallel trends, no anticipation, and consistency, and derive the DiD result from them.
          3. Write the bias of DiD when parallel trends fails, and recognize its sources in education settings.
          4. Show that the interaction regression and TWFE reproduce the hand calculation when adoption happens at one date.
          5. Distinguish panel data from repeated cross-sections, and say what each requires.
          Question

          Why two simple comparisons fail

          In the shared dataset, 60 schools start a tutoring program in 2020 and 60 never do. Scores are observed from 2015 to 2024. The program schools averaged 49.36 points before 2020 and 56.99 after. The comparison schools averaged 50.41 and 54.69. The true average effect in program schools after adoption is 3.30 points.

          Two comparisons come to mind first. The before-after comparison looks only at program schools and attributes their 7.63-point gain to the program. It ignores that scores rose everywhere. The comparison schools gained 4.28 points without any program. The after-only comparison looks only at 2020 to 2024 and attributes the 2.30-point gap between the groups to the program. It ignores that program schools started about a point lower.

          Each comparison fails for a different reason. Before-after confuses the effect with change over time. After-only confuses it with differences between groups. Difference-in-differences subtracts both. It takes the program schools’ gain and removes the gain the comparison schools made over the same years, which gives 3.34 when computed from unrounded means, close to the truth. The rest of the lesson makes precise why this works and when it does not.

          Estimand

          The effect we want, in potential outcomes

          Each school \(i\) has two potential outcomes in period \(t\). \(Y_{it}(1)\) is its score if it runs the program, and \(Y_{it}(0)\) is its score if it does not. \(D_i = 1\) marks program schools. In the two-period version, period 0 is before adoption and period 1 is after. The target is the average treatment effect on the treated (ATT):

          \[\text{ATT} = E\bigl[\,Y_{i1}(1) - Y_{i1}(0) \mid D_i = 1\,\bigr]\]

          The first term is observed. It is the program schools’ average score after adoption. The second term is not. It is the score program schools would have had in the same years without the program, and no data set records it. Every causal design is a strategy for standing in for that missing counterfactual. DiD builds it from two observed quantities, the program schools’ own starting level and the comparison schools’ change.

          The estimand is the effect on the treated. DiD says how much the program changed outcomes in the schools that adopted it. It does not, without further assumptions, say what the program would do in schools that did not adopt it, which may differ in leadership, students, or readiness.

          Identification

          Three assumptions and a four-line proof

          A1 · Parallel trends

          \[\begin{aligned}&E[Y_{i1}(0) - Y_{i0}(0) \mid D=1]\\&\quad= E[Y_{i1}(0) - Y_{i0}(0) \mid D=0]\end{aligned}\]

          Without the program, both groups’ average outcomes would have changed by the same amount.

          A2 · No anticipation

          \[Y_{i0}(1) = Y_{i0}(0) \ \text{ for } D=1\]

          The program does not affect outcomes before it starts.

          A3 · Consistency (SUTVA)

          \[Y_{it} = D_{it}Y_{it}(1) + (1 - D_{it})Y_{it}(0)\]

          Observed scores equal the potential outcome for the condition received, with no spillovers between schools.

          Under these assumptions the observed difference in changes equals the ATT.

          1. \(E[\Delta Y \mid D=1] - E[\Delta Y \mid D=0]\)observed changes
          2. \(= E[Y_1(1) - Y_0(0) \mid D=1] - E[Y_1(0) - Y_0(0) \mid D=0]\)A2 and A3
          3. \(= \text{ATT} + \bigl\{ E[Y_1(0) - Y_0(0) \mid D=1] - E[Y_1(0) - Y_0(0) \mid D=0] \bigr\}\)add and subtract \(E[Y_1(0) \mid D=1]\)
          4. \(= \text{ATT}\)A1 sets the braces to zero

          Line 3 is the most useful line in the derivation. It holds whether or not parallel trends is true, so it says exactly what DiD estimates in general, the ATT plus the difference between the two groups’ untreated trends. The term in braces is the bias. Everything else in this module, from event studies to sensitivity analysis, is a way of arguing about that term.

          Notice what parallel trends does not require. The groups may differ in level, as they do here. Treatment need not be assigned at random. The assumption concerns the change in averages, not individual schools, and only the untreated outcome. It is also untestable in the post period, because \(Y_1(0)\) is never observed for program schools.

          Bias

          What breaks each assumption in schools

          A simulation shows the decomposition directly, because it records the score each school would have had without the program. The second script below computes the DiD estimate, the true ATT, and the trend gap, which is the DiD computed on the untreated outcomes. As simulated, the estimate is 3.34, the ATT 3.30, and the trend gap 0.04, a small sampling difference. If program schools had been improving 0.3 points a year faster even without the program, the estimate rises to 4.84 and the trend gap to 1.54, and the decomposition is exact. If program schools gain one point in 2019 in anticipation, the estimate falls to 3.14. The trend gap is unchanged, and the −0.20 left over is the anticipation bias, because part of the effect has moved into the “before” period.

          Each assumption has familiar threats in education research.

          • Parallel trends. Schools that adopt a program are often those already improving, or those whose scores just fell. A principal who applies for a grant after a bad year may see scores rebound regardless, a pattern known as regression to the mean. Differences in student composition, such as FRL share, can also produce different trends. The covariates lesson returns to this case.
          • No anticipation. Schools announced as program sites may change staffing or test preparation before the program formally starts. Then the “before” period is already partly treated.
          • Consistency and no spillovers. If the program draws students or teachers away from nearby comparison schools, the comparison group is no longer untreated. Parallel trends in scores also does not imply parallel trends in log scores. Roth and Sant’Anna (2023) show that parallel trends holds for every monotonic transformation of the outcome only under strong conditions, so the choice of scale is part of the assumption. The Expert level returns to this point.
          Design

          Choosing a comparison group

          Parallel trends is a claim about a particular comparison group, so the choice of that group is the main design decision in a DiD study. Statistical tools come later and cannot repair a poor choice. A good comparison group is exposed to the same influences on the outcome as the program group, except the program, over the same years.

          Kahn-Lang and Lang (2020) give three recommendations that apply directly to education research. First, any DiD study should explain why the groups differed in level before the program. A difference in levels means that something already separates the groups, and the researcher should argue that the same mechanism would not also separate their trends. If the researcher claims the groups did not differ before the program, that too must be shown. Second, the choice of scale for the outcome is part of the assumption and needs a substantive justification, not only a statistical one. Third, parallel trends before adoption suggests, but cannot establish, parallel trends afterwards. It is neither necessary nor sufficient.

          In practice, several sources of comparison groups are common in education studies.

          • Schools or districts that did not adopt. The most common choice, and the most exposed to selection, because adoption is often a choice made for reasons linked to recent performance.
          • Units not yet treated. Schools that adopt later are often more similar to early adopters than schools that never adopt. Staggered designs rely on them, as the Advanced level explains.
          • Groups defined by eligibility rules. The student-aid example later in this lesson compares students who were eligible with students who never could be, across the same cohorts.
          • Groups defined by policy history. Dee and Jacob (2011), discussed in the event-study lesson, compare states with and without earlier accountability systems, a contrast justified by history rather than chosen to fit the data.

          Whatever the choice, state it before seeing the post-period results, explain why the groups’ trends should match, and report how the conclusion changes with a reasonable alternative group.

          Estimation

          From the 2×2 table to TWFE

          With two groups and two periods, the interaction regression reproduces the hand calculation exactly.

          \[Y_{it} = \alpha + \beta D_i + \gamma\,\text{Post}_t + \delta\,(D_i \times \text{Post}_t) + \varepsilon_{it}\]

          Each coefficient is one cell of the table. \(\alpha\) is the comparison mean before, \(\beta\) the starting gap between groups, \(\gamma\) the comparison group’s change, and \(\delta\) the difference in changes. The regression adds no assumptions to the subtraction. Its value is that it delivers a standard error and accepts extra terms.

          With many schools and years, researchers replace the group and period dummies with school and year fixed effects. This is the two-way fixed effects (TWFE) model.

          \[Y_{it} = \alpha_i + \lambda_t + \delta D_{it} + \varepsilon_{it}, \qquad D_{it} = 1 \text{ once school } i \text{ has adopted}\]

          \(\alpha_i\) absorbs every difference between schools that is constant over time, such as neighborhood or baseline achievement. \(\lambda_t\) absorbs shocks common to all schools in a year, such as a new state test. The coefficient \(\delta\) can be computed in two steps. Remove each school’s mean and each year’s mean from both the outcome and the treatment indicator, then regress one on the other. This is the Frisch–Waugh–Lovell logic behind fixed effects, and it is how the next lesson builds cluster-robust standard errors.

          The first script computes six numbers. Before-after gives 7.63 and after-only 2.30. The DiD of four means, the interaction regression, TWFE on all ten years, and TWFE by hand all give 3.34. With one adoption date, never-treated comparison schools, and a balanced panel, TWFE is the DiD of before and after averages.

          Wooldridge (2025) shows that the same TWFE estimate can also be obtained by pooled OLS that adds each unit’s time average and each period’s cross-sectional average as regressors, which he calls the two-way Mundlak regression. Extending that regression with cohort-by-period interactions allows effects to differ across adoption cohorts and years, which matters at the Advanced level.

          The equivalence breaks once schools adopt in different years and effects change over time. Then TWFE uses already-treated schools as comparisons, and its coefficient can be a poor summary of any effect. That is the subject of the Advanced level.

          Run it yourself in R: code and output
          Data structures

          Panels and repeated cross-sections

          In a panel, the same units are observed before and after. The shared dataset is a panel of schools, and Card and Krueger (1994) followed the same 410 fast-food restaurants in New Jersey and Pennsylvania before and after New Jersey raised its minimum wage. In a repeated cross-section, different individuals are sampled each period, as when each year’s tenth graders are tested or a survey draws a new sample.

          The DiD formula is the same, but the assumption carries more weight with repeated cross-sections. Parallel trends must hold for groups whose membership changes. If the kind of students in program schools shifts after adoption, for example because families move toward or away from the program, the change in averages mixes the program effect with a change in who is measured. Stuart et al. (2014) make this point and propose weighting all four group-by-period cells to the same covariate profile. Sant’Anna and Zhao (2020) give estimators for both data structures and show the efficiency gains that panel data provide.

          Example · published study

          Ending a student-aid program

          From 1965 to 1982, Social Security paid monthly benefits to the 18- to 22-year-old children of deceased, disabled, or retired beneficiaries while they were enrolled full time in college. In 1980, the average annual benefit for the child of a deceased parent was $6,700 in 2000 dollars, more than tuition and fees at a public four-year college. Congress voted to eliminate the program in 1981, and students not enrolled in college by May 1982 lost eligibility. Dynarski (2003) uses this change to estimate the effect of aid on college attendance with data from the National Longitudinal Survey of Youth.

          The design compares high school seniors from the 1979 to 1981 cohorts, who could still receive the benefit, with seniors from 1982 and 1983, who could not. Eligibility is proxied by the death of a father. Dynarski uses deaths rather than parental disability or retirement because those could themselves respond to the availability of benefits. Among seniors whose fathers had died, 56.0 percent of the earlier cohorts attended college by age 23, against 35.2 percent of the later cohorts. That 20.8-point drop compares cohorts, so it would include any general change in college-going between those years. Among seniors whose fathers were alive, and who were never eligible, attendance fell from 50.2 percent to 47.6 percent, a 2.6-point drop. The difference in differences is 18.2 percentage points, with a standard error of 9.6 points, statistically significant at the 6 percent level.

          Three features of the study illustrate this lesson. First, the estimand is explicit. Because eligibility is measured rather than receipt, the coefficient is the effect of being offered aid, an intention-to-treat effect, which is the quantity a policy maker controls. Second, the comparison group is argued for, not assumed. Dynarski shows that differences in family income, test scores, and other background characteristics between the two groups were stable across the cohorts, which supports the claim that the groups’ trends would have matched. Third, the regression form allows covariates. Adding family background, test scores, state of residence, and their interactions with cohort and group moves the estimate to 21.9 points (standard error 10.2), while the share of variance explained rises from 0.002 to 0.339. A stable estimate under such a large change in explained variance is reassuring. Murnane and Willett (2011) use the same study to teach DiD in an education context.

          Practice

          What to report

          • The estimand.Say that the target is the ATT, for which units, and over which post-adoption years.
          • The 2×2 table.Show the four group means before and after. Readers can then see the level gap and both changes.
          • The comparison group and why.Explain why the groups differ in level and why their untreated trends should still match.
          • Threats to each assumption.Discuss anticipation, spillovers, and changes in who is measured, with evidence where available.
          • The scale of the outcome.Say whether effects are in points, standard deviations, or percent, and why parallel trends is plausible on that scale.
          Limitations

          What the basic design cannot do

          • Parallel trends cannot be tested where it matters.Pre-period trends can support it, as the event-study lesson explains, but not establish it.
          • The estimate is local to the treated.It describes adopting schools in the years observed, not the effect anywhere else.
          • Staggered adoption changes the regression.With adoption in different years, TWFE is no longer the DiD of means (Advanced level).
          • Scale matters.An effect in points and an effect in percent rest on different parallel-trends assumptions.
          • Precision needs its own treatment.Default regression standard errors are usually too small in panels. The next lesson explains why.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.Natural experiments and DiD with education examples, including the student-aid case. Start here.
          • Dynarski, S. M. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279–288. https://doi.org/10.1257/000282803321455287The student-aid study used in this lesson.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008The canonical DiD model and a map of recent extensions.
          • Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828Chapter 5 presents DiD and fixed effects as regression.
          • Card, D., & Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793.The minimum-wage study that made the 2×2 design famous.
          • Wooldridge, J. M. (2025). Two-way fixed effects, the two-way Mundlak regression, and difference-in-differences estimators. Empirical Economics, 69(5), 2545–2587. https://doi.org/10.1007/s00181-025-02807-zTWFE, the two-way Mundlak regression, and their extensions.
          • Roth, J., & Sant'Anna, P. H. C. (2023). When is parallel trends sensitive to functional form? Econometrica, 91(2), 737–747. https://doi.org/10.3982/ecta19402When parallel trends depends on the scale of the outcome.
          • Stuart, E. A., Huskamp, H. A., Duckworth, K., Simmons, J., Song, Z., Chernew, M. E., & Barry, C. L. (2014). Using propensity scores in difference-in-differences models to estimate the effects of a policy change. Health Services and Outcomes Research Methodology, 14(4), 166–182. https://doi.org/10.1007/s10742-014-0123-zRepeated cross-sections and changing group composition.
          • Kahn-Lang, A., & Lang, K. (2020). The promise and pitfalls of differences-in-differences: Reflections on 16 and Pregnant and other applications. Journal of Business & Economic Statistics, 38(3), 613–620. https://doi.org/10.1080/07350015.2018.1546591Why a DiD study must explain differences in levels and justify its comparison group.
          • Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement. Journal of Policy Analysis and Management, 30(3), 418–446. https://doi.org/10.1002/pam.20586A comparison group defined by policy history.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Intermediate · Lesson I3

          Standard errors that match the design.

          A DiD estimate is only as informative as its standard error. This lesson explains why default standard errors are too small in panel data, what clustering does and where to cluster, and what to do when there are few clusters or only a handful of treated ones.

          Before thisIntermediate: notation and TWFEThe ATT, parallel trends, and DiD as a regression with school and year effects.
          After thisIntermediate: event studies and pre-trendsYear-by-year effects, joint tests, and uniform confidence bands.
          By the end

          What you will be able to do

          1. Explain why serial correlation makes default DiD standard errors too small.
          2. Write the cluster-robust variance estimator and state what it assumes.
          3. Choose the level of clustering from the way treatment was assigned.
          4. Recognize when there are too few clusters, and use the wild cluster bootstrap.
          5. Distinguish few clusters from few treated clusters, and choose an inference method for each.
          Question

          How much do 1,200 observations tell us?

          In the shared dataset, 60 schools start a tutoring program in 2020 and 60 comparison schools never do. Each school is observed for ten years, so the TWFE regression from the previous lesson uses 1,200 school-years. It estimates that the program raised math scores by 3.34 points, close to the true average effect of 3.30.

          A district leader now asks how precise that number is. Could a program with no effect have produced an estimate this large by chance? Answering requires a standard error, and the standard error depends on how much independent information the data contain. The 1,200 school-years are not 1,200 independent pieces of information. A school that scores high this year tends to score high next year, and the program was assigned to whole schools, not to single school-years.

          This lesson is about uncertainty, not bias. Every method below leaves the estimate of 3.34 unchanged. What changes is how wide the confidence interval is, and whether a test at the 5 percent level really rejects a true null hypothesis 5 percent of the time.

          The problem

          Why default standard errors are too small

          Default (ordinary least squares) standard errors assume that the regression errors are independent across observations. Two features of DiD panels break that assumption at once. First, the outcome is serially correlated. Unobserved influences on a school’s scores, such as its staff or its neighborhood, persist from year to year, and school fixed effects remove only the part that is constant. Second, the treatment indicator is itself highly correlated over time, because it switches from 0 to 1 once and then stays on. When both the errors and the regressor of interest move slowly, the data contain far less independent variation than the row count suggests, and the default formula overstates precision.

          Bertrand et al. (2004) documented how common and how serious the problem was. They surveyed 92 DiD papers published in six economics journals between 1990 and 2000. Of these, 65 used more than two periods without collapsing them and so faced a potential serial correlation problem, and only 5 made any correction. The papers used 16.5 periods on average. The authors then assigned fake state laws at random in Current Population Survey data on women’s wages, so that no law could have had an effect. Conventional DiD standard errors found a statistically significant “effect” at the 5 percent level for up to 45 percent of these placebo laws.

          Heteroskedasticity-robust standard errors do not fix this. They allow the error variance to differ across observations, but they still treat observations as independent. In the shared data, the robust and default standard errors are both 0.24.

          Why serial correlation shrinks the information

          Suppose a school contributes \(T\) yearly errors that each have variance \(\sigma^2\) and a common pairwise correlation \(\rho\). The variance of their average is

          \[ \operatorname{Var}\!\left(\frac{1}{T}\sum_{t=1}^{T} \varepsilon_t\right) = \frac{\sigma^2}{T}\,\bigl[1 + (T-1)\rho\bigr]. \]

          The default formula uses \(\sigma^2/T\) and ignores the bracket. With \(T = 10\) and \(\rho = 0.5\), the bracket equals 5.5, so the true variance is more than five times what the default formula assumes. The bracket is the design effect familiar from survey sampling and multilevel models. In DiD the treatment indicator is also correlated within schools, which is what turns correlated errors into too-small standard errors for the DiD coefficient.

          Estimator

          The cluster-robust variance estimator

          A cluster is a group of observations whose errors may be correlated with each other in any way, while errors in different clusters are assumed to be independent. With schools as clusters, all ten years of a school form one cluster. Write \(\tilde D_{gt}\) for the treatment indicator after removing school and year means, \(\hat\varepsilon_{gt}\) for the TWFE residual, and \(G\) for the number of clusters. The cluster-robust variance of the DiD coefficient \(\hat\beta\) is

          \[ \widehat{\operatorname{Var}}(\hat\beta) = \frac{G}{G-1}\;\frac{\sum_{g=1}^{G}\Bigl(\sum_{t} \tilde D_{gt}\,\hat\varepsilon_{gt}\Bigr)^{2}}{\Bigl(\sum_{g}\sum_{t} \tilde D_{gt}^{2}\Bigr)^{2}}. \]

          The key step is the inner sum. The products \(\tilde D_{gt}\hat\varepsilon_{gt}\) are added up within each cluster before squaring, so all within-cluster covariances enter the variance, whatever their pattern. The default formula would square each product separately and drop those covariances. The factor \(G/(G-1)\) is a small-sample correction; software packages apply slightly different versions of it, which is why standard errors can differ in the second decimal across programs.

          The estimator asks for very little about the correlation within clusters, but it asks a lot about their number. It is consistent as the number of clusters grows, not as the number of observations grows. With 120 schools there are 120 clusters. With five states there are five, however many students they contain. Tests and confidence intervals are therefore usually based on a \(t\) distribution with \(G - 1\) degrees of freedom rather than the normal distribution.

          Bertrand et al. (2004) also studied a simpler alternative. Average each unit’s outcome over the years before treatment and over the years after, and run the DiD on this two-period panel. Collapsing discards the time-series information that caused the problem. In their simulations it held the rejection rate at 5.3 percent with 50 states and performed well even with few states, although its power falls as the number of states shrinks. It applies most directly when all treated units adopt at the same time.

          Design

          Where to cluster

          Two views of uncertainty lead to similar advice. In the familiar sampling-based view, the clusters are a sample from a larger population, and one clusters at the level at which errors are correlated. This can be hard to apply. Students are nested in classrooms, schools, districts, and states, and errors may be correlated at every level.

          In the design-based view, the units in the data are treated as fixed, and the uncertainty comes from which units happened to receive treatment. Roth et al. (2023) summarize recent work showing that, from this view, cluster-robust standard errors are valid, although possibly conservative, when clustering is done at the level at which treatment is independently assigned. This gives a clear rule even when no natural larger population exists, as with the 50 U.S. states.

          • If schools choose the program independently, cluster by school.
          • If districts adopt the program for all their schools, cluster by district, even if the outcome is measured for students.
          • For a state policy, cluster by state.

          Clustering at a level finer than assignment is the most common error. If five districts adopt a program for their 200 schools, school-level clustering treats the 200 schools as independent pieces of evidence, when the decisions behind them were made five times. Clustering at a coarser level than needed is usually safer, but it costs precision and can leave too few clusters, which is the problem in the next section.

          Example · shared school data

          One estimate, four standard errors

          The first script below estimates the same TWFE regression and computes its standard error in four ways. The estimate is 3.34 in every case. The default and heteroskedasticity-robust standard errors are both 0.24. Clustering by school, the level at which the program was assigned, gives 0.41, about 1.7 times larger. Collapsing each school to a before and an after average gives the same estimate and a standard error of 0.41.

          The ratio has a useful reading. The variance is (0.41 / 0.24)² ≈ 2.9 times what the default formula assumes. In this sense, the 1,200 school-years carry roughly as much information about the program effect as 420 independent observations would. The 95 percent confidence interval, based on 119 degrees of freedom, runs from 2.54 to 4.15. The program effect is still clearly distinguishable from zero, but the interval is much wider than the default one.

          Run it yourself in R: code and output
          When clusters are few

          Few clusters and the wild cluster bootstrap

          With few clusters, cluster-robust standard errors tend to be too small and tests reject a true null hypothesis too often. Bertrand et al. (2004) found that clustering by state gave rejection rates of 6.3 percent with 50 states and 5.8 percent with 20, but 8 percent with 10 states and 11.5 percent with 6. Cameron et al. (2008) report over-rejection with roughly five to thirty clusters.

          Cameron et al. (2008) proposed the wild cluster bootstrap-t, which in their simulations brought rejection rates of about 10 percent down to the nominal 5 percent, in examples with as few as five clusters. It works as follows.

          1. Estimate the model with the null hypothesis imposed, here \(\beta = 0\), and keep its fitted values and residuals.
          2. For each cluster, draw a weight of +1 or −1 with equal probability (Rademacher weights). Multiply all of that cluster’s residuals by its weight, so the within-cluster pattern is preserved.
          3. Add the reweighted residuals to the fitted values to create a new outcome, re-estimate the full model, and compute the cluster-robust \(t\) statistic.
          4. Repeat many times. The p-value is the share of bootstrap \(t\) statistics that are at least as large in absolute value as the original one.

          The method bootstraps the \(t\) statistic rather than the coefficient, which is what gives it better accuracy with few clusters. With \(G\) clusters there are only \(2^G\) distinct sets of weights, 64 when \(G = 6\), so with very few clusters it is better to use all of them than to draw at random.

          The second script checks these claims with placebo programs that have no effect, in panels with the same serial correlation as the shared data. The default test rejects in 26 percent to 29 percent of samples, whatever the number of schools. Clustered standard errors reject in 8.3 percent of samples with 6 schools, 7.0 percent with 10, and close to 5 percent with 20 or 60. The wild cluster bootstrap rejects in 4.4 percent to 5.5 percent of samples at every size.

          Simulate · invite a test

          The placebo lab

          Every study in this lab evaluates a program that has no effect. Choose the number of schools, how many get the program, and how strongly scores persist over time, then see how often each method finds a “significant” effect.

          How often is a true null rejected?

          Each run draws 200 panels of 10 years with school and year effects and autocorrelated errors, then tests the placebo program at the 5 percent level with three methods. The dashed line marks the 5 percent target.

          Each school is one cluster.
          Few treated clusters is a different problem from few clusters.
          0.7
          At 0, errors are independent over time.
          Try this
          1. Set the correlation to 0. Does the default method still over-reject?

          2. With 6 schools, compare clustered standard errors with the wild bootstrap.

          3. With 60 schools, give the program to 1 school. What happens to each method?

          Default standard errors–rejection rate
          Clustered by school–rejection rate
          Wild cluster bootstrap–rejection rate

          Press the button to run 200 placebo studies.

          A harder problem

          When only a few clusters are treated

          Many education policies are adopted by one state or a handful of districts and compared with many that did not adopt. The total number of clusters can be large while the number of treated clusters is tiny. Cluster-robust inference needs many treated and many untreated clusters, because the DiD estimate averages the treated clusters’ shocks, and with one treated cluster that average is a single draw that no formula can learn much about (Roth et al., 2023).

          The wild cluster bootstrap does not solve this. MacKinnon and Webb (2018) show that when only a few clusters are treated, the version with the null imposed can under-reject severely, while the version without it can over-reject severely. They propose a subcluster variant that can work well in some treatment models, but they note that its key requirement, similar cluster sizes, is unlikely to hold in typical DiD regressions.

          The third script shows the pattern with 60 schools, varying how many receive the placebo program. With one program school, clustered standard errors reject a true null in 78.8 percent of samples, and the wild bootstrap almost never rejects (0.0 percent). With three program schools the rates are 20.0 percent and 0.5 percent; with six, 9.8 percent and 4.4 percent; with 30, both are close to 5 percent.

          Methods for this case borrow strength from the many comparison clusters. Conley and Taber (2011) assume that the treated clusters’ errors come from the same distribution as the comparison clusters’ errors. The many comparison clusters then reveal that distribution, and the estimate is compared with it. Ferman and Pinto (2019) show that this can fail when clusters differ in size, because averages over small clusters are noisier, and they extend the approach to allow for such heteroskedasticity. The third script uses a simplified version of the Conley and Taber idea, drawing comparison-school residuals in place of the treated schools’ errors. It rejects in 5.5 percent to 6.0 percent of samples with one to six program schools. It over-rejects (18.8 percent) when half the schools are treated, because it ignores the uncertainty in the comparison group’s average, a reminder that it was built for the case of few treated units.

          Every option here rests on some homogeneity assumption across clusters. Roth et al. (2023) recommend choosing the method whose homogeneity assumption is most plausible in context. They also describe two alternatives. One treats the treated cluster’s own shock as a possible violation of parallel trends and examines it with the sensitivity analysis in the Advanced level. The other uses permutation tests, which reassign treatment across clusters and are exact if treatment was randomly assigned, but test the sharp null of no effect for every unit.

          Example · published study

          Georgia’s HOPE scholarship and one treated state

          Georgia’s HOPE scholarship, which began in 1993, pays full tuition and some fees at in-state public colleges for students who keep a 3.0 grade point average in high school. Conley and Taber (2011) use it to illustrate their method. Their data are the October Current Population Survey from 1989 to 2000, covering 34,902 people aged 18 and 19, and the outcome is college enrollment. Georgia is the only treated state, compared with 40 states and the District of Columbia that did not adopt merit aid during that period.

          The point estimate is a 7.8 percentage-point increase in enrollment. Clustering by state gives a 95 percent confidence interval from 5.8 to 9.7 points, which looks precise. That interval relies on many treated states, yet only Georgia contributes to the estimate. The Conley and Taber interval, which learns the distribution of state-level shocks from the 41 comparison units, runs from −1.0 to 20.7 points and includes zero.

          The authors present this as an illustration of their method rather than a full analysis of merit aid. The lesson for applied work is that the same point estimate can support very different conclusions depending on how uncertainty is assessed. With one treated state, the number of students in the sample does little to reduce that uncertainty.

          Practice

          What to report

          • The clustering level and why.Name the level at which treatment was assigned and cluster there.
          • The number of clusters and of treated clusters.Report both. Readers cannot judge the inference without them.
          • The reference distribution.State the degrees of freedom, or for a bootstrap, the weights, whether the null was imposed, and the number of draws.
          • A method suited to few clusters, when needed.With fewer than about 30 clusters, add wild cluster bootstrap p-values. With few treated clusters, use a method built for that case and say which homogeneity assumption it makes.
          • Robustness.Where the level of assignment is ambiguous, show results at more than one level of clustering.
          Limitations

          What clustering cannot do

          • It does not fix bias.A correct standard error around a biased estimate gives a precise answer to the wrong question. Parallel trends still has to be defended.
          • It needs independence between clusters.Shocks that hit many districts at once, such as a statewide test change, are not covered by district clustering.
          • Rules of thumb are approximate.How many clusters are “enough” depends on cluster sizes, how many are treated, and the error structure. Simulation evidence, like the lab above, is specific to its design.
          • Few treated clusters remains difficult.All current methods trade one strong assumption for another. Report what each assumes.
          • Software defaults differ.Packages apply different small-sample corrections and degrees of freedom. State the software and options used.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249–275. https://doi.org/10.1162/003355304772839588The placebo-law study that made clustering standard practice in DiD. Start here.
          • Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3), 414–427. https://doi.org/10.1162/rest.90.3.414The wild cluster bootstrap-t and its performance with few clusters.
          • MacKinnon, J. G., & Webb, M. D. (2018). The wild bootstrap for few (treated) clusters. The Econometrics Journal, 21(2), 114–135. https://doi.org/10.1111/ectj.12107Why the wild cluster bootstrap can fail when few clusters are treated, and the subcluster alternative.
          • Conley, T. G., & Taber, C. R. (2011). Inference with “difference in differences” with a small number of policy changes. Review of Economics and Statistics, 93(1), 113–125. https://doi.org/10.1162/rest_a_00049Inference with a small number of policy changes, illustrated with merit aid.
          • Ferman, B., & Pinto, C. (2019). Inference in differences-in-differences with few treated groups and heteroskedasticity. Review of Economics and Statistics, 101(3), 452–467. https://doi.org/10.1162/rest_a_00759Extending that approach when cluster sizes differ.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008A review of inference with few clusters and of the design-based case for clustering at the level of assignment.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Intermediate · Lesson I4

          Event studies and pre-trends.

          An event study shows how an effect unfolds year by year and offers a check on the comparison group. This lesson explains what each coefficient estimates, how to build and read the plot, what a pre-trend test can and cannot tell you, and which tools do better than a pass or fail verdict.

          Before thisIntermediate: notation, TWFE, standard errorsThe ATT, parallel trends, and clustering by school.
          After thisAdvanced: staggered adoption and sensitivity analysisEvent studies when schools adopt in different years, and bounds when trends may diverge.
          By the end

          What you will be able to do

          1. State what each event-study coefficient estimates and the assumptions that give it a causal reading.
          2. Specify an event study: reference period, event window, endpoint bins, and the sample of units.
          3. Build a plot that a reader can evaluate, with pointwise intervals, a uniform band, and joint tests.
          4. Explain why passing a pre-trend test is weak evidence, and report an equivalence bound instead.
          5. Choose a response when pre-trends appear: extrapolation, a proxy covariate, or sensitivity analysis.
          Question

          Two questions in one plot

          In the shared dataset, 60 schools start a tutoring program in 2020 and 60 comparison schools never do. A district leader asks two questions. Did scores rise because of the program, and did the gain grow, fade, or hold steady? The two-period DiD from the Foundations level answers neither question well. It averages all post-adoption years into one number, and it gives no direct view of whether the two groups were moving together before the program began.

          An event study estimates one coefficient for each year relative to adoption. The coefficients after adoption, called lags, trace the dynamics of the effect. The coefficients before adoption, called leads, are placebo estimates. The program cannot have affected scores before it existed, so leads far from zero suggest that the comparison schools are a poor guide to what program schools would have done.

          Miller (2023) notes that event studies rest on one of three data structures. In a DiD-type design, treated units share an event date and never-treated units supply the comparison, as in this lesson. In a timing-based design, every unit is eventually treated and identification comes from differences in timing. A hybrid design has both. The structure determines which coefficients the data can identify and which comparisons drive them.

          Estimand

          What each coefficient estimates

          Let \(G_i\) be the adoption year of school \(i\) and \(e = t - G_i\) the event time. The standard specification with unit and year fixed effects is

          \[Y_{it} = \alpha_i + \lambda_t + \sum_{\substack{e=-L \\ e \neq -1}}^{M} \delta_e \, \mathbf{1}[t - G_i = e] + \varepsilon_{it}\]

          With one adoption year and a never-treated comparison group, each coefficient equals a two-period DiD that compares event year \(e\) with the reference year \(e = -1\):

          \[\delta_e = E[Y_{G+e} - Y_{G-1} \mid D = 1] - E[Y_{G+e} - Y_{G-1} \mid D = 0]\]

          For \(e \geq 0\), and under the assumptions below, \(\delta_e\) equals the average effect on program schools \(e\) years after adoption, \(\text{ATT}_e\). For \(e \leq -2\), the true value is zero, so the estimate shows how far the two groups drifted apart before the program, relative to the reference year.

          Choice 1 · Reference period

          Something must be left out, because the event-time dummies and the fixed effects are collinear. The common default, \(\delta_{-1} = 0\), measures every effect relative to the last year before adoption. If schools may have responded early, Freyaldenhoven et al. (2021, Suggestion 1) recommend normalizing the last period before any anticipation instead.

          Choice 2 · Window and endpoint bins

          Choose leads and lags that most units contribute to, so coefficients do not shift because the sample changes (Miller, 2023). Bin the endpoints: one dummy for "\(L\) or more years before" and one for "\(M\) or more years after". Binning is a natural identifying restriction, and with binned endpoints the event study is numerically identical to a distributed-lag model (Schmidheiny & Siegloch, 2023).

          Choice 3 · Restrictions, stated openly

          Software may drop collinear dummies without warning. Miller (2023) recommends checking which variables were dropped and reporting how many coefficients were estimated and which restrictions were imposed. A reader cannot interpret the plot without knowing them.

          Assumptions

          When the coefficients have a causal reading

          Parallel trends in every period

          A stronger assumption than the 2×2 needs

          The 2×2 DiD needs parallel trends only between the reference year and the post period. Reading the leads as placebo tests requires parallel trends in every pre-period too, an "augmented" version of the assumption (Dette & Schumann, 2024). Few settings satisfy one version but not the other, which is why leads are informative at all.

          No anticipation

          Nothing changes before adoption

          Fails when schools start tutoring early, or families sort into program schools before the official start. Then the reference year is already treated and every coefficient is shifted.

          Stable composition

          The same schools in every event year

          Fails when the window is unbalanced, so distant coefficients rest on a different set of schools than nearby ones.

          One adoption date, or a robust estimator

          Homogeneous timing

          Fails when schools adopt in different years and effects differ by cohort. TWFE event-study coefficients can then mix effects from other event times, including in the leads (Sun & Abraham, 2021). The Advanced level covers estimators that avoid this.

          Kahn-Lang and Lang (2020) add a design question that no regression answers. Why did the two groups start at different levels? If program schools scored lower because they serve more disadvantaged students, the same disadvantage may shape their trends. A credible event study explains the source of the level gap and argues that it does not also produce different trends.

          Example · shared school data

          Estimating and reading the plot

          The code below estimates the event study on the shared dataset with standard errors clustered by school, then adds the inference that Freyaldenhoven et al. (2021) recommend, which are a joint test of the leads, a test of whether the dynamics level off, and a 95% uniform confidence band. All three versions use the same random draws for the band, so R, Python and Stata print identical results.

          Run it yourself in R: code and output

          Reading the output. The leads range from −0.07 to 0.29, and the joint test of all four does not reject (\(p = 0.88\)). The lags rise from 1.91 in the adoption year to 4.52 four years later, close to the true path of 2, 3, 3.5, 4, and 4. The test that the effect stopped changing between years 3 and 4 does not reject (\(p = 0.23\)).

          Pointwise intervals and the uniform band answer different questions. A pointwise 95% interval covers one coefficient 95% of the time. When readers scan the whole plot, they test many hypotheses at once. A sup-t band is built to contain the entire path 95% of the time, so it is wider: here the critical value is 2.66 rather than 1.96 (Freyaldenhoven et al., 2021, Suggestion 3). A lead that barely excludes zero pointwise but sits inside the band is weak evidence of a pre-trend.

          A plot a reader can evaluate

          • Normalize \(\delta_{-1} = 0\) and mark the reference year
          • Label the outcome mean at the reference year, here (51.1), so magnitudes have a scale
          • Show pointwise intervals and a sup-t band
          • Report p-values for no pre-trends and for leveling off

          In the caption or notes

          • The data structure and comparison group
          • The window, the bins, and the restrictions imposed
          • How many schools contribute to each coefficient
          • The clustering level

          The checklist follows Suggestions 1 to 4 of Freyaldenhoven et al. (2021) and the reporting advice in Miller (2023). The xtevent package for Stata implements these suggestions (Freyaldenhoven et al., 2025).

          Assumptions, tested

          What a passed pre-test shows, and what it does not

          A passed pre-test is often read as confirmation of parallel trends. Four lines of evidence suggest more caution.

          It tests the wrong period. The assumption concerns the post-adoption years, which are never observed without the program. Parallel pre-trends are neither necessary nor sufficient for parallel post-trends (Kahn-Lang & Lang, 2020). Their illustration is memorable: the heights of boys and girls grow in parallel through early childhood, then diverge at puberty, so a pre-period that ends before puberty would wrongly predict parallel growth.

          It often lacks power. A test that fails to reject is not evidence that the null is true. Roth (2022) shows that conventional pre-tests often have low power against violations large enough to matter. In one simulation design of Freyaldenhoven et al. (2019), the pre-test rejected in fewer than 30% of draws, although a pre-trend was present in every draw.

          Screening on it can add bias. If researchers report results only when the pre-test passes, the published estimates come from samples where noise happened to mask the violation. Roth (2022) shows that this selection can make the remaining bias larger. You can see this in the lab below.

          It answers a yes or no question. With very large samples, the test can reject trivial differences. With small samples, it can pass large ones. Neither outcome says how big a violation the data allow, which is the quantity that matters for the estimate.

          The second example: a violation the pre-test misses

          The second code tab adds a small violation to the same data. In it, program schools improve 0.3 points a year faster, with or without the program. The joint pre-test passes (\(p = 0.15\)). The standard estimates are biased upward, and the average post-adoption effect is 4.36 against a true 3.30, about 32% too large.

          Tools

          Better than pass or fail

          Equivalence and non-inferiority tests

          Reverse the burden of proof. The null hypothesis becomes "the pre-trend is at least as large as a threshold \(U\)", so rejecting it is evidence that pre-trends are small (Dette & Schumann, 2024). When no threshold is agreed in advance, report the smallest \(U\) the data can rule out. In the shared data, that bound is 1.16 points: the passed pre-test is still compatible with pre-period deviations of about 60% of the first-year effect. Bilinski and Hatfield (2026) state the threshold on the scale of the treatment effect, by comparing the estimate from the standard model with one that allows a trend difference.

          Extrapolating a pre-trend

          If the violation is linear, fit a line through the leads and subtract its extension from the lags. In the second example, this recovers an average effect of 3.48 against the truth of 3.30. The method is only as good as the linearity assumption. Kahn-Lang and Lang (2020) show that adding group-specific trends when effects build up over time can reverse the sign of an estimate, and Freyaldenhoven et al. (2019) find that linear extrapolation is biased across their simulation designs.

          A proxy for the confounder

          Freyaldenhoven et al. (2019) use a covariate that responds to the unobserved confounder but not to the policy. Its own pre-trend reveals the confounder's path, and two-stage least squares, the instrumental variables estimator of Module 3, with leads of the policy as instruments removes it. The method requires finding such a covariate and arguing that the policy does not affect it.

          Sensitivity analysis

          Instead of assuming parallel trends exactly, ask how large a post-adoption violation would have to be to overturn the conclusion, for example relative to the largest pre-period violation (Rambachan & Roth, 2023). The Advanced level works through this approach.

          These tools complement design reasoning rather than replace it. Each one rests on its own assumption about how the untreated paths would have evolved, and the plot cannot verify any of them.

          Simulate · invite a test

          The pre-test lab

          Choose a violation and a sample size, then draw many samples. The lab reports how often the pre-test passes and how biased the estimates are among the samples that pass.

          One sample at a time

          Program schools adopt in 2020 and the true effects are 0 before adoption and 2, 3, 3.5, 4, and 4 afterwards (dashed gold line). Squares (leads) and circles (lags) are estimates with pointwise 95% intervals; the pale bars show the 95% sup-t band.

          0.30
          A violation of parallel trends that begins before adoption and continues after it.
          0.0
          Points gained in the reference year itself.
          More schools give a more powerful pre-test and a tighter equivalence bound.
          Try this
          1. With a trend of 0.30 and 60 schools, run 200 samples. Compare the bias among samples that pass with the bias overall.

          2. Switch to 200 schools. What happens to the pass rate and to the equivalence bound?

          3. Set the trend to 0 and add 1.5 points of anticipation. Why do the leads and the lags all shift down?

          Joint pre-testp = 0.00
          Equivalence bound0.00smallest pre-trend ruled out at 5%
          Bias in the average lag0.00
          Pass rate (p > 0.05)0%
          Bias, all samples0.00
          Bias, samples that pass0.00

          Example · published study

          No Child Left Behind and comparative interrupted time series

          Dee and Jacob (2011) estimate the effect of No Child Left Behind (NCLB) on state NAEP scores. NCLB reached every state at once, so there is no untouched comparison group. The authors compare states that had no school accountability before NCLB, which received the strongest new treatment, with states that already had consequential accountability, which were affected less, if at all.

          The pre-period shows why a simple DiD would be risky. In 1992, states that never adopted accountability before NCLB scored about 5 scale points (0.18 standard deviations) higher in fourth-grade math. Between 1992 and 2000, the states that adopted accountability improved faster. The groups differed in both level and trend.

          The authors therefore use a comparative interrupted time series. Each group's pre-NCLB trend is estimated, and the effect is the post-NCLB deviation from trend in the newly treated states, minus the corresponding deviation in the comparison states. This is a DiD that allows group-specific linear trends, the extrapolation approach from the previous section. They estimate that NCLB raised fourth-grade math scores by 0.23 standard deviations by 2007, find more targeted gains in eighth-grade math, and find no reliable effect on fourth-grade reading.

          Three features make the study a useful model. The comparison group is justified by policy history rather than chosen to fit the pre-period. The authors test their identifying assumption on outcomes such as test exclusion rates. They also report the sensitivity of their estimates to the dating of the policy. Treating NCLB as starting a year later reduces the fourth-grade math estimate to 0.10 standard deviations, still statistically significant. The linear-trend assumption remains an assumption. If the accountability states' gains were already slowing in 2000, the extrapolated counterfactual would be too optimistic for them, and the estimated effect would change.

          Limitations

          When an event-study plot misleads

          • Staggered adoption with heterogeneous effects.TWFE leads and lags can mix effects from other event times, so a clean-looking pre-period may be an artifact (Sun & Abraham, 2021). Use the robust estimators in the Advanced level.
          • Shifting samples.In an unbalanced window, distant coefficients rest on fewer or different schools. Report the number of units behind each coefficient.
          • Silent restrictions.Software that drops collinear dummies changes what the coefficients mean. State the reference period and the bins.
          • A reference year that is already treated.Anticipation shifts every coefficient. If early responses are plausible, move the reference period earlier.
          • Functional form.Parallel trends in scores do not imply parallel trends in log scores. The Expert level returns to this point.
          • Reading the plot as proof.Flat leads are consistent with parallel trends but cannot establish them. Pair the plot with design reasoning, an equivalence bound, or a sensitivity analysis.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Miller, D. L. (2023). An introductory guide to event study models. Journal of Economic Perspectives, 37(2), 203–230. https://doi.org/10.1257/jep.37.2.203An accessible guide to data structures, specification choices, and reporting. Start here.
          • Freyaldenhoven, S., Hansen, C., Pérez Pérez, J., & Shapiro, J. M. (2021). Visualization, identification, and estimation in the linear panel event-study design (NBER Working Paper No. 29170). National Bureau of Economic Research. https://doi.org/10.3386/w29170Seven suggestions for more informative plots, and a comparison of identifying assumptions. Working paper.
          • Freyaldenhoven, S., Hansen, C. B., Pérez Pérez, J., Shapiro, J. M., & Carreto, C. (2025). xtevent: Estimation and visualization in the linear panel event-study design. The Stata Journal, 25(1), 97–135. https://doi.org/10.1177/1536867X251322964The xtevent package for Stata, which implements those suggestions.
          • Schmidheiny, K., & Siegloch, S. (2023). On event studies and distributed-lags in two-way fixed effects models: Identification, equivalence, and generalization. Journal of Applied Econometrics, 38(5), 695–713. https://doi.org/10.1002/jae.2971Binning endpoints as an identifying restriction, and the equivalence with distributed-lag models.
          • Roth, J. (2022). Pretest with caution: Event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305–322. https://doi.org/10.1257/aeri.20210236The low power of pre-tests and the bias created by screening on them.
          • Kahn-Lang, A., & Lang, K. (2020). The promise and pitfalls of differences-in-differences: Reflections on 16 and Pregnant and other applications. Journal of Business & Economic Statistics, 38(3), 613–620. https://doi.org/10.1080/07350015.2018.1546591Why parallel pre-trends are neither necessary nor sufficient, and the role of design reasoning.
          • Dette, H., & Schumann, M. (2024). Testing for equivalence of pre-trends in difference-in-differences estimation. Journal of Business & Economic Statistics, 42(4), 1289–1301. https://doi.org/10.1080/07350015.2024.2308121Equivalence tests for pre-trends, and reporting the smallest bound the data can rule out.
          • Bilinski, A., & Hatfield, L. A. (2026). Nothing to see here? A non-inferiority approach to parallel trends. Statistics in Medicine, 45(3–5), Article e70296. https://doi.org/10.1002/sim.70296Non-inferiority tests on the scale of the treatment effect, with a review of practice in medical journals.
          • Freyaldenhoven, S., Hansen, C., & Shapiro, J. M. (2019). Pre-event trends in the panel event-study design. American Economic Review, 109(9), 3307–3338. https://doi.org/10.1257/aer.20180609Using a covariate that tracks the confounder to correct for pre-trends.
          • Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591. https://doi.org/10.1093/restud/rdad018Sensitivity analysis when parallel trends may fail. Covered in the Advanced level.
          • Sun, L., & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006Why TWFE event studies mislead under staggered adoption with heterogeneous effects.
          • Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement. Journal of Policy Analysis and Management, 30(3), 418–446. https://doi.org/10.1002/pam.20586An education application of comparative interrupted time series.
          Module 5, Lesson I4 · Last revised November 2026 · Content checked against the full texts of all cited sources on November 9, 2026. Every work this lesson draws on is listed above. Figures from Dee and Jacob (2011) are taken from the published article; all other explanations, data, and code were written for this module.
          Learn · Module 5 · Intermediate · Lesson I5

          Covariates and conditional parallel trends.

          Parallel trends may fail for the whole sample yet hold among similar schools. This lesson explains the conditional version of the assumption, why adding covariates to a TWFE regression is not enough, how outcome regression, weighting, and doubly robust estimators work, how to check overlap, and which covariates should never be adjusted for.

          Before thisIntermediate: event studies and pre-trendsWhat a pre-trend test can and cannot show.
          After thisExpert: doubly robust and machine-learning DiDEfficiency, flexible nuisance models, and cross-fitting.
          By the end

          What you will be able to do

          1. State conditional parallel trends and the overlap condition, and explain what each requires.
          2. Explain why a TWFE regression with covariates can miss the ATT even when conditional parallel trends holds.
          3. Compute outcome regression, inverse probability weighting, and doubly robust DiD estimates.
          4. Check overlap and covariate balance, and recognize when weights are too concentrated.
          5. Separate good controls from bad ones, including mediators and variables affected by both the program and the outcome.
          Question

          When the comparison schools serve different students

          The shared covariate dataset has 60 schools that start a tutoring program in 2020 and 60 that never do. The true effect is 3.0 points in every program school. One feature makes the comparison difficult. Scores grow more slowly in schools that serve more students eligible for free or reduced-price lunch (FRL), and program schools serve more of them, about 60 percent against 40 percent.

          Without the program, program schools would therefore have grown more slowly than comparison schools, and unconditional parallel trends fails. The standard TWFE regression estimates 0.70 instead of 3.0. The comparison schools’ faster growth is mistaken for a weaker program.

          A weaker assumption may still hold. Among schools with the same FRL share, program and comparison schools may have been on the same path. This lesson is about using that assumption well. The covariate is measured before the program starts, the program cannot change it, and it plausibly explains why the groups would have grown at different rates. Those three properties are what make a covariate useful here.

          Estimand and assumptions

          Conditional parallel trends and overlap

          The target is still the ATT, the average effect in the post period among program schools. Write \(D = 1\) for program schools, \(X\) for pre-treatment covariates, and \(Y_t(0)\) for the outcome a school would have had without the program. Conditional parallel trends states that, for every value of the covariates,

          \[ E\bigl[Y_1(0) - Y_0(0) \mid D = 1, X\bigr] = E\bigl[Y_1(0) - Y_0(0) \mid D = 0, X\bigr]. \]

          The untreated change for program schools equals the untreated change for comparison schools with the same \(X\). Sant’Anna and Zhao (2020) note that this allows trends to differ by covariate value, while it still rules out trends specific to individual units.

          The second assumption is overlap. Every program school must have comparison schools with similar covariates. Formally, the propensity score \(p(X) = P(D = 1 \mid X)\) must stay below 1 by some margin. If some program schools have covariate values that no comparison school shares, the data say nothing directly about how those schools would have changed without the program. Any estimate for them rests on extrapolation from a model.

          Conditional parallel trends is not a weaker version of the same claim at no cost. It requires the right covariates, measured before treatment, and it moves part of the burden from the design to the specification of models. A pre-trend check conditional on covariates is still useful, with the limits described in the previous lesson.

          The problem

          Why adding covariates to TWFE is not enough

          The first instinct is to add the covariate to the TWFE regression. A covariate that does not change over time does nothing in that regression, because the school fixed effects absorb it. FRL share measured before the program is exactly such a covariate.

          The next step is to let the year effects differ by FRL, for example with i(year, frl) in fixest or pyfixest. Each year then has its own slope on FRL, so the regression allows FRL-specific trends. In the shared data this gives 3.03, close to the truth.

          This works here because extra conditions happen to hold. Sant’Anna and Zhao (2020, Remark 1) show that the common TWFE regression with covariates assumes that the treatment effect does not vary with \(X\) and rules out covariate-specific trends. Letting the year effects vary with FRL relaxes the second restriction, but only in the form the regression specifies, and the first restriction remains. If the effect is larger in some kinds of schools, the regression coefficient is in general not the ATT.

          The second script shows this. When the effect is the same in every school, TWFE with FRL-specific year effects is unbiased across 200 samples. When the effect grows with FRL share, by 8 points for every unit of FRL share, its average bias is −0.76 points, about a quarter of the ATT, while the three estimators in the next section remain close to unbiased. In real data there is rarely a reason to assume effects are the same in high-poverty and low-poverty schools.

          Estimators

          Outcome regression, weighting, and doubly robust DiD

          All three estimators start from each school’s change in the outcome, \(\Delta Y = Y_1 - Y_0\), here the mean score from 2020 on minus the mean before. They differ in how they build the counterfactual change for program schools.

          Outcome regression (OR). Model how comparison schools’ scores changed as a function of the covariates, \(\mu_0(X) = E[\Delta Y \mid X, D = 0]\). Predict that change for each program school and average the gaps. Heckman, Ichimura, and Todd (1997) introduced a nonparametric version, a conditional difference-in-differences extension of matching. Using data from a randomized job training experiment, they rejected the assumptions behind conventional matching but not those behind their difference-in-differences version, which also removes bias from unobserved differences that are constant over time.

          \[ \hat\tau_{OR} = \frac{1}{n_1}\sum_{i:\,D_i = 1}\bigl(\Delta Y_i - \hat\mu_0(X_i)\bigr). \]

          It is consistent if the model for \(\mu_0\) is correctly specified.

          Inverse probability weighting (IPW). Abadie (2005) showed that the ATT can be recovered without modeling the outcome. Estimate the propensity score, then weight each comparison school by \(\hat p(X)/(1 - \hat p(X))\). Comparison schools that look like program schools count more. With the weights normalized to sum to one,

          \[ \hat\tau_{IPW} = \frac{1}{n_1}\sum_{i:\,D_i = 1}\Delta Y_i \;-\; \sum_{j:\,D_j = 0} w_j\,\Delta Y_j, \qquad w_j \propto \frac{\hat p(X_j)}{1 - \hat p(X_j)}. \]

          It is consistent if the propensity score model is correctly specified.

          Doubly robust (DR). Sant’Anna and Zhao (2020) combine the two. Apply the IPW comparison to the residuals from the outcome model, \(\Delta Y - \hat\mu_0(X)\). The estimator is consistent if either the outcome model or the propensity model is correct, and it attains the semiparametric efficiency bound when both are. The two approaches rest on different, non-nested conditions that are hard to rank in practice, which is one reason for combining them.

          These estimators also have a practical advantage. They make the comparison explicit. The outcome model shows which comparison schools’ changes are being extrapolated, and the weights show which comparison schools carry the estimate.

          Example · shared school data

          Five estimates of one effect

          The first script computes all five estimates on the shared covariate dataset. TWFE without covariates gives 0.70. TWFE with FRL-specific year effects gives 3.03, outcome regression 2.82, IPW 2.37, and the doubly robust estimator 2.96. The true ATT is 3.00.

          One sample cannot rank estimators, so the second script repeats the exercise in 200 new samples from the same design. With the same effect in every school, the four covariate-adjusted estimators are all close to unbiased, with average errors between −0.07 and 0.04 points, while TWFE without covariates is off by −1.97. They differ in precision. The standard deviation across samples is 0.54 for TWFE with FRL-specific year effects, 0.63 for outcome regression, 0.78 for the doubly robust estimator, and 0.89 for IPW. When its extra assumptions hold, the TWFE specification is the most precise. When they fail, as in the second scenario, it is the only covariate-adjusted estimator that is biased.

          The large spread of IPW is not an accident of this design. It reflects limited overlap, which the next section examines.

          Run it yourself in R: code and output
          Diagnostics

          Checking overlap and balance

          Before trusting any of these estimates, look at the propensity scores. In the shared data, program schools’ scores run from 0.10 to 0.99 and comparison schools’ from 0.01 to 0.84. Eleven of the 60 program schools have scores above 0.9. Their FRL shares are so high that few comparison schools resemble them.

          For the ATT, the warning sign is program schools with scores near 1, because they lack comparable comparison schools. Comparison schools with scores near 0 matter less. They receive almost no weight, which is appropriate, since they resemble no program school. Crump et al. (2009) proposed a rule of thumb for average treatment effects, discarding units with estimated propensity scores outside the range from 0.1 to 0.9. Trimming changes the population the estimate describes, so report it as an estimate for schools with overlap.

          Two summaries show whether weighting has worked. The first is the standardized difference in each covariate, the difference in means divided by a pooled standard deviation. Stuart et al. (2014) describe values above 0.1 or 0.2 as a substantial imbalance. In the shared data, the standardized difference in FRL is 1.47 before weighting and 0.40 after. Weighting reduced the imbalance, but did not remove it. The second is the effective number of comparison schools, one over the sum of squared normalized weights. Here the 60 comparison schools act like 21.5 equally weighted schools, and the single most heavily weighted school carries 11 percent of the weight. Concentrated weights are why IPW was the least precise estimator.

          When overlap is limited, three responses are common. Restrict the sample to the region of overlap and say so. Prefer the doubly robust estimator, which leans on the outcome model where weights are unstable. Or reconsider the comparison group, for example by drawing comparison schools from districts that serve similar students.

          Which covariates

          Good controls and bad controls

          The covariates in conditional parallel trends should explain why the groups would have trended differently. Cinelli et al. (2024) use causal diagrams to sort candidate controls into good, neutral, and bad, and several of their cases matter for DiD.

          • Good. Pre-treatment characteristics that affect both who adopts and how outcomes evolve, such as FRL share, prior enrollment trends, or urbanicity.
          • Neutral, possibly helpful. Variables that predict the outcome but not adoption. They do not reduce bias, but they can improve precision.
          • Bad, mediators. Variables the program changes on the way to the outcome, such as tutoring hours or attendance after adoption. Controlling for them removes part of the effect being estimated, which Cinelli et al. call overcontrol bias.
          • Bad, colliders and their descendants. Variables affected by both the program and the outcome. Conditioning on them creates an association between treatment and outcome that has nothing to do with the effect. Elwert and Winship (2014) call this endogenous selection bias.
          • Bad, bias amplifiers. Pre-treatment variables that affect only adoption. In linear models, controlling for them can amplify any remaining bias (Cinelli et al., 2024).

          The third script builds a collider. After 2020, enrollment rises in program schools because the program attracts families, and it also rises where scores improve. A regression of score changes on the program and FRL share averages 3.03 across 200 samples. Adding the change in enrollment reduces the average estimate to 0.75. Nothing in the regression output signals the problem. The coefficient is precise, and the model fits better.

          The rule that follows is simple to state. Use covariates measured before adoption, or covariates the program cannot affect. Time-varying covariates measured after adoption need a specific argument for why the program does not change them.

          Caetano et al. (2022), a working paper, examine the common practice of adding time-varying covariates to a TWFE regression. Even with only two periods, they show that the regression can mislead when the program affects the covariates, when trends depend on the level of a covariate rather than only its change, when trends or effects depend on time-invariant covariates, when effects vary with the covariates, or when strong linearity assumptions fail. When a covariate is not affected by the program, they propose doubly robust estimators that condition on its values in both periods. When it may be affected, they show conditions under which conditioning on its pre-adoption value is enough, which gives a formal basis for the advice above. Stuart et al. (2014) make the same point for propensity score weighting in DiD, warning against balancing on any variable the program may have affected.

          Simulate · invite a test

          The bad control lab

          Enrollment after adoption depends on the program and on score changes. Set how strongly each one drives enrollment, then compare the estimate with and without enrollment as a control.

          What controlling for a post-adoption variable does

          Each run draws 200 samples of 120 schools, with a true effect of 3.0 points and FRL-specific trends. Bars show the average estimate from a regression of score changes on the program and FRL share, with and without enrollment change. The dashed line marks the true effect.

          2.0
          Percent change in enrollment caused by the program.
          0.4
          Percent change in enrollment per point of score change.
          Try this
          1. Set the score effect to 0. Does controlling for enrollment still bias the estimate?

          2. Set the program effect to 0 and keep the score effect. What happens now, and why?

          3. Raise both. Which one drives the bias more?

          Program and FRL only–average estimate, truth 3.00
          Also enrollment change–average estimate, truth 3.00

          Press the button to run 200 samples.

          Examples · published studies

          Covariates for identification and for precision

          Propensity score weighting in a policy evaluation. Stuart et al. (2014) evaluate the Alternative Quality Contract, a voluntary global-budget payment model that Blue Cross Blue Shield of Massachusetts introduced in 2009, and its effect on enrollees’ out-of-pocket mental health spending. Participation was voluntary, so participating and non-participating groups could differ in ways related to their trends, and the mix of enrollees could change over time. The authors weight all four groups (participants and non-participants, before and after) to resemble participants before the change, using age, sex, co-occurring substance use, and a risk score. They chose these covariates because the program was unlikely to change who enrolled, and they report that weighting removed the standardized differences. They find no evidence of an effect on out-of-pocket spending. Weighting also had a cost. Standard errors for the main coefficient were 35 percent and 67 percent larger than in the unweighted models, for the two parts of their spending model.

          Covariates for precision in an education study. Dynarski (2003) estimates the effect of Social Security student benefits on college attendance by comparing students whose fathers had died with other students, before and after the program ended. The simple DiD estimate is 18.2 percentage points with a standard error of 9.6. Adding family income, parental education, test scores, state of residence, demographics, and their interactions with cohort and group gives 21.9 points with a standard error of 10.2, while the share of variance explained rises from 0.002 to 0.339. The interactions allow, for example, a general decline in college attendance among Black students over the period, which would otherwise be attributed to the program because Black students were more likely to have lost a father. Here covariates mainly check robustness and absorb group-specific changes, and that differs from the role FRL share plays in this lesson.

          Practice

          What to report

          • Why each covariate is there.Say how it relates to adoption and to trends, and confirm it was measured before adoption.
          • The estimator and its models.Report the outcome model, the propensity model, and how covariates enter each.
          • Overlap.Show the distribution of propensity scores by group, and say whether any program schools were trimmed.
          • Balance.Report standardized differences before and after weighting, and the effective number of comparison units.
          • Robustness.Compare outcome regression, IPW, and doubly robust estimates. Large gaps point to misspecification or weak overlap.
          Limitations

          What covariates cannot do

          • Unmeasured differences in trends remain.Conditioning on observed covariates helps only if they capture why trends differ.
          • Double robustness is not a guarantee.If both models are wrong, the doubly robust estimator can be biased too.
          • Overlap failures cannot be fixed by modeling.Where no comparison schools resemble a program school, any estimate for it is an extrapolation.
          • Time-varying covariates are delicate.Values measured after adoption may be outcomes of the program (Caetano et al., 2022).
          • Staggered adoption adds complications.With adoption in different years, use estimators built for that design, which allow covariates (Advanced level).
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Sant'Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101–122. https://doi.org/10.1016/j.jeconom.2020.06.003Doubly robust DiD, the efficiency bound, and the restrictions hidden in TWFE with covariates. Start here.
          • Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Economic Studies, 64(4), 605–654. https://doi.org/10.2307/2971733The conditional DiD extension of matching, tested against experimental data.
          • Caetano, C., Callaway, B., Payne, S., & Rodrigues, H. S. (2022). Difference in differences with time-varying covariates (arXiv:2202.02903) [Working paper]. arXiv. https://doi.org/10.48550/arXiv.2202.02903Time-varying covariates, TWFE, and doubly robust alternatives. Working paper.
          • Abadie, A. (2005). Semiparametric difference-in-differences estimators. Review of Economic Studies, 72(1), 1–19. https://doi.org/10.1111/0034-6527.00321The inverse probability weighting estimator for DiD.
          • Stuart, E. A., Huskamp, H. A., Duckworth, K., Simmons, J., Song, Z., Chernew, M. E., & Barry, C. L. (2014). Using propensity scores in difference-in-differences models to estimate the effects of a policy change. Health Services and Outcomes Research Methodology, 14(4), 166–182. https://doi.org/10.1007/s10742-014-0123-zPropensity score weighting across the four groups of a DiD, with a worked policy evaluation.
          • Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1), 187–199. https://doi.org/10.1093/biomet/asn055Limited overlap and a rule of thumb for trimming.
          • Cinelli, C., Forney, A., & Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research, 53(3), 1071–1104. https://doi.org/10.1177/00491241221099552A visual guide to good, neutral, and bad controls.
          • Elwert, F., & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31–53. https://doi.org/10.1146/annurev-soc-071913-043455How conditioning on a collider creates bias, with social science examples.
          • Dynarski, S. M. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279–288. https://doi.org/10.1257/000282803321455287DiD with covariates and their interactions in an education study.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Advanced · Lesson A1

          When schools adopt at different times.

          Most education policies roll out in waves. This lesson defines the effects of interest when adoption is staggered, opens up the TWFE regression to show which comparisons it makes, and explains why the familiar estimate can be far from any effect of interest, sometimes even in sign, when effects change over time.

          Before thisIntermediate: TWFE, standard errors, event studiesThe 2×2 design, clustering, and reading leads and lags.
          After thisAdvanced: robust estimatorsEstimators that use only clean comparisons.
          By the end

          What you will be able to do

          1. Define cohorts, group-time effects, and the main ways of aggregating them.
          2. State the parallel trends and no-anticipation assumptions for staggered adoption.
          3. Decompose a TWFE estimate into the 2×2 comparisons it averages, and identify the ones that use already-treated units as controls.
          4. Explain negative weights and why TWFE event-study leads can show pre-trends that are not there.
          5. Judge when a published TWFE estimate is likely to be a good summary and when it needs a robust estimator.
          Question

          One regression, many adoption dates

          By 2016, 44 states had passed laws that reformed teacher evaluation, many of them in different years. A researcher who wants to know how these reforms affected the supply of new teachers has a panel of states over time, a treatment that switches on at different dates, and a standard tool, the TWFE regression with state and year fixed effects. Kraft et al. (2020) study exactly this question, and we return to their study at the end of the lesson.

          The shared staggered dataset has the same structure on a small scale. Thirty schools start tutoring in 2018, 30 in 2020, 30 in 2022, and 30 never do. The effects grow with time since adoption. Early adopters gain 4 points in their first year and 1 more point each year after. The 2020 cohort gains 2 points plus 0.5 a year, and the 2022 cohort 1 point plus 0.5 a year. Every effect is positive. Averaged over all treated school-years, the true effect is 4.57 points.

          The TWFE regression estimates 2.80, about 39 percent too small. Nothing is wrong with the data. Parallel trends holds by construction, and there is no anticipation. The problem lies in what the regression compares. With one adoption date, the previous lessons showed that TWFE is the DiD of before and after means. With many adoption dates, it is something else.

          Estimands

          Cohorts and group-time effects

          Call a school’s adoption year its cohort, \(G_i\), with \(G_i = \infty\) for schools that never adopt. Treatment is absorbing. Once a school adopts, it stays treated. Write \(Y_{it}(g)\) for the outcome school \(i\) would have in year \(t\) if it adopted in year \(g\), and \(Y_{it}(\infty)\) for its outcome if it never adopted. The natural building block is the group-time average treatment effect (Callaway & Sant’Anna, 2021).

          \[\text{ATT}(g,t) = E\bigl[\,Y_t(g) - Y_t(\infty) \mid G = g\,\bigr], \qquad t \geq g\]

          That is the effect in year \(t\) for schools that adopted in year \(g\). The shared data contain 15 of them, and a state panel can contain hundreds. Researchers summarize them in a few ways. The overall average weights each treated school-year equally. The dynamic average groups by years since adoption, \(e = t - g\), and gives an event study. The cohort average groups by adoption year. Each answers a different question. A school board deciding whether to keep a program may care about the dynamic path, while a state evaluating a reform may care about the overall average.

          Two assumptions identify each \(\text{ATT}(g,t)\). Parallel trends requires that, without the program, cohort \(g\) would have changed like the comparison group. The comparison group can be the never-treated units or the units not yet treated by year \(t\), and the two choices rest on different assumptions. No anticipation requires that outcomes do not respond before adoption. With these, each \(\text{ATT}(g,t)\) is a simple 2×2 DiD between cohort \(g\) and a clean comparison group, from the year before adoption to year \(t\). The Advanced lesson on robust estimators builds on exactly this.

          The problem

          What the TWFE regression compares

          Goodman-Bacon (2021) shows that the TWFE coefficient is exactly a weighted average of every 2×2 DiD that can be formed from the timing groups in the data. There are three kinds.

          • Treated against never treated. Each cohort compared with never-treated units, before and after its adoption. These are clean.
          • Earlier against not-yet-treated later. An early cohort compared with a later cohort, in the window before the later cohort adopts. These are also clean.
          • Later against already-treated earlier. A later cohort compared with an early cohort that adopted years ago, in the window after the early cohort’s adoption. The “control” group is itself treated.

          The third kind is the source of the problem. Its 2×2 DiD subtracts the early cohort’s change in outcomes, and that change includes the growth in the early cohort’s own treatment effect. If effects are constant over time, that change is zero and the comparison is harmless. If effects grow, the comparison subtracts that growth from the later cohort’s effect.

          The weights are proportional to group sizes and to the variance of the treatment indicator within each pair. That variance is highest for groups treated near the middle of the panel. The weights therefore reflect the timing of adoption relative to the sample period, not the policy relevance of each group.

          The first script computes all nine 2×2 comparisons in the shared data, their weights, and their true values. The three comparisons against never-treated schools are close to the truth, 7.68, 2.69, and 1.50. The three comparisons that use already-treated schools as controls have true values of −0.50, −2.00, and 0.25, although every school-level effect is positive, because the early cohort’s growing effect is being subtracted. Those comparisons carry 22.8 percent of the total weight. The weighted average of all nine is 2.80, exactly the TWFE coefficient.

          Run it yourself in R: code and output
          Another view

          Negative weights on treated school-years

          de Chaisemartin and D’Haultfœuille (2020) look at the same coefficient from the side of the treated cells. Under parallel trends, the TWFE coefficient equals a weighted sum of the effects in every treated school-year, with weights that sum to one but may be negative. A regression with a negative weight on some effects can, in principle, return a negative estimate even when every effect is positive. Their survey found that 19 percent of empirical articles published in the American Economic Review between 2010 and 2012 used two-way fixed effects regressions of this kind.

          The weights are easy to compute. Regress the treatment indicator on school and year fixed effects and keep the residuals for the treated cells. The third script does this for the shared data. Ninety of the 450 treated school-years, 20 percent, receive negative weights, with a combined weight of −0.073. All of them belong to the 2018 cohort, in 2022 to 2024, four to six years after adoption, which are exactly the cells with the largest effects. The weighted sum of the true effects is 2.78. The TWFE coefficient on the data is 2.80, and the gap of 0.02 is the small sampling difference in untreated trends.

          The two decompositions agree on the diagnosis. TWFE is a reliable summary when effects are the same across cohorts and over time, or when there is a single adoption date. With staggered adoption and effects that build up, which is the usual pattern for education programs, it weights effects in ways no analyst would choose and can subtract some of them.

          Event studies

          Leads that show a pre-trend that is not there

          The dynamic TWFE regression, with a lead or lag for each year relative to adoption, seems to avoid the problem because it estimates a separate coefficient for each event time. Sun and Abraham (2021) show that, with staggered adoption, the coefficient on a given lead or lag can be contaminated by effects from other event times, and that apparent pre-trends can arise solely from heterogeneity in treatment effects.

          The second script shows this. It runs the usual event-study regression on the shared data, and then the same regression on the untreated outcomes, which the simulation records. On the untreated outcomes the leads are close to zero, averaging 0.00, as parallel trends requires. On the actual outcomes, the leads range from −0.16 to 1.80 and average 1.00. A reader would conclude that program schools were already improving faster before adoption, and might reject a design that is in fact valid. The lags are also distorted, by up to one point.

          The lesson for applied work is that, with staggered adoption, a TWFE event study can fail in both directions. It can show a pre-trend where there is none, and it can show flat leads where trends do differ. The estimators in the next lesson construct each coefficient from clean comparisons only.

          Simulate · invite a test

          The comparison lab

          Choose how fast effects grow, whether early adopters gain more, and how many schools never adopt. The lab computes every 2×2 comparison and its weight, with no sampling noise, so any gap between TWFE and the true effect comes from the comparisons alone.

          What TWFE averages

          Three equal cohorts adopt in 2018, 2020, and 2022, observed from 2015 to 2024, with parallel trends holding exactly. Bars show each 2×2 comparison’s value. Bar thickness shows its weight in the TWFE coefficient. Gold bars use already-treated schools as controls.

          0.5
          Points added to the effect each year after adoption.
          Cohorts may differ in how much they gain.
          Without them, every comparison relies on timing.
          Try this
          1. Set the yearly growth to 0 and the first-year effect to the same in every cohort. Does TWFE recover the truth?

          2. Keep the growth at 0 but let early adopters gain more. Why is TWFE still off, and why are no bars negative?

          3. Remove the never-treated schools and raise the growth to 1.5. What happens to the gold bars and to TWFE?

          True average effect0.00over all treated school-years
          TWFE coefficient0.00
          Weight on already-treated controls0 percentshare of the TWFE coefficient

          Judgment

          When is the TWFE estimate a fair summary?

          TWFE does not always mislead, and some designs allow the coefficient to be read at face value.

          • One adoption date. With a single cohort and never-treated comparisons, TWFE is the 2×2 DiD of the earlier lessons.
          • Effects that are constant over time. Goodman-Bacon (2021) shows that negative weights arise only when average effects change over time. With constant effects, all weights are positive, although the estimate still weights cohorts by the variance of treatment rather than by their size.
          • Many never-treated units. The more weight falls on comparisons with never-treated units, the less the already-treated comparisons matter. The decomposition reports this share directly.

          Effects that build up over time are common in education. Teachers learn a new curriculum, students accumulate exposure, and reforms take years to reach classrooms. When adoption is staggered and effects may grow, Baker et al. (2022) recommend that researchers use one of the robust estimators. Applying them to published finance and accounting studies, they found that the conclusions in many cases differed substantially from the originals. At a minimum, report the decomposition, so readers can see how much of the estimate comes from already-treated controls.

          Example · published study

          Teacher evaluation reforms and the supply of new teachers

          Kraft et al. (2020) study how high-stakes teacher evaluation reforms affected new teacher supply. Federal incentives, including conditions attached to waivers from No Child Left Behind, pushed states to adopt the reforms, and by 2016, 44 states had done so, in different years. The authors use a state-by-year panel from 2002 to 2016, with state fixed effects and region-by-year fixed effects, so that each reforming state is compared with other states in its region. Standard errors are clustered by state.

          Their event study shows no clear trend in new teaching licenses before the reforms and a decline that starts in the year of implementation and grows over time. The pooled DiD estimate is a reduction of 2.69 licenses per 10,000 adults aged 18 to 65, about 17 percent of the pre-reform mean. They also find that reforms made unfilled vacancies more likely, especially in hard-to-staff schools.

          The design has the two features that this lesson warns about. Adoption is staggered, with few states that never reformed, so much of the identifying variation comes from timing. The effects also appear to grow over time. Under these conditions, the pooled TWFE coefficient can understate an effect that builds up. The authors anticipate part of this. Following Lafortune et al. (2018), they also estimate a model that lets the effect change linearly after the reform, and it shows a significant downward trend in supply after reforms. Their study was written before the robust estimators in the next lesson became standard. Re-estimating it with those estimators would be the natural check today, and we cannot say in advance whether the conclusions would change.

          Practice

          What to report

          • The adoption timeline.Show how many units adopt in each period and how many never adopt.
          • The target estimand.Say whether you report an overall, dynamic, or cohort average, and why.
          • The comparison group.Never treated, not yet treated, or both, and the parallel trends assumption each implies.
          • A diagnostic for TWFE.If you report a TWFE estimate, show the decomposition and the weight on already-treated controls.
          • A robust estimate.With staggered adoption and effects that may change over time, report at least one estimator from the next lesson.
          Limitations

          What this lesson does not solve

          • Diagnosis only.The decompositions explain the TWFE coefficient. They do not by themselves give an unbiased estimate.
          • Parallel trends is still required.Robust estimators fix the comparisons, not the identifying assumption.
          • Treatment that switches off.This lesson assumes adoption is permanent. Programs that start and stop need other estimators (de Chaisemartin & D’Haultfœuille, 2020).
          • Few never-treated units.When almost every unit adopts, late adopters must serve as controls, and the last periods may not be identified.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014The decomposition of TWFE into 2×2 comparisons. Start here.
          • de Chaisemartin, C., & D'Haultfœuille, X. (2020). Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9), 2964–2996. https://doi.org/10.1257/aer.20181169Negative weights on group-period effects, and an alternative estimator.
          • Sun, L., & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006Contamination of TWFE event-study coefficients.
          • Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001Group-time effects and their aggregation.
          • Baker, A. C., Larcker, D. F., & Wang, C. C. Y. (2022). How much should we trust staggered difference-in-differences estimates? Journal of Financial Economics, 144(2), 370–395. https://doi.org/10.1016/j.jfineco.2022.01.004An accessible review with re-analyses of published studies.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008The place of staggered designs in the wider DiD literature.
          • Kraft, M. A., Brunner, E. J., Dougherty, S. M., & Schwegman, D. J. (2020). Teacher accountability reforms and the supply and quality of new teachers. Journal of Public Economics, 188, Article 104212. https://doi.org/10.1016/j.jpubeco.2020.104212The education case in this lesson.
          • Lafortune, J., Rothstein, J., & Schanzenbach, D. W. (2018). School finance reform and the distribution of student achievement. American Economic Journal: Applied Economics, 10(2), 1–26. https://doi.org/10.1257/app.20160567School finance reforms, with a trend-break specification. Used again in the next lessons.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Advanced · Lesson A2

          Estimators that use only clean comparisons.

          The previous lesson showed that TWFE mixes clean comparisons with ones that use already-treated units as controls. This lesson builds the alternatives step by step, from group-time estimates and their aggregation to the interaction-weighted event study and imputation. It then compares their assumptions and precision and shows how to choose among them.

          Before thisAdvanced: staggered adoption and why TWFE failsCohorts, ATT(g, t), and the decomposition of TWFE.
          After thisAdvanced: sensitivity analysisHow conclusions change if parallel trends fails by a bounded amount.
          By the end

          What you will be able to do

          1. Compute group-time effects by hand and aggregate them into overall, dynamic, and cohort summaries.
          2. Explain how the interaction-weighted event study avoids contamination.
          3. Carry out imputation, and recognize that extended TWFE and two-stage DiD give the same estimate.
          4. Compare the parallel trends assumption each estimator needs and what it implies for precision.
          5. Choose an estimator for a given design and report it so that readers can check it.
          The idea

          Separate estimation from aggregation

          The TWFE regression does two jobs at once. It estimates effects, and it averages them, with weights set by the regression rather than by the analyst. The robust estimators separate the two jobs. First, estimate each building block, an effect for one cohort in one year or one event time, using only comparisons that are clean, meaning that the comparison units are untreated in the years being compared. Second, average the building blocks with weights the analyst chooses and states.

          The estimators differ mainly in how they build the blocks. Callaway and Sant’Anna (2021) use one 2×2 DiD per block, anchored in the year before adoption. Sun and Abraham (2021) use one saturated regression that estimates a separate coefficient for every cohort and event time. Borusyak et al. (2024) model untreated outcomes from all untreated observations and impute what treated units would have scored without the program. In the shared staggered data, where the true average effect is 4.57 and TWFE gives 2.80, all of them land between 4.6 and 4.8.

          Estimator 1

          Group-time effects, one 2×2 at a time

          Callaway and Sant’Anna (2021) estimate each \(\text{ATT}(g,t)\) with a 2×2 DiD. The treated group is cohort \(g\). The baseline is year \(g - 1\), the last year before adoption. The comparison group is either the never-treated units or the units not yet treated by year \(t\).

          \[\widehat{\text{ATT}}(g,t) = \bigl(\bar Y_{g,t} - \bar Y_{g,g-1}\bigr) - \bigl(\bar Y_{C,t} - \bar Y_{C,g-1}\bigr)\]

          The first script computes all 15 of these for the shared data with never-treated comparisons. Each is close to its true value. For the 2018 cohort, the estimates rise from 3.84 in 2018 to 10.99 in 2024, against true values from 4.00 to 10.00. Because each block uses only cohort \(g\) and units that are untreated in both years, no block can subtract another cohort’s treatment effect.

          The blocks are then aggregated. The overall average weights each block by the size of its cohort, which here, with equal cohorts, means averaging over treated school-years. It gives 4.64 against a true 4.57. Using not-yet-treated schools as the comparison gives 4.70. The dynamic aggregation averages blocks with the same number of years since adoption and gives an event study. Its estimates after adoption track the truth closely, for example 2.00 against 2.33 in the first year and 5.38 against 5.25 in the fourth. The cohort aggregation gives 7.35, 3.10, and 0.90 for the 2018, 2020, and 2022 cohorts, against true values of 7.00, 3.00, and 1.50.

          For years before adoption, the default compares consecutive years, so each pre-period estimate is a placebo DiD over one year. In the shared data these range from −0.38 to 0.42 around a true value of zero. Because they compare consecutive years rather than a fixed reference year, they are not on the same footing as the post-adoption estimates, and a plot should say which baseline it uses.

          Two further features matter in practice. The method accepts covariates through the doubly robust estimators of the covariates lesson, so conditional parallel trends can be used cohort by cohort. Its standard errors can be computed with a bootstrap that also gives uniform bands for the event study, as in the event-study lesson.

          Run it yourself in R: code and output
          Estimator 2

          The interaction-weighted event study

          Sun and Abraham (2021) keep the regression form that applied researchers know but remove the source of contamination. Instead of one lead or lag coefficient per event time, the regression includes a separate coefficient for every combination of cohort and event time, with never-treated units, or the last cohort to adopt, as the comparison. Each cohort’s event-time path is then estimated without borrowing from other cohorts. The estimator averages these cohort-specific coefficients at each event time, weighting each cohort by its share of the units observed at that event time.

          Because each cohort-specific coefficient is a clean comparison, the averaged leads are no longer contaminated by effects from other periods. With never-treated comparisons, a balanced panel, and no covariates, the estimates equal those of Callaway and Sant’Anna with a fixed reference year. Its main appeal is practical. It runs as one regression in fixest (sunab) and can be presented exactly like a conventional event study.

          Estimator 3

          Imputation, extended TWFE, and two-stage DiD

          Borusyak et al. (2024) start from a model of what outcomes would be without treatment, \(Y_{it}(0) = \alpha_i + \lambda_t + \varepsilon_{it}\). The estimator has three steps.

          1. Estimate the school and year effects using only untreated observations, meaning never-treated schools and adopters before they adopt.
          2. For every treated school-year, predict the untreated outcome, \(\hat Y_{it}(0) = \hat\alpha_i + \hat\lambda_t\), and take the difference from the observed outcome.
          3. Average these differences with the weights that define the target, for example over all treated school-years or by years since adoption.

          The second script does this for the shared data and gets 4.804. It then estimates two other estimators that look different. Extended TWFE (Wooldridge, 2025) replaces the single treatment dummy with a separate dummy for every treated cohort-year cell, so the regression no longer imposes a common effect. Averaging its cell coefficients over treated school-years gives 4.804. Two-stage DiD (Butts & Gardner, 2022) subtracts the step-one school and year effects from every outcome and regresses the residual on treatment. It also gives 4.804. Without covariates the three are the same estimator, so a reader who meets them under different names can treat them as one.

          Imputation uses every untreated observation to learn the school and year effects. That makes it efficient when its model is right. Borusyak et al. (2024) show it is the most efficient estimator when parallel trends holds in all periods and the errors are homoskedastic and uncorrelated over time. The price is a stronger assumption. Callaway and Sant’Anna’s estimator relies on parallel trends from the year before adoption onward, while imputation relies on it across the whole untreated history, including early years far from adoption. Dube et al. (2025) note that when errors are strongly autocorrelated, an estimator anchored in the year before adoption can be more precise.

          Imputation also changes how pre-trends are checked. Since all pre-adoption observations are used to fit the model, the pre-period fit cannot serve as a placebo for the same model. Borusyak et al. recommend a separate test that adds pre-adoption event-time indicators to the model for untreated outcomes.

          Evidence

          Bias and precision across 200 samples

          One sample cannot rank estimators, so the third script draws 200 new samples from the staggered design and records each estimator’s error, the estimate minus that sample’s true average effect. TWFE is off by −1.80 on average, with a standard deviation of 0.31. The robust estimators are close to unbiased, with average errors of 0.00 for Callaway–Sant’Anna with never-treated comparisons, −0.02 with not-yet-treated comparisons, and −0.03 for imputation. Their standard deviations are 0.44, 0.42, and 0.39.

          Two lessons follow. First, TWFE’s small standard deviation is no comfort. A precise estimate of the wrong quantity is still wrong. Second, among the robust estimators the differences in precision are modest in this design. Imputation is the most precise, as theory predicts when its stronger assumption holds, and using not-yet-treated units adds comparisons and slightly reduces variance. In real data the choice should be driven first by which parallel trends assumption is credible, and only then by precision.

          Inference

          Standard errors for robust estimators

          The lessons on standard errors carry over directly. Cluster at the level at which treatment is assigned, which in staggered designs is usually the state or district that adopts. A building block for one cohort uses only that cohort’s units, so a block from a cohort of three states rests on three clusters, however many students those states contain. Aggregates that combine many cohorts are more stable than any single block.

          For event studies, report uniform bands rather than only pointwise intervals, because a reader judges the whole path at once. Callaway and Sant’Anna (2021) compute uniform bands with a multiplier bootstrap that resamples clusters. When there are few adopting clusters overall, the problems of few treated clusters from the standard errors lesson apply, and a design-based or permutation approach may be needed.

          The aggregation weights are estimated too, because cohort shares are estimated from the sample. Software that implements these estimators accounts for this. Hand calculations like the ones in this lesson are useful for understanding the estimators, but their standard errors should come from the packages.

          Two more

          Stacked DiD and local projections

          Stacked DiD. Cengiz et al. (2019), studying minimum wages, build a separate clean dataset for each adoption event, with the treated units plus units not treated within an event window, stack them, and run one regression with dataset-specific fixed effects. Wing et al. (2024), a working paper, show that unweighted stacking targets an unclear mix of effects and propose weights that recover a well-defined average. The design is easy to explain and to audit, because every comparison is visible in the stacked data.

          Local projections DiD. Dube et al. (2025) estimate, separately for each horizon \(h\), a regression of the change in the outcome from the year before adoption to \(h\) years after, using only newly treated units and units still untreated at that horizon. The estimator is a sequence of simple regressions, and a reweighted version reproduces the Callaway–Sant’Anna estimates.

          The estimator comparison lab in this level runs these estimators side by side on the shared data.

          Judgment

          Choosing among them

          • Where do you trust parallel trends?If only near adoption, anchor in the year before adoption (Callaway–Sant’Anna, local projections). If across the whole untreated history, imputation is more efficient.
          • Which comparison group is credible?Never-treated units may differ in kind, for example states that never faced a court order. Not-yet-treated units are often more similar but must not anticipate adoption.
          • Do you need covariates?Callaway–Sant’Anna supports doubly robust adjustment cohort by cohort.
          • Does treatment switch off, repeat, or vary in dose?Use estimators designed for that case, such as those of de Chaisemartin and D’Haultfœuille (2020, 2026).
          • What do readers need to see?Sun–Abraham and stacked designs present results in familiar regression form.

          Report one main estimator and show that at least one other gives a similar answer. When they disagree, the disagreement is informative. It may mean that parallel trends holds near adoption but not in earlier years, or that the comparison groups differ.

          Example · published study

          School finance reform and student achievement

          Lafortune et al. (2018) study school finance reforms adopted after 1990, many in response to court orders, and their effects on spending and achievement in low-income districts. Twenty-six states implemented or were ordered to implement reforms in this period, at different dates, while 23 did not. Achievement comes from the National Assessment of Educational Progress, which tests representative samples of fourth and eighth graders in every state with the same instrument, from 1990 to 2011.

          The identifying assumption is stated plainly. The exact timing of reforms is treated as good as random, an argument the authors base on the idiosyncrasies of judicial processes, so that states without a reform in a given year can serve as comparisons for states with one. Their event-study specification includes a jump at the reform, a phase-in slope that lets effects build up, and a pre-reform trend term that serves as a falsification test. They find no sign of changes in funding or test scores before reforms. After reforms, funding in low-income districts rises by about $1,200 per pupil per year in absolute terms and $700 relative to high-income districts. Relative achievement in low-income districts rises gradually, by about 0.1 standard deviation after ten years, roughly one-fifth of the baseline gap.

          The study connects the two Advanced lessons. Its effects build up over time and its reforms are staggered, the conditions under which a single TWFE coefficient can mislead. The authors address this by modeling the phase-in directly rather than imposing one constant effect, and their pre-trend term tests the timing assumption. A robust estimator that builds each event-time effect from clean comparisons would be a natural complement today. It would relax the linear form of the phase-in, at some cost in precision with 49 states.

          Practice

          What to report

          • The building blocks and the weights.Name the estimator, the comparison group, the baseline year, and how blocks are aggregated.
          • The event study.Plot the dynamic aggregation with uniform bands and state the baseline used for pre-periods.
          • A second estimator.Show that a robust estimator with a different assumption gives a similar answer, or explain why it does not.
          • Cohort detail.Report cohort-specific estimates when cohorts differ in size or context, as early adopters often do.
          • Software and options.Report the package, version, and options, since defaults differ.
          Limitations

          What the robust estimators do not solve

          • They still need parallel trends.They fix the comparisons, not the assumption. The next lesson asks how much a violation would change conclusions.
          • Some effects are not identified.Without never-treated units, effects in the last periods, after everyone has adopted, cannot be estimated.
          • Small cohorts are noisy.Each building block uses one cohort, so small cohorts give imprecise blocks.
          • Aggregation is a choice.Overall, dynamic, and cohort averages answer different questions and can differ substantially.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001Group-time effects, comparison groups, and aggregation. Start here.
          • Sun, L., & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006The interaction-weighted event study.
          • Borusyak, K., Jaravel, X., & Spiess, J. (2024). Revisiting event-study designs: Robust and efficient estimation. Review of Economic Studies, 91(6), 3253–3285. https://doi.org/10.1093/restud/rdae007Imputation, its efficiency, and separate pre-trend tests.
          • Wooldridge, J. M. (2025). Two-way fixed effects, the two-way Mundlak regression, and difference-in-differences estimators. Empirical Economics, 69(5), 2545–2587. https://doi.org/10.1007/s00181-025-02807-zExtended TWFE and its equivalence with imputation.
          • Butts, K., & Gardner, J. (2022). did2s: Two-stage difference-in-differences. The R Journal, 14(3), 162–173. https://doi.org/10.32614/rj-2022-048Two-stage DiD and the did2s package.
          • Cengiz, D., Dube, A., Lindner, A., & Zipperer, B. (2019). The effect of minimum wages on low-wage jobs. Quarterly Journal of Economics, 134(3), 1405–1454. https://doi.org/10.1093/qje/qjz014Stacked event-by-event comparisons in a minimum-wage study.
          • Wing, C., Freedman, S. M., & Hollingsworth, A. (2024). Stacked difference-in-differences (NBER Working Paper No. 32054). National Bureau of Economic Research. https://doi.org/10.3386/w32054Weights for stacked DiD. Working paper.
          • Dube, A., Girardi, D., Jordà, Ò., & Taylor, A. M. (2025). A local projections approach to difference-in-differences. Journal of Applied Econometrics, 40(7), 741–758. https://doi.org/10.1002/jae.70000Local projections DiD.
          • de Chaisemartin, C., & D'Haultfœuille, X. (2026). Difference-in-differences estimators of intertemporal treatment effects. Review of Economics and Statistics, 108(4), 863–880. https://doi.org/10.1162/rest_a_01414Estimators for treatments that switch on and off or vary in dose.
          • Lafortune, J., Rothstein, J., & Schanzenbach, D. W. (2018). School finance reform and the distribution of student achievement. American Economic Journal: Applied Economics, 10(2), 1–26. https://doi.org/10.1257/app.20160567The education case in this lesson.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Advanced · Lesson A3

          What if parallel trends is only roughly true?

          Every DiD estimate rests on parallel trends, and no test can confirm it. This lesson replaces the yes-or-no question with a more useful one, namely how large a violation it would take to overturn the conclusion. It introduces the bounds of Rambachan and Roth, shows how to compute and read them, and explains how to report them.

          Before thisAdvanced: robust estimatorsEvent-study estimates built from clean comparisons.
          After thisExpert: doubly robust and machine-learning DiDFlexible adjustment for covariates.
          By the end

          What you will be able to do

          1. Explain why a post-period effect is only partially identified when parallel trends may fail.
          2. State the relative-magnitudes and smoothness restrictions and what each assumes about confounding.
          3. Compute identified sets and read a sensitivity plot, including the breakdown value.
          4. Choose restrictions that match the substantive threat to a design.
          5. Report a sensitivity analysis that lets readers apply their own judgment.
          Question

          From “does it hold?” to “how wrong could it be?”

          The event-study lesson showed that a passed pre-trend test is weak evidence. Tests often lack power, and studies that pass a pre-test can be more biased than studies in general (Roth, 2022). Rambachan and Roth (2023) note that researchers are therefore often unsure whether parallel trends holds, and they propose a way forward that does not require certainty.

          The idea is to allow parallel trends to fail by an amount that is bounded, and to state the bound in terms the reader can judge. If post-adoption violations are assumed to be no larger than those seen before adoption, what range of effects is consistent with the data? If the difference in trends is assumed to change slowly, what then? Instead of one estimate that is right only if an untestable assumption holds exactly, the analysis gives a set of estimates, one for each degree of possible violation. The reader can see what conclusions survive.

          This is a sensitivity analysis in the tradition of partial identification, building on work by Manski and Pepper described in Rambachan and Roth (2023). It does not replace design reasoning. It makes the consequences of that reasoning explicit.

          Framework

          Event-study coefficients as effect plus bias

          Write the event-study coefficient for event time \(e\) as the sum of a treatment effect and a difference in untreated trends, \(\beta_e = \tau_e + \delta_e\), with the reference year normalized so that \(\delta_{-1} = 0\). Before adoption, the effect is zero under no anticipation, so the leads estimate \(\delta_e\) directly. After adoption, the data reveal only the sum. Parallel trends is the assumption that every post-period \(\delta_e\) is zero.

          \[\beta_e = \tau_e + \delta_e, \qquad \tau_e = 0 \text{ for } e < 0, \qquad \delta_{-1} = 0\]

          If, instead, \(\delta\) is only known to lie in some set \(\Delta\), the effect \(\tau_e\) is partially identified. Its identified set is every value \(\beta_e - \delta_e\) consistent with a \(\delta\) in \(\Delta\) that also matches the observed pre-period coefficients. The choice of \(\Delta\) is where the researcher’s knowledge of the setting enters. Rambachan and Roth propose several families, each indexed by one number that controls how much violation is allowed. Two are most used.

          Restriction 1

          Relative magnitudes

          \[\Delta^{RM}(\bar M) = \bigl\{\delta : |\delta_{t+1} - \delta_t| \le \bar M \cdot \max_{s<0} |\delta_{s+1} - \delta_s| \text{ for all } t \ge 0\bigr\}\]

          The relative-magnitudes restriction bounds how much the difference in trends can change from one post-adoption year to the next, as a multiple \(\bar M\) of the largest such change seen before adoption. It fits settings where the threat is confounding shocks, and there is no reason for shocks after adoption to be much larger than shocks before it. With \(\bar M = 0\), parallel trends holds exactly after adoption. Rambachan and Roth suggest \(\bar M = 1\) as a natural benchmark when the numbers of pre and post periods are similar. It allows post-adoption shocks as large as the worst pre-adoption shock in every year.

          Because the bound applies to each year-to-year change, the possible bias accumulates with distance from the reference year. Five years after the reference year, the bias can be up to \(5 \bar M\) times the largest pre-period change. Effects far from adoption are therefore less tightly bounded than effects close to it.

          Restriction 2

          Smoothness

          \[\Delta^{SD}(M) = \bigl\{\delta : |(\delta_{t+1} - \delta_t) - (\delta_t - \delta_{t-1})| \le M \text{ for all } t\bigr\}\]

          The smoothness restriction fits a different threat, a secular trend that differs between groups and evolves gradually, such as long-run changes in enrollment, demographics, or labor supply. It bounds how much the slope of the difference in trends can change between consecutive years. With \(M = 0\), the difference in trends must be exactly linear, which corresponds to the common practice of adding group-specific linear trends or extrapolating the pre-trend, as in the event-study lesson. Larger values of \(M\) allow the trend to bend.

          The two restrictions answer different worries. Relative magnitudes is about the size of shocks, and smoothness is about the shape of trends. A design threatened by both can combine them, and Rambachan and Roth also describe restrictions on the sign or direction of the bias, for example when the bias is believed to push the estimate upward.

          Example · shared school data

          Two datasets, two sets of bounds

          The first script applies both restrictions to the effect four years after adoption, whose true value is 4.0, in two versions of the shared single-adoption data. To keep the logic visible, it computes identified sets from the estimated coefficients and widens them by 1.96 standard errors of the estimate. This is a simplified illustration. The HonestDiD package for R implements the confidence sets of Rambachan and Roth, which account for sampling error in the pre-period coefficients and have proven coverage.

          When parallel trends holds. The estimate is 4.52 with a standard error of 0.66, and the largest change in the leads between consecutive years is 0.35, which is sampling noise. Under relative magnitudes with \(\bar M = 1\), the identified set is 2.79 to 6.26 and the wider interval is 1.50 to 7.55. The breakdown value, the smallest \(\bar M\) at which the interval includes zero, is 1.87. A reader would have to believe that post-adoption shocks were almost twice as large as the largest pre-adoption change before doubting that the effect is positive.

          When program schools improve 0.3 points a year faster. The estimate rises to 6.02, too high by 2.0. The leads now show the violation, and their largest change is 0.65. With \(\bar M = 0\), the interval of 4.74 to 7.31 misses the truth. With \(\bar M = 1\), the identified set widens to 2.79 to 9.26 and contains it. The breakdown value falls to 1.46. Under smoothness with \(M = 0\), the linear extrapolation removes the linear violation exactly, and the set collapses to 4.78 in both datasets, because the violation in the second dataset is precisely linear. The extrapolation uses only the last pre-period slope, so it is noisy, which is why valid inference needs the full procedure.

          The second script repeats the second scenario in 200 samples. The pre-test passes in 67 percent of them. The conventional 95 percent interval contains the true effect in only 36 percent of samples, and in 27 percent of those that pass the pre-test. The relative-magnitudes interval with \(\bar M = 1\) contains it in all 200. The price is width. Robustness to violations always costs precision, and the analysis makes that trade-off visible rather than hiding it.

          Run it yourself in R: code and output
          Simulate · invite a test

          The sensitivity lab

          Pick a dataset and a restriction, then move the parameter. The plot shows how the interval for the effect four years after adoption widens, where it first includes zero, and whether it contains the true effect of 4.0.

          How much violation can the conclusion withstand?

          Shaded band: identified set widened by 1.96 standard errors, for every parameter value from zero to the maximum. Dashed gold line: true effect. The vertical line marks the parameter you selected. The inputs come from the first script.

          1.00
          M-bar from 0 to 2.
          Try this
          1. With the violated dataset and relative magnitudes, find the smallest M-bar at which the interval contains 4.0.

          2. Switch to smoothness. Why does the interval at zero now contain the truth?

          3. Compare the breakdown values in the two datasets. Why is it lower when trends diverge?

          Interval at this value–effect four years after adoption
          Contains the true 4.0?–
          Breakdown value–where the interval first includes 0

          Extensions

          Signs, shapes, and how this fits with pre-tests

          Relative magnitudes and smoothness are not the only options. Applied researchers often argue informally that a violation can only push the estimate in one direction, or that pre-existing trends would have continued in the same direction. Rambachan and Roth (2023) formalize such arguments as sign and monotonicity restrictions, which can be combined with the bounds above and can narrow them considerably when they are credible. They note, for example, that monotone violations of parallel trends are often discussed in applied work.

          Sensitivity analysis also changes the role of the pre-test. A pre-test asks whether the leads are statistically distinguishable from zero, a question whose answer depends heavily on sample size. The equivalence bounds of Dette and Schumann (2024), covered in the event-study lesson, ask how large the pre-period violations could be. Sensitivity analysis asks what those violations would imply for the effect after adoption. The three are complementary. The first is the least informative on its own, and the last connects most directly to the conclusion a reader cares about.

          Finally, the analysis should be applied to the right event-study coefficients. With staggered adoption, apply it to estimates from a robust estimator, so that the leads measure pre-trends and not contamination from other periods, as the first Advanced lesson showed.

          Interpretation

          Reading a sensitivity plot

          A sensitivity analysis is usually shown as a plot. The horizontal axis is the restriction parameter, \(\bar M\) or \(M\), starting at zero. For each value, the plot shows a confidence interval for the effect of interest. At zero, the interval is the conventional one, or close to it. As the parameter grows, the interval widens. The breakdown value is where the interval first includes zero, or whatever value would change the conclusion.

          The breakdown value is a summary a reader can judge without accepting the author’s choice of parameter. A breakdown value of 2 under relative magnitudes says that the conclusion holds unless post-adoption confounding is more than twice as large as anything seen before adoption. Whether that is reassuring depends on the setting. If the pre-period was calm and the post-period included a major disruption, such as school closures, a value of 2 may not be reassuring at all.

          Rambachan and Roth recommend reporting confidence sets under restrictions motivated by economic or substantive knowledge, together with formal sensitivity analyses over a range of restrictions. Both make clear which assumptions are needed for a particular conclusion.

          Judgment

          Choosing restrictions in education studies

          • Shocks around adoption.A district adopts a program after a bad testing year, or during a leadership change. Relative magnitudes, benchmarked on pre-period shocks, fits this threat.
          • Diverging long-run trends.Program schools serve growing populations, or gentrifying neighborhoods. Smoothness, which allows the trend gap to bend gradually, fits this threat.
          • A known direction of bias.If schools adopt when they are already improving, the bias is likely positive. A sign restriction can sharpen the bounds.
          • Calibration.Translate the parameter into substantive units. A smoothness parameter of 0.05 scale points per year can be compared with the typical yearly change in scores.
          • Staggered adoption.Apply the analysis to event-study estimates from a robust estimator, not to a contaminated TWFE event study. Rambachan and Roth note that their method can be used with such estimators.
          Example · published study

          Teacher collective bargaining and adult outcomes

          Rambachan and Roth (2023) illustrate their method with a study of state duty-to-bargain laws, which required school districts to bargain in good faith with teachers’ unions (Lovenheim and Willen, 2019, as cited in Rambachan & Roth, 2023). The study compares adults across states and birth cohorts, using the staggered timing of the laws, and asks how exposure to the laws as a student affected labor market outcomes later in life. The authors of the original study name their main concern as secular trends that differ systematically with treatment, such as changes in labor supply or educational attainment.

          Because the threat is a slowly evolving trend, Rambachan and Roth apply the smoothness restriction to the effect on employment 15 years after a law passed. For men, the leads are close to zero and the later effects are negative. The robust confidence sets resemble the conventional ones when only nearly linear violations are allowed, but widen as more curvature is allowed, and the breakdown value for a significant effect is about \(M = 0.01\). For women, the leads show a downward pre-existing trend. Allowing for a linear violation changes the picture substantially. For small values of \(M\), the robust set contains only positive values, because the estimate lies above the extrapolated negative trend.

          The example shows what sensitivity analysis adds. Under the same smoothness assumption for both groups, the data imply either significant effects of opposite sign for men and women or effects that cannot be distinguished from zero for both. A single pair of conventional estimates hides that tension. Rambachan and Roth then calibrate \(M\) against plausible confounders, which lets readers judge whether a deviation of that size is credible.

          Practice

          What to report

          • The target.Name the effect being bounded, for example the effect four years after adoption or the average post-period effect.
          • The restriction and its motivation.Explain why relative magnitudes, smoothness, or a sign restriction fits the threat.
          • A sensitivity plot.Show confidence sets over a range of parameter values, starting at zero.
          • The breakdown value.Report the parameter value at which the conclusion changes, with a substantive interpretation.
          • Software.Report the package and options, for example HonestDiD with the confidence-set method used.
          Limitations

          What sensitivity analysis cannot do

          • It moves the assumption, it does not remove it.Every restriction is still an assumption about what is unobserved.
          • Pre-periods are the benchmark.With few or noisy pre-periods, the benchmark itself is uncertain.
          • Bounds widen with distance from adoption.Long-run effects are hard to bound tightly.
          • The simplified intervals here are illustrations.Use the published procedures for inference in applied work.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591. https://doi.org/10.1093/restud/rdad018Restrictions on violations of parallel trends, valid confidence sets, and two applications. Start here.
          • Roth, J. (2022). Pretest with caution: Event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305–322. https://doi.org/10.1257/aeri.20210236Why pre-tests are weak evidence, the motivation for sensitivity analysis.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008Where sensitivity analysis fits in current DiD practice.
          • Dette, H., & Schumann, M. (2024). Testing for equivalence of pre-trends in difference-in-differences estimation. Journal of Business & Economic Statistics, 42(4), 1289–1301. https://doi.org/10.1080/07350015.2024.2308121Equivalence bounds for pre-trends, a complementary summary.
          • Kahn-Lang, A., & Lang, K. (2020). The promise and pitfalls of differences-in-differences: Reflections on 16 and Pregnant and other applications. Journal of Business & Economic Statistics, 38(3), 613–620. https://doi.org/10.1080/07350015.2018.1546591Design reasoning about why trends might diverge.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Expert · Lesson X1

          Doubly robust and machine-learning DiD.

          When parallel trends holds only conditional on many covariates, simple regression adjustment can fail and hand-built models become guesswork. This lesson develops the doubly robust estimator in full, explains Neyman orthogonality and cross-fitting, and shows how they let flexible learners estimate the nuisance functions without biasing the effect.

          Before thisIntermediate: covariates and conditional parallel trendsOutcome regression, weighting, and the basic doubly robust estimator.
          After thisExpert: functional form and changes-in-changesWhen the scale of the outcome matters.
          By the end

          What you will be able to do

          1. Write the doubly robust DiD estimand for panel data and explain each of its parts.
          2. Explain double robustness and local efficiency, and what each requires.
          3. Define Neyman orthogonality and explain why it protects the estimate from errors in the nuisance models.
          4. Carry out cross-fitting and explain why it is needed with flexible learners.
          5. Plan a machine-learning DiD analysis for administrative school data, including diagnostics.
          Question

          Many covariates, unknown functional forms

          State administrative data give researchers dozens of school characteristics measured before a program starts, such as enrollment, demographic shares, prior achievement in several subjects, attendance, staffing, and locale. Suppose a state offers a turnaround grant, and schools that apply differ from those that do not in ways that also shape how their scores would have changed. Conditional parallel trends may be credible given these covariates. The difficulty is that nobody knows how they enter. Trends may depend on prior achievement nonlinearly, or on the interaction of poverty and enrollment change.

          The covariates lesson showed that a TWFE regression with linear covariates imposes restrictions beyond conditional parallel trends, and that outcome regression and weighting each depend on one model being right. With many covariates, choosing those models by hand invites both misspecification and specification searching. Machine learning offers flexible fits, but plugging flexible fits naively into an estimator can bias the effect. This lesson explains how the doubly robust estimator, combined with cross-fitting, makes flexible fits usable (Chernozhukov et al., 2018; Chang, 2020).

          Estimand

          The doubly robust estimand, piece by piece

          With panel data, write \(\Delta Y\) for each unit’s change in outcome, \(\mu_0(X) = E[\Delta Y \mid X, D = 0]\) for the expected change among comparison units with covariates \(X\), and \(p(X) = P(D = 1 \mid X)\) for the propensity score. Sant’Anna and Zhao (2020) give the doubly robust estimand

          \[\tau^{DR} = E\Bigl[\bigl(w_1(D) - w_0(D, X)\bigr)\bigl(\Delta Y - \mu_0(X)\bigr)\Bigr], \quad w_1 = \frac{D}{E[D]}, \quad w_0 = \frac{\frac{p(X)(1-D)}{1-p(X)}}{E\Bigl[\frac{p(X)(1-D)}{1-p(X)}\Bigr]}\]

          Read it in two steps. The term \(\Delta Y - \mu_0(X)\) is each unit’s change minus the change predicted for an untreated unit with the same covariates. For program units it is an estimate of their effect, and for comparison units it is a residual that should average zero. The weights then average these terms, program units with equal weight and comparison units reweighted by the odds of treatment, so that the comparison residuals are evaluated where program units are.

          If the outcome model is correct, the comparison residuals average zero under any weights, and the estimand reduces to outcome regression. If the propensity score is correct, the weighting balances the covariates, and any error in \(\mu_0\) cancels between the two groups. Either way the estimand equals the ATT. This is double robustness. Sant’Anna and Zhao also show that when both models are correct, the estimator attains the semiparametric efficiency bound, so nothing is lost by using it. They further show that panel data allow more efficient estimation than repeated cross-sections, and that with suitable choices of how the nuisance models are estimated, the estimator can also be doubly robust for inference.

          Theory

          Why errors in the nuisance models matter less

          The propensity score and outcome model are nuisance functions. They are not the target, but the estimate depends on them. Chernozhukov et al. (2018) explain that machine-learning methods estimate such functions well by trading off regularization bias against overfitting, and that both kinds of error can bias an estimator that simply plugs in the fitted functions. The bias shrinks more slowly than the usual rate, so confidence intervals built in the usual way are wrong.

          Their solution has two parts. The first is a Neyman-orthogonal estimating equation, one whose expected value does not change, to first order, when the nuisance functions are slightly wrong. Small errors in \(\hat p\) or \(\hat\mu_0\) then affect the estimate only through their product, which is much smaller than either error alone. The doubly robust DiD score has this property. Chang (2020) builds the corresponding orthogonal score for Abadie’s semiparametric DiD and shows that it allows lasso, random forests, boosting, neural networks, and other learners to be used for the nuisance functions, including when the number of potential covariates exceeds the sample size.

          The intuition is visible in the formula above. An error in the outcome model shifts every residual, but the weights make the shift cancel between program and comparison units. An error in the weights changes how residuals are averaged, but if the residuals are centered correctly, reweighting them changes little. Only when both are wrong in related ways does the error survive.

          Method

          Cross-fitting

          The second part is cross-fitting. A flexible learner fitted and evaluated on the same units can overfit, and its fitted values then contain part of each unit’s own noise. Cross-fitting removes this dependence.

          1. Split the units at random into \(K\) folds, typically five.
          2. For each fold, fit the propensity score and the outcome model on the other \(K - 1\) folds, and predict for the units in this fold.
          3. Plug the out-of-fold predictions into the doubly robust formula and average over all units.
          4. Optionally, repeat with several random splits and combine the results to reduce dependence on one split.

          Every unit’s nuisance values then come from a model that never saw that unit. Combined with orthogonality, this yields an estimator that is approximately normal and whose standard error can be computed from the estimated score, under conditions on how fast the learners converge (Chernozhukov et al., 2018). Chang (2020) applies this to DiD with a cross-fitting algorithm and illustrates it with the effect of tariff reductions on corruption.

          Example · simulation

          Four estimators, one nonlinear world

          The first script simulates 1,000 schools with six covariates. Selection into the program depends on a squared term and an interaction. The untreated trend depends on the same nonlinear terms, and the effect varies with the first covariate. In 100 samples, it compares four estimators of the ATT. The learners are ridge regressions on all squares and pairwise products of the covariates, written out so that R, Python and Stata give identical numbers. In applied work, a library of learners such as lasso, random forests, and boosting would take their place.

          The simple DiD is off by 2.10 points on average. A regression of the change on treatment and linear covariates, which mimics TWFE with covariate-by-period terms, is off by 1.02. The doubly robust estimator with linear models for both nuisances is off by 1.24. Double robustness offers no protection when both models are wrong. The doubly robust estimator with flexible models and five-fold cross-fitting is off by 0.02. The standard deviations are similar, 0.15 to 0.16, so the gain is in bias, not precision.

          The second script isolates orthogonality. With a heavy penalty, the flexible learners are strongly regularized toward zero. Plugging the regularized outcome model directly into outcome regression gives an average error of 0.74. The doubly robust estimator, using the same regularized fits, reduces it to 0.47 without cross-fitting and 0.54 with it. Orthogonality reduces the damage from regularization bias but does not remove it when the nuisance errors are large. With light regularization, all three are close to unbiased. In this low-dimensional design overfitting is mild, so cross-fitting makes little difference here. Its value grows with the flexibility of the learners and the number of covariates.

          Run it yourself in R: code and output
          Example · published study

          Tariff cuts and corruption, re-estimated

          Chang (2020) illustrates the method by revisiting a study by Sequeira of trade between South Africa and Mozambique from 2006 to 2014. In 2008, tariffs on some goods fell sharply while others were unaffected, a natural experiment for asking whether lower tariffs reduce bribery at the border. The original study used a linear DiD.

          Chang’s re-analysis shows how the estimators in this lesson can change conclusions about magnitude. Adding interactions between treatment and the control variables to the linear model made the estimate statistically indistinguishable from zero, which suggests heterogeneous effects that the linear specification averages in an unclear way. Abadie’s semiparametric weighting estimator, with nonparametric first steps, gave an effect at least twice as large as the original, with the same sign. Chang attributes part of that size to the lack of robustness of the weighting estimator and to finite-sample bias in its first step. The orthogonal machine-learning estimator gave an effect about 60 percent larger than the original estimate. Its conclusion matches the original study, that lower tariffs reduced corruption, with a larger magnitude.

          Three lessons carry over to education research. The linear model’s estimate can hide heterogeneous effects. Non-orthogonal estimators with flexible first steps can be unstable in moderate samples. And reporting the estimates side by side lets readers see how much the conclusion depends on functional form, which is the purpose of the robustness comparison recommended above.

          Inference

          Standard errors and confidence intervals

          Because the doubly robust score is orthogonal and the nuisance functions are cross-fitted, the estimator behaves, to first order, like an average of the score evaluated at the true nuisance functions. Its variance can therefore be estimated from the sample variance of the estimated score, with clustering at the level of treatment assignment when units within a cluster are correlated (Chernozhukov et al., 2018). A multiplier bootstrap based on the same score gives uniform bands for event studies with many group-time effects, as in the Advanced lessons.

          Two practical points matter. First, the variance formula assumes the learners converge fast enough. A learner that fits poorly, for example because the sample is small relative to the number of covariates, can leave bias that the standard error does not reflect. Second, randomness from the sample split adds variation that the standard error ignores. Chernozhukov et al. (2018) recommend repeating the split several times and combining the estimates, for example by taking the median, and reporting the combined estimate.

          Practice

          Applying it to school data

          • Choose covariates by design, not by fit.Include pre-adoption characteristics that plausibly drive both adoption and trends. Machine learning chooses functional forms, not which variables are legitimate controls. Variables affected by the program remain bad controls.
          • Use pre-adoption outcomes carefully.Lagged outcomes can be strong predictors of adoption, but conditioning on them changes the identifying assumption. Caetano et al. (2022) discuss when this is justified.
          • Check overlap after fitting.Flexible propensity models can push scores toward 0 or 1. Inspect their distribution and trim or restrict the sample to the region of overlap, as in the covariates lesson.
          • Use an ensemble.Combining several learners guards against relying on one that happens to fit poorly. Report which learners were used and how they were tuned.
          • Cluster appropriately.When treatment is assigned to districts, form folds and compute standard errors at the district level, so that units from one cluster are not split across training and evaluation.
          • Compare with simpler estimators.Report the outcome regression, weighting, and parametric doubly robust estimates alongside the machine-learning one. Large differences point to nonlinearity or limited overlap.
          Extensions

          Staggered adoption and repeated cross-sections

          The estimand above is for two periods. With staggered adoption, the same doubly robust construction is applied to each group-time effect in the Callaway and Sant’Anna framework, comparing each cohort with never-treated or not-yet-treated units from the year before adoption, with covariates entering through the propensity score and outcome model for that comparison. The group-time estimates are then aggregated as in the Advanced lessons.

          With repeated cross-sections, the units observed before and after adoption differ, so \(\Delta Y\) is not available for each unit. Sant’Anna and Zhao (2020) give doubly robust estimators for this case as well, which require modeling the outcome in each group and period. Their results show that panel data, when available, allow more precise estimates. Chang (2020) also covers repeated cross-sections.

          Practice

          What to report

          • The covariates and why.List the pre-adoption covariates and the design argument for each.
          • The learners and tuning.Name the learners, the tuning method, the number of folds, and the number of repeated splits.
          • Overlap diagnostics.Show the propensity score distribution and any trimming.
          • Nuisance fit.Report out-of-fold predictive performance for both nuisance models.
          • Comparisons.Show parametric and machine-learning estimates side by side.
          Limitations

          What machine learning cannot do here

          • It cannot create overlap.Where no comparison units resemble program units, flexible models extrapolate.
          • It does not fix unobserved confounding.Conditional parallel trends is still an assumption about covariates the researcher chose.
          • Results depend on the split.Use repeated splits and report how much the estimate varies.
          • Small samples limit learners.With a few dozen states, flexible learners have little to learn from, and parametric models may be wiser.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Sant'Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101–122. https://doi.org/10.1016/j.jeconom.2020.06.003The doubly robust DiD estimand, efficiency, and inference. Start here.
          • Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097Neyman orthogonality, cross-fitting, and the general theory of double machine learning.
          • Chang, N.-C. (2020). Double/debiased machine learning for difference-in-differences models. The Econometrics Journal, 23(2), 177–191. https://doi.org/10.1093/ectj/utaa001The orthogonal score for semiparametric DiD with machine-learning first steps.
          • Abadie, A. (2005). Semiparametric difference-in-differences estimators. Review of Economic Studies, 72(1), 1–19. https://doi.org/10.1111/0034-6527.00321The weighting estimator that Chang’s method extends.
          • Caetano, C., Callaway, B., Payne, S., & Rodrigues, H. S. (2022). Difference in differences with time-varying covariates (arXiv:2202.02903) [Working paper]. arXiv. https://doi.org/10.48550/arXiv.2202.02903Time-varying covariates and doubly robust alternatives to TWFE. Working paper.
          • Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001Doubly robust estimation for each group-time effect under staggered adoption.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Expert · Lesson X2

          Functional form and changes-in-changes.

          Parallel trends is a statement about a particular scale. Trends that are parallel in test-score points need not be parallel in percentiles, logs, or proficiency rates. This lesson explains when the choice of scale matters, how to test whether it does, and how changes-in-changes identifies effects without choosing a scale at all.

          Before thisExpert: doubly robust and machine-learning DiDConditional parallel trends with flexible models.
          After thisExpert: triple differences and continuous treatmentsRicher comparisons and treatments that vary in dose.
          By the end

          What you will be able to do

          1. Explain why parallel trends in levels and in logs generally cannot both hold.
          2. State when parallel trends is insensitive to functional form, and check its testable implication.
          3. Choose a scale for test scores, proficiency rates, and other bounded or skewed outcomes, and justify it.
          4. State the changes-in-changes model and compute its counterfactual distribution.
          5. Recognize the trade-offs of changes-in-changes, including its assumptions about unobservables and discrete outcomes.
          Question

          Points, percents, or proficiency?

          A district evaluates a reading program in schools whose students score well below the state average. Over the evaluation period, scores rise everywhere. Should the analysis compare changes in scale-score points, in percent, in standardized units, or in the share of students reaching proficiency? Each choice implies a different counterfactual. If low-scoring schools tend to gain the same number of points as high-scoring schools, parallel trends holds in points. If they tend to gain the same percentage, it holds in logs. If they move the same share of students across a proficiency cut, it holds in rates. Usually, at most one of these can be true.

          Roth and Sant’Anna (2023) note that theory rarely says on which scale trends should be parallel, so a DiD design is more credible when its validity does not depend on that choice. Kahn-Lang and Lang (2020) argue that the choice of functional form is part of the identifying assumption and needs a substantive justification, not only a statistical one. This lesson makes those points concrete.

          The problem

          Why levels and logs disagree

          Suppose untreated outcomes grow by 20 percent for everyone. In logs, every group’s trend is the same, so parallel trends holds. In levels, a group that starts higher gains more points, so parallel trends fails whenever the groups start at different levels. The reverse also holds. If every group gains the same number of points, the group that starts lower grows faster in percent.

          The first script shows the size of the problem. Untreated outcomes grow by 20 percent in both groups, the program group starts higher, and the program raises outcomes by 10 percent. The true ATT in levels averages 4.98 across 100 samples. DiD in levels is off by 3.78 on average, because it attributes the program group’s larger point gain to the program. DiD in logs recovers the 10 percent effect, 10.1 percent on average, because the assumption holds on that scale. Changes-in-changes, described below, recovers the level effect with an average error of 0.18, without being told which scale is right.

          Dynarski (2003) addresses a version of this concern in the student-aid study. Students with deceased fathers were more likely to attend college than other students before the aid was cut, and less likely afterwards. When the ranking of the groups reverses like this, she notes, linear and nonlinear analyses must produce estimates of the same sign, though their magnitudes may differ. That is a useful check, but it does not fix the size of the effect.

          Theory

          When is parallel trends insensitive to scale?

          Roth and Sant’Anna (2023) characterize exactly when parallel trends holds for every strictly monotonic transformation of the outcome. It does if and only if a stronger, distributional version of parallel trends holds, one for the whole cumulative distribution function of untreated outcomes.

          \[F_{Y_1(0) \mid D=1}(y) = F_{Y_0(0) \mid D=1}(y) + \bigl[F_{Y_1(0) \mid D=0}(y) - F_{Y_0(0) \mid D=0}(y)\bigr] \quad \text{for all } y\]

          They show this condition holds essentially only if the population can be split into a subgroup for which treatment is effectively randomly assigned and a subgroup whose distribution of untreated outcomes is stable over time. Few observational designs fit that description, so in most applications the scale is a substantive choice.

          The condition has a testable implication. The right-hand side of the equation must itself be a valid CDF, never below zero and never decreasing. The second script computes this implied CDF. When the groups start at the same level, it is valid. When the program group starts higher, as in the first script, it dips to −0.07, which no distribution can do. Roth and Sant’Anna propose formal falsification tests along these lines. A rejection says that parallel trends cannot hold on every scale, so the researcher must defend the scale used. It does not say which scale is right.

          Run it yourself in R: code and output
          Judgment

          Choosing a scale for education outcomes

          • Scale scores.Vertically scaled scores are intended to give a point a similar meaning at different levels, which can support parallel trends in points. Check how the test was scaled and whether the scale changed during the study.
          • Standardized scores.Standardizing within year removes common growth. Trends then compare relative positions, and parallel trends concerns rank-like changes.
          • Proficiency rates.Rates depend on where the cut score falls relative to each group’s distribution. Groups near the cut can move more than groups far from it, even with identical score gains.
          • Skewed outcomes.Spending, absences, or suspensions are often analyzed in logs. Parallel trends in percent is then the assumption, and zeros need explicit handling.
          • Binary outcomes.Kahn-Lang and Lang (2020) argue that linear probability, probit, and logit models assume different counterfactuals, and that it is often appropriate to use the probit or logit interaction coefficient when its counterfactual is the credible one.

          Whatever the choice, show that pre-period trends are parallel on that scale, report results on at least one alternative scale, and explain any differences.

          Estimator

          Changes-in-changes

          Athey and Imbens (2006) propose a model whose assumptions do not depend on the scaling of the outcome. Untreated outcomes are generated as \(Y(0) = h(U, T)\), where \(U\) is an unobserved characteristic, such as ability, and \(h\) is strictly increasing in \(U\). The function \(h\) may change over time in any way, but the same function applies to both groups. The distribution of \(U\) may differ between groups but must not change over time within a group.

          Under these assumptions, a unit’s rank among comparison units carries its position in \(U\). The counterfactual for a program unit is built in three steps. Find its outcome’s rank in the comparison group’s distribution before the change. Find the comparison group’s outcome at that same rank after the change. That value is the unit’s counterfactual untreated outcome.

          \[Y^{N}_{11} \sim F^{-1}_{Y,01}\bigl(F_{Y,00}(Y_{10})\bigr), \qquad \text{ATT}^{CIC} = E[Y_{11}] - E\bigl[F^{-1}_{Y,01}\bigl(F_{Y,00}(Y_{10})\bigr)\bigr]\]

          Here the first subscript is the group and the second the period. Because ranks do not change under monotone transformations, the estimate in levels is the same whether the analysis is done in levels, logs, or any other increasing scale, transformed back. The method also delivers the entire counterfactual distribution, so it can estimate effects at quantiles, for example on the lowest-scoring students. It works with repeated cross-sections or panels. Athey and Imbens show that the estimator of the average effect is consistent at the usual rate and asymptotically normal, and they propose methods for inference.

          Trade-offs

          What changes-in-changes assumes

          Changes-in-changes replaces the choice of scale with assumptions about unobservables, and those deserve the same scrutiny.

          • One unobservable. All relevant differences between units must be captured by a single index. Athey and Imbens note that if \(h\) is nonlinear, this rules out, for example, classical measurement error in the outcome, which is common with test scores.
          • Stable groups. The distribution of \(U\) within each group must not change over time. They describe this as a strong assumption at the heart of both DiD and changes-in-changes. Changing student composition violates it.
          • Support. The program group’s values of \(U\) must lie within the comparison group’s. Otherwise some counterfactual ranks fall outside the comparison distribution.
          • Discrete outcomes. With binary or coarse outcomes, such as proficiency, the counterfactual is only partially identified. Athey and Imbens derive bounds and two ways to restore point identification, one through an additional assumption on the unobservables and one through covariates independent of them.

          Athey and Imbens also discuss a simpler alternative, quantile DiD, which adds the comparison group’s change at each quantile to the program group’s quantile. They propose a model that justifies it at every quantile, but show that the model has several unattractive features. In their application to disability benefits and injury durations, using data from an earlier study, the changes-in-changes results differ from standard DiD in magnitude and significance, which shows that the scale assumption can matter for policy.

          Simulate · invite a test

          The scale lab

          Decide how untreated outcomes grow, in equal percentages, in equal points, or a mix, and how far apart the groups start. Then see which estimator recovers the true effect in levels.

          Which scale makes trends parallel?

          Each run draws 2,000 units per group and period from skewed (lognormal) distributions. The program raises outcomes by 10 percent. Bars show the average over 20 samples of each estimate of the ATT in levels. The dashed line is the true ATT.

          100 percent percent
          Right: everyone grows by 20 percent. Left: everyone gains the same number of points.
          0.6
          Difference in the mean of log outcomes.
          Try this
          1. Set the starting gap to 0. Do the estimators agree?

          2. With a large gap, move growth fully to equal points. Which estimator now fails?

          3. Changes-in-changes works for every mix of growth here. What do all these growth patterns have in common?

          True ATT in levels–average over samples
          DiD in levels–average estimate
          Changes-in-changes–average estimate

          Press the button to draw samples.

          Beyond the mean

          Effects across the distribution

          Many education programs are judged by what they do for students who start furthest behind. A mean effect cannot answer that question. Because changes-in-changes recovers the whole counterfactual distribution of untreated outcomes for the program group, it can compare the observed and counterfactual distributions quantile by quantile. The difference at the 10th percentile, for example, says how much the program shifted the low end of the distribution.

          Two cautions apply. First, a quantile treatment effect compares quantiles of two distributions. It is not the effect on the students who would have been at the 10th percentile, because the program can reorder students. Interpreting it as an effect on low performers requires that ranks are preserved, an additional assumption. Second, quantiles in the tails are estimated from few observations and are sensitive to the support condition, so the bootstrap intervals that Athey and Imbens (2006) discuss are especially important there.

          The same logic explains why changes-in-changes can be attractive for proficiency analyses even when the outcome is coarse. Proficiency is a threshold on an underlying score. If the underlying scores are available, estimating the counterfactual score distribution and then applying the cut gives a counterfactual proficiency rate that does not depend on where the cut falls relative to each group.

          Example · published study

          Scale choices in published education DiD

          The studies used in earlier lessons show how education researchers handle scale. Dee and Jacob (2011) report effects of No Child Left Behind on NAEP scale scores and translate them into standard deviations, so that the trend assumption concerns NAEP points. NAEP is designed to be comparable over time, which supports that choice. Lafortune et al. (2018) study the relative achievement of low-income districts, using NAEP scores standardized to the distribution in the first year each test was given. Their parallel trends assumption concerns that standardized measure. Dynarski (2003) uses a linear probability model for college attendance and notes that the reversal of the groups’ ranking guarantees the same sign under nonlinear models.

          None of these studies needed changes-in-changes, but each made its scale choice visible. That is the main practical lesson. Readers can only judge a scale-dependent assumption if the scale is stated and defended.

          Practice

          What to report

          • The scale and why.Explain why trends should be parallel on the chosen scale.
          • An alternative scale.Report results on at least one other credible scale.
          • A distributional check.Show the implied counterfactual distribution, or use a formal test of insensitivity to scale.
          • Distributional effects.When effects on low or high performers matter, consider changes-in-changes or related methods.
          Limitations

          What this lesson does not solve

          • No scale is assumption-free.Changes-in-changes trades the scale choice for assumptions about unobservables.
          • Tests can only reject.Passing a test of insensitivity to scale does not prove parallel trends on any scale.
          • Staggered designs add complexity.Extensions to multiple groups and periods exist, but software and practice are less settled.
          • The simulation is stylized.Real test score distributions have ceilings, floors, and measurement error.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Roth, J., & Sant'Anna, P. H. C. (2023). When is parallel trends sensitive to functional form? Econometrica, 91(2), 737–747. https://doi.org/10.3982/ecta19402When parallel trends is insensitive to functional form, and how to test it. Start here.
          • Athey, S., & Imbens, G. W. (2006). Identification and inference in nonlinear difference-in-differences models. Econometrica, 74(2), 431–497. https://doi.org/10.1111/j.1468-0262.2006.00668.xThe changes-in-changes model, discrete outcomes, and inference.
          • Kahn-Lang, A., & Lang, K. (2020). The promise and pitfalls of differences-in-differences: Reflections on 16 and Pregnant and other applications. Journal of Business & Economic Statistics, 38(3), 613–620. https://doi.org/10.1080/07350015.2018.1546591Functional form as part of the identifying assumption, including binary outcomes.
          • Dynarski, S. M. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279–288. https://doi.org/10.1257/000282803321455287An education example where the reversal of group rankings fixes the sign.
          • Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement. Journal of Policy Analysis and Management, 30(3), 418–446. https://doi.org/10.1002/pam.20586NAEP scale scores and standard deviation units.
          • Lafortune, J., Rothstein, J., & Schanzenbach, D. W. (2018). School finance reform and the distribution of student achievement. American Economic Journal: Applied Economics, 10(2), 1–26. https://doi.org/10.1257/app.20160567Achievement gaps in standardized NAEP scores.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Expert · Lesson X3

          Triple differences and treatments with a dose.

          Two extensions stretch the basic design. A triple difference adds a third comparison to remove a bias that the ordinary DiD cannot. A continuous treatment replaces “treated or not” with “how much”. Both are common in education research, and both rest on assumptions that are easy to misstate.

          Before thisExpert: functional form and changes-in-changesHow the scale of the outcome enters the assumption.
          After thisExpert: few treated units and design-based inferenceWhen only a handful of units adopt.
          By the end

          What you will be able to do

          1. Compute a triple difference and state the single assumption it needs.
          2. Recognize when a triple difference helps and when it only moves the problem.
          3. Define level effects and causal responses for a treatment with a dose.
          4. Explain why comparing dose groups needs a stronger assumption than standard parallel trends.
          5. Interpret, and avoid over-interpreting, a TWFE regression on treatment intensity.
          Question

          Two common designs that go beyond the 2×2

          A state expands free tutoring for students eligible for free or reduced-price lunch (FRL). A comparison of FRL students in this state with FRL students in a neighboring state is threatened by differences in the states’ trends. A comparison of FRL and non-FRL students within the state is threatened by differences in the groups’ trends. A triple difference uses both comparisons at once.

          A second state funds tutoring for every school but lets schools choose how many hours to offer. Here, either no school is untreated or only some are, and the natural question is how much more students gain from more hours. That is a question about a dose, and the DiD logic extends to it only with care.

          Design 1

          The triple difference

          Index the program state by \(S = 1\), FRL students by \(F = 1\), and the period after the expansion by \(P = 1\). The triple difference takes the within-state DiD between FRL and non-FRL students in the program state, and subtracts the same DiD in the comparison state.

          \[\widehat{DDD} = \bigl[\Delta\bar Y_{S=1,F=1} - \Delta\bar Y_{S=1,F=0}\bigr] - \bigl[\Delta\bar Y_{S=0,F=1} - \Delta\bar Y_{S=0,F=0}\bigr]\]

          The triple difference was introduced by Gruber (1994), as Olden and Møen (2022) document. They note that its use in top journals rests largely on intuition, with identifying assumptions rarely derived, and they give the formal result. Although the triple difference can be computed as the difference between two DiD estimators, it does not need parallel trends to hold for either one. It needs only that the two DiD estimators have the same bias, so that the bias cancels when one is subtracted from the other. In the tutoring example, the within-state DiD is biased if FRL students trend differently from other students. The triple difference is unbiased if that FRL gap in trends is the same in the comparison state.

          The first script shows this. The program state trends 1.5 points faster than the comparison state, and FRL students trend 0.8 points faster than other students in both states. The true effect is 3.0 points. Across 200 samples, the DiD across states averages 4.49 and the DiD within the program state averages 3.83, each biased by one of the two trend differences. The triple difference averages 3.03. When the FRL trend gap is instead 0.6 points larger in the program state, the triple difference averages 3.65, biased by exactly that difference.

          The cost is precision. A triple difference uses eight cell means instead of four, so its standard error is larger. In the example, its standard deviation across samples is 0.69. The design is worth that cost only when the bias it removes is real.

          Run it yourself in R: code and output
          Diagnostics

          Checking the triple-difference assumption

          The equal-bias assumption concerns the post-adoption period, so, like parallel trends, it cannot be tested directly. Its pre-period analogue can be examined. With several years before the expansion, estimate a triple-difference event study, with leads and lags of the three-way interaction relative to the year before adoption. Leads close to zero are consistent with the FRL trend gap being the same in both states before the program. Everything the event-study and sensitivity lessons said about leads applies here too. Passing a test is weak evidence, and an equivalence bound or a sensitivity analysis says more.

          Plotting the four group trends separately is also informative. A triple difference that looks clean can hide two large and offsetting biases. If FRL students in the program state were already diverging from FRL students elsewhere, and non-FRL students show the same divergence, the third difference removes it. That is exactly the case the design is meant for, but a reader should see it rather than infer it from a single coefficient.

          Finally, the choice of the third group deserves the same care as the choice of the comparison group in the first Intermediate lesson. Ineligible students in the same schools share teachers, buildings, and leadership with eligible students. That makes them good at absorbing school-level shocks, and also makes spillovers likely.

          Judgment

          When a triple difference helps

          • A clear within-unit comparison group.Ineligible students, grades, or subjects not targeted by the policy, measured in the same schools and years.
          • Shared shocks.The third difference helps when the threat is a shock that hits eligible and ineligible groups in the program state alike, such as a state budget cut or a change in the state test.
          • A credible equal-bias assumption.Argue that the gap in trends between eligible and ineligible groups would have been the same in both places. Show pre-period trends for all four groups, not only the final difference.
          • No spillovers to the ineligible group.If tutoring for FRL students also helps their classmates, the within-state comparison understates the effect.
          • Regression form.The triple difference equals the coefficient on the three-way interaction in a fully saturated regression, which makes it easy to add fixed effects and covariates. With staggered adoption, the warnings of the Advanced level apply to that regression too.
          Design 2

          Treatments with a dose

          Callaway et al. (2025), a working paper, analyze DiD designs in which treatment has a dose, such as hours of tutoring, funding per pupil, or the share of a cohort exposed to a reform. They distinguish two kinds of effects. A level effect compares a unit’s outcome under dose \(d\) with its outcome under no treatment. A causal response is the change in outcome from a marginal increase in the dose. With a binary treatment the two coincide. With a dose they do not, and they need different assumptions.

          Comparing units with dose \(d\) to untreated units identifies the average level effect for the units that received dose \(d\), written \(\text{ATT}(d \mid d)\), under a parallel trends assumption like the binary one. Comparing units that received different doses, for example 3 hours against 2, is harder. The difference mixes the causal response with what the authors call selection bias, the possibility that units choosing more hours would benefit differently from the same dose. To interpret such comparisons as causal responses, they introduce strong parallel trends, which says that the path of outcomes for the whole treated population under dose \(d\) would equal the path that units with dose \(d\) actually followed. That restricts how effects vary across units and may not be plausible in many applications. They observe that empirical papers often write as if they assume standard parallel trends but interpret results as if they had assumed strong parallel trends.

          Example · simulation

          Hours of tutoring and selection on gains

          The second script simulates schools that choose between one and four hours of tutoring a week, while others choose none. Schools that benefit more per hour choose more hours, with gains of 0.6, 0.7, 0.8, and 0.9 points per hour in the four dose groups. Parallel trends holds, so each dose group compared with untreated schools recovers its own level effect. Across 200 samples, the estimates are 0.60, 1.41, 2.40, and 3.60, against true values of 0.60, 1.40, 2.40, and 3.60.

          Comparing adjacent dose groups is a different matter. The implied per-hour gains are 0.80, 0.99, and 1.20, larger than the per-hour gain of any group, because each comparison adds the difference in how much the two groups benefit. A school considering adding an hour would be misled. The TWFE coefficient on hours averages 0.90, which also does not correspond to the per-hour gain of the typical school. Callaway et al. (2025) show that this TWFE estimand admits several decompositions, none of which is a clean summary of effects when they vary across doses and units.

          Practice

          Analyzing a treatment with a dose

          • Estimate level effects by dose.Compare each dose group, or a flexible function of dose, with untreated units. These need only standard parallel trends.
          • Be explicit before comparing doses.A dose-response curve or a per-unit effect needs strong parallel trends. Argue for it, or describe the results as level effects for each group.
          • Ask why units chose their dose.When the dose is chosen, as hours of tutoring usually are, selection on gains is likely. When it is set by a formula or a court order, as with many school finance reforms, the stronger assumption may be more plausible.
          • Avoid a single TWFE slope as the summary.Use estimators that target well-defined parameters, such as those implemented in the contdid package for R that the authors provide.
          • Mind staggered timing.When doses start in different years, the problems of the Advanced level combine with those of this section.
          A common shortcut

          Turning a dose into a binary treatment

          Faced with a dose, many studies split units into “high” and “low” exposure, for example districts above and below the median increase in funding, and run a binary DiD. This avoids estimating a dose-response curve but does not avoid the assumption. If both groups received some dose, the binary DiD compares two treated groups. It identifies the difference between their level effects only if their untreated trends were parallel, and it can be read as the effect of more dose only under the stronger assumption of this lesson. If the “low” group received no dose at all, the comparison is an ordinary DiD for the high group, but the estimate averages over everything above the cut.

          The shortcut is most defensible when the dose is set externally and the cut separates units that differ mainly in exposure. It is least defensible when units chose their dose. In both cases, state which units are compared, what dose each received, and which assumption gives the estimate a causal reading.

          Examples · published studies

          A dose in practice

          Callaway et al. (2025) illustrate their methods with a study by Acemoglu and Finkelstein of a 1983 Medicare reform that removed hospitals’ labor subsidies, where the dose is each hospital’s share of Medicare patients before the reform. Their estimates support the original conclusion that the reform led hospitals to favor capital over labor. They also find causal response estimates that are large at low doses but slightly negative for most positive doses, which they read as casting doubt on strong parallel trends, on the simple model of hospital production, or on both.

          Education studies often use a dose without naming it. Lafortune et al. (2018), discussed in the Advanced level, translate their achievement effects into effects per $1,000 of spending, between 0.12 and 0.24 standard deviations per $1,000 per pupil per year. Such a translation divides an effect by a dose, so it carries the dose assumptions of this lesson. It is most credible when the funding increase is set by formulas and courts rather than chosen by districts according to how much they expect to benefit, and readers should weigh it with that in mind.

          Practice

          What to report

          • For a triple difference.All eight cell means or the four group trends, the equal-bias argument, and pre-period evidence for it.
          • For a dose.The distribution of doses, how the dose was set, the parameter estimated, and the assumption it needs.
          • Level effects by dose group.Before any dose-response curve, show each group’s level effect against untreated units.
          Limitations

          What these designs cannot do

          • A triple difference trades one assumption for another.Equal bias is still untestable after adoption.
          • Few groups, few clusters.A triple difference with one program state has the inference problems of the standard errors lesson.
          • Doses without untreated units.If every unit receives some dose, level effects are not identified without further assumptions.
          • Working paper.The continuous treatment results come from a working paper and may be revised.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Olden, A., & Møen, J. (2022). The triple difference estimator. The Econometrics Journal, 25(3), 531–553. https://doi.org/10.1093/ectj/utac010The identifying assumption of the triple difference, formally derived. Start here for DDD.
          • Callaway, B., Goodman-Bacon, A., & Sant'Anna, P. H. C. (2025). Difference-in-differences with a continuous treatment (arXiv:2107.02637v8) [Working paper]. arXiv. https://doi.org/10.48550/arXiv.2107.02637Level effects, causal responses, strong parallel trends, and TWFE with a dose. Working paper.
          • Lafortune, J., Rothstein, J., & Schanzenbach, D. W. (2018). School finance reform and the distribution of student achievement. American Economic Journal: Applied Economics, 10(2), 1–26. https://doi.org/10.1257/app.20160567Effects per $1,000 of school spending.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008Where these designs sit in the wider DiD literature.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 5 · Expert · Lesson X4

          Few treated units and design-based inference.

          Many education policies change in one state, one district, or a handful of schools. With few treated units, the average comparison unit may be a poor guide and conventional standard errors stop working. This lesson builds synthetic DiD step by step, places it beside synthetic control and factor models, and asks where the randomness in a DiD estimate really comes from.

          Before thisExpert: triple differences and treatments with a doseRicher comparisons and doses.
          After thisExpert: Monte Carlo studies and open debatesTesting estimators and the questions still open.
          By the end

          What you will be able to do

          1. Explain why equal weighting of comparison units can fail when few units are treated.
          2. Compute synthetic DiD weights for units and periods and the resulting estimate.
          3. Compare synthetic DiD with DiD, synthetic control, and factor models.
          4. Construct placebo standard errors and interpret them.
          5. Explain design-based inference and when conventional standard errors are conservative.
          Question

          When the average comparison unit will not do

          The few-treated version of the shared data has 6 program schools that adopt in 2021 and 114 comparison schools, observed from 2015 to 2024. A hidden factor, such as enrollment growth, shifts schools’ paths over time, and program schools are more exposed to it than the typical comparison school. Some comparison schools are just as exposed. The true effect is 3.0 points.

          An ordinary DiD weights all 114 comparison schools equally and estimates 6.75. The comparison group’s average path does not resemble the program schools’ path, so parallel trends fails for the average. It may hold for a well-chosen weighted average of comparison schools. Choosing that weighted average from the data is the idea behind synthetic control and synthetic DiD.

          Inference is the second problem. The standard errors lesson showed that cluster-robust standard errors need many treated clusters, and that with one to six treated units they can badly over-reject. Few treated units call for both a different estimator and a different way to measure uncertainty.

          Estimator

          Synthetic DiD, step by step

          Arkhangelsky et al. (2021) combine the strengths of DiD and synthetic control. Synthetic DiD chooses two sets of weights and then runs a weighted DiD.

          1. Unit weights \(\omega\). Choose non-negative weights on comparison schools, summing to one, so that their weighted pre-period path runs parallel to the program schools’ average path. An intercept allows the paths to differ in level, as in DiD. A small penalty on the size of the weights spreads them across many schools.
          2. Time weights \(\lambda\). Choose non-negative weights on pre-period years, summing to one, so that the weighted pre-period average of each comparison school predicts its post-period average. These weights emphasize pre-period years that resemble the post-period.
          3. Weighted DiD. Compute the program schools’ change from the time-weighted pre-period to the post-period, minus the unit-weighted change for comparison schools.
          \[\hat\tau^{sdid} = \Bigl(\bar Y_{tr,post} - \sum_{s \in pre}\hat\lambda_s \bar Y_{tr,s}\Bigr) - \sum_{j \in co}\hat\omega_j\Bigl(\bar Y_{j,post} - \sum_{s \in pre}\hat\lambda_s Y_{j,s}\Bigr)\]

          With equal unit weights and equal time weights, this is ordinary DiD. With no intercept, so that levels must match, and no time weights, it is close to synthetic control. Synthetic DiD sits between them. It asks the weighted comparison group to be parallel rather than identical, which is easier to achieve and more robust, and it lets the most relevant pre-period years carry more weight.

          Arkhangelsky et al. study the estimator when untreated outcomes follow a model with latent unit factors interacted with latent time factors, the structure of the hidden factor in the shared data, and give conditions for consistency and asymptotic normality. They find, theoretically and empirically, that it has desirable robustness properties compared with DiD and synthetic control, and illustrate it with the effect of California’s 1989 cigarette tax increase and with placebo studies based on Current Population Survey data.

          Example · shared school data

          DiD, synthetic control, and synthetic DiD

          The first script implements the three steps directly, solving each weighting problem by projected gradient descent so that R, Python and Stata give identical results. The penalty follows the choices in Arkhangelsky et al. (2021), based on the variability of year-to-year changes in comparison schools. DiD estimates 6.75. Synthetic control, which matches levels, estimates 4.17. Synthetic DiD estimates 4.25. Both synthetic methods remove most of DiD’s bias of 3.75 points, though neither removes all of it in this small design with six pre-period years.

          The unit weights are spread out. Forty of the 114 comparison schools receive more than 1 percent of the weight, and no school receives more than 2 percent. The weights favor comparison schools whose pre-period paths resemble the program schools’, which, in this design, are the schools most exposed to the hidden factor.

          Run it yourself in R: code and output
          Inference

          Placebo standard errors

          With six treated schools, a cluster-robust standard error is unreliable. Arkhangelsky et al. (2021) describe several alternatives and use a placebo method for applications with few treated units. Pretend that a randomly chosen set of comparison schools, the same size as the treated group, had adopted, re-run synthetic DiD on the comparison schools alone, and repeat many times. The spread of these placebo estimates measures how much the estimator varies when there is no effect.

          In the first script, 100 placebo runs give a standard error of 0.69 and a 95 percent interval from 2.90 to 5.60, which contains the true effect of 3.0. The method assumes that the comparison schools are informative about the noise affecting program schools. That is the same homogeneity assumption Conley and Taber (2011) make, and the same caution applies. If program schools differ systematically, for example in size, their noise may differ too, as Ferman and Pinto (2019) emphasize.

          Alternatives

          Factor models and matrix completion

          A different route models untreated outcomes directly as unit effects, time effects, and a low-rank interaction between unit and time factors, then predicts the missing untreated outcomes of treated school-years. With a fixed number of factors, this is an interactive fixed effects model. Athey et al. (2021) instead treat the untreated outcomes as a matrix with missing entries and complete it with a penalty on the matrix’s nuclear norm, chosen by cross-validation, so the data decide how many factors matter.

          These methods generalize the imputation estimator of the Advanced level, which is the special case with no interaction term. They are attractive with many pre-periods, where factors can be estimated well. With few pre-periods and few treated units, as in the shared data, estimating factors is hard. The few-treated example on the Expert page shows that, in this dataset, a one-factor model performs like synthetic DiD while cross-validated matrix completion removes little of the bias, because cross-validation chose heavy regularization.

          Judgment

          Choosing a method when few units are treated

          • Many comparison units, one or a few treated.Synthetic DiD or synthetic control, with placebo inference. Check that the weights are not concentrated on one or two units whose idiosyncrasies could drive the result.
          • Long pre-period.Factor models and matrix completion become more attractive, because there is enough history to estimate the factors. Synthetic control’s level matching also becomes more credible.
          • Short pre-period.Prefer methods that ask only for parallel paths, such as synthetic DiD or ordinary DiD with a careful comparison group, and be cautious about any method that fits many parameters to few years.
          • Several treated units adopting at different times.Apply the logic of the Advanced level. Synthetic DiD extends to staggered adoption by estimating each adoption cohort separately and averaging.
          • A known source of differential trends.If the hidden factor can be measured, for example enrollment growth, conditioning on it through the methods of the covariates lessons may be more transparent than weighting.

          Whatever the choice, the appropriate summary with few treated units is usually a range rather than a single number. When DiD, synthetic DiD, and a factor model disagree, the disagreement measures how much the conclusion depends on which comparison is trusted, and readers should see it.

          Diagnostics

          Reading pre-period fit

          Synthetic methods are often judged by how closely the weighted comparison group tracks the treated units before adoption. A close fit is necessary but not sufficient. With few pre-period years and many comparison units, weights can match noise rather than the underlying trend, and the fit then says little about the post-period. The penalty on the size of the unit weights in synthetic DiD guards against this by spreading weight across many units.

          Two checks help. The first is a placebo in time. Pretend adoption happened a few years earlier, fit the weights on the years before that, and see whether the method predicts the years that followed, which are still untreated. The second is a leave-one-out check. Drop the comparison units with the largest weights one at a time and see whether the estimate changes. A conclusion that rests on one comparison unit deserves less confidence than one that survives such changes.

          Where randomness comes from

          Design-based inference

          Standard errors usually assume that units are sampled from a larger population. In education, researchers often observe every school in a state, and it is unclear what larger population they were drawn from. Athey and Imbens (2022) take a different view. Potential outcomes are fixed, and the randomness comes from which units adopt and when. Under random assignment of adoption dates, they show that the standard DiD estimator is unbiased for a particular weighted average of causal effects, characterize its exact finite-sample properties, and show that the standard variance estimator is conservative.

          The second script illustrates this with the shared single-adoption data. The 120 schools are treated as the whole population with fixed outcomes and fixed, varying effects averaging 3.01. Across 1,000 random choices of which 60 schools adopt, the DiD estimate averages 3.02. Its standard deviation across assignments is 0.431, and the average clustered standard error is 0.446, slightly larger, as the theory predicts. The resulting 95 percent intervals contain the population average effect in 96.1 percent of assignments.

          The design-based view also supports the rule from the standard errors lesson, to cluster at the level at which treatment is assigned (Roth et al., 2023). It gives inference a clear meaning in settings where the whole population is observed, provided adoption is plausibly as good as random, which is a strong assumption in many policy settings.

          Example · published study

          One treated state, revisited

          The standard errors lesson described Conley and Taber’s (2011) analysis of Georgia’s HOPE scholarship, with Georgia as the only treated state and 41 comparison units. Clustering by state gave a 95 percent interval of 5.8 to 9.7 percentage points for the effect on college enrollment, while their method, which learns the distribution of state-level shocks from the comparison states, gave −1.0 to 20.7. The example shows the two problems of this lesson in one study. One treated unit makes the choice of comparison group decisive, which is what synthetic methods address, and it makes conventional inference unreliable, which is what placebo and residual-based methods address.

          For a single-state education reform, a credible analysis typically combines both. It reports the synthetic comparison and its weights, so readers can see which states stand in for the treated one. It shows the pre-period fit. And it uses placebo or permutation inference with a clear statement of what it assumes.

          Practice

          What to report

          • The weights.List the comparison units with the largest weights and explain why they are plausible stand-ins.
          • Pre-period fit.Plot the treated units’ path against the weighted comparison path before adoption.
          • Several estimators.Report DiD, synthetic DiD, and at least one other, and explain differences.
          • Inference that fits the design.Use placebo, permutation, or residual-based methods with few treated units, and state their assumptions.
          • The source of randomness.Say whether inference is sampling-based or design-based, and why.
          Limitations

          What these methods cannot do

          • Good fit is not proof.A close pre-period fit can be achieved by overfitting noise, especially with few pre-periods.
          • Shocks at adoption remain a threat.If something else happens to the treated units when they adopt, no weighting removes it.
          • Placebo inference needs comparable units.Its validity rests on the comparison units’ noise resembling the treated units’.
          • Random adoption is a strong assumption.Design-based guarantees hold only to the extent that adoption timing is as good as random.
          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118. https://doi.org/10.1257/aer.20190159Synthetic DiD, its theory, and placebo inference. Start here.
          • Athey, S., Bayati, M., Doudchenko, N., Imbens, G., & Khosravi, K. (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association, 116(536), 1716–1730. https://doi.org/10.1080/01621459.2021.1891924Matrix completion for panel data with missing untreated outcomes.
          • Athey, S., & Imbens, G. W. (2022). Design-based analysis in difference-in-differences settings with staggered adoption. Journal of Econometrics, 226(1), 62–79. https://doi.org/10.1016/j.jeconom.2020.10.012Design-based estimands and variances under staggered adoption.
          • Conley, T. G., & Taber, C. R. (2011). Inference with “difference in differences” with a small number of policy changes. Review of Economics and Statistics, 93(1), 113–125. https://doi.org/10.1162/rest_a_00049Inference with a small number of policy changes.
          • Ferman, B., & Pinto, C. (2019). Inference in differences-in-differences with few treated groups and heteroskedasticity. Review of Economics and Statistics, 101(3), 452–467. https://doi.org/10.1162/rest_a_00759Few treated groups with heteroskedasticity.
          • Roth, J., Sant'Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008A review of inference with few clusters and design-based arguments.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 1 · Foundations

          Why random assignment works.

          A randomized experiment answers a causal question by letting chance, rather than people, decide who receives a program. This lesson explains what random assignment achieves, why other designs need stronger assumptions to answer the same question, and what can still go wrong after assignment.

          Before thisNo background neededAverages and differences are enough.
          After thisIntermediate: estimation and inferenceStandard errors, covariates, and blocking.
          By the end

          What you will be able to do

          1. Describe a causal effect using potential outcomes.
          2. State which average effect a study targets, and why that choice comes before the method.
          3. Explain selection bias and why a comparison of participants with non-participants contains it.
          4. Read a simple causal diagram and distinguish confounders, mediators and colliders.
          5. Explain why random assignment makes groups comparable on average, including on characteristics nobody measured.
          6. Explain why a single randomized trial can still be unbalanced by chance, and why a baseline significance test does not check randomization.
          7. Distinguish the bias of an estimate from its chance error.
          8. Identify common threats to a randomized experiment in schools.
          9. Distinguish allocation concealment from blinding, and state who can be blinded in a school study.
          Conditions

          What a causal claim requires

          Before any method, a causal claim has to meet three conditions. In the analysis that Shadish et al. (2002) trace to John Stuart Mill, a causal relationship exists if the cause preceded the effect, if the cause was related to the effect, and if no plausible alternative explanation for the effect can be found. The conditions sound modest, but only the first is usually easy to meet.

          • The cause came first. Tutoring starts in September and the test is in May, so the order is clear. It is far less clear when the treatment and the outcome are recorded at the same moment. A principal’s leadership style and a school’s climate, both taken from one survey in March, carry no information about which came first. What settles this condition is a known start date, repeated measurement, or a rule that fixes the timing.
          • The cause and the outcome are related. Tutored students score differently from untutored students. This is the part a correlation can show, and it is necessary. On its own it settles very little, as the sections on selection and on correlation explain.
          • Nothing else explains the difference. Students who enroll in tutoring differ from students who do not, in prior achievement, in family support, and in much that no administrative file records. This condition is the hard one, and every design on this site is a strategy for meeting it.

          The three conditions also fix what the word effect means. For Shadish et al. (2002), an effect is the difference between what did happen and what would have happened. That second quantity is never observed, which is why the next section gives it a notation.

          Condition 2, relatedCondition 3, an alternative explanationDYUTutoringTest scorePrior achievementSeptemberMayCondition 1, the cause came first
          The three conditions in one picture. Tutoring starts in September and the test is written in May, so the cause came first. Tutored and untutored students differ on the test, so the two are related. Prior achievement may drive both enrollment and the score, and that third condition is what the rest of this site is about. The first two conditions can be checked from the data and the calendar. The third rests on an argument about what the data do not show.

          When a program cannot be assigned by the researcher, other grounds have long been used to judge an association. The best-known list comes from Hill (1965), who set out nine viewpoints, which he named strength, consistency, specificity, temporality, biological gradient, plausibility, coherence, experiment, and analogy. Applied to education, temporality is the first condition above, and biological gradient becomes a dose-response pattern in which more of the program goes with more of the outcome. Hill did not call these items criteria, and later writers have cautioned against applying them rigidly (Rosner, 2015). They are aids to judgment rather than a test to pass. An association can satisfy several of them and still be confounded, which is why this site asks a narrower question throughout. What stands in for what would have happened, and why should a reader believe it?

          The idea

          Two outcomes for every student

          Imagine a school offering an after-school tutoring program. For any one student, we can picture two test scores at the end of the year. One is the score the student would earn with tutoring, and the other is the score the same student would earn without it. Statisticians call these the student’s potential outcomes, and the difference between them is the effect of tutoring for that student (Rubin, 1974).

          The difficulty is that we only ever see one of the two. A student who attends tutoring reveals the first score and hides the second. Holland (1986) called this the fundamental problem of causal inference. Because the individual effect can never be observed, research designs aim at something more modest, the average effect across a group of students.

          Averages are within reach because we do not need to know any one student’s missing score. We need a group whose average outcome can stand in for what the treated group’s average would have been without the program. Everything in this module, and in every other module on this site, is a strategy for finding such a group.

          5055606570Student AtutoredStudent Bnot tutoredStudent CtutoredStudent DtutoredStudent Enot tutoredscore with tutoringscore without tutoringnever observed
          The fundamental problem, drawn. Each student has two potential scores, and the vertical distance between them is that student’s own effect of tutoring. For every student one dot is filled, the score we observe, and one is hollow, the score that was never produced. No design recovers the hollow dots. Designs aim instead at a group whose average can stand in for them.
          The target

          Decide which effect you want

          Saying that every student has two outcomes leaves open which students the effect is for. “The effect of tutoring” is not one number. It can mean the average effect across every student in the district, the average for the students who actually enrolled, or the average for a particular group, such as students near a score cutoff. Each is a different target, called an estimand (Imbens & Rubin, 2015), and each can have a different value when tutoring helps some students more than others.

          The estimand comes from the question and the decision, not from the method. A district deciding whether to make tutoring mandatory wants the average across everyone. A district deciding whether to keep a voluntary program wants the average for the students who choose it. Choosing a method first and describing its answer afterward risks answering a question nobody asked.

          Whether the data can answer the question is a separate matter, called identification. A design identifies an effect when its assumptions, rather than the number of students, are enough to recover it. More students narrow the chance error around an estimate and do nothing for an assumption that fails. Each later module pairs a design with the estimand it can identify. Module 2 asks whether the target is the effect for everyone or for those treated. Module 3 recovers the effect for students whom an offer or a lottery moved. Module 4 recovers it near a cutoff. Modules 5 and 6 estimate the effect for the units that were treated, and Module 7 asks how effects differ from student to student.

          The problem

          When students choose, comparisons mislead

          Suppose tutoring is voluntary and we compare the end-of-year scores of students who signed up with those of students who did not. That difference can be split into two parts.

          • The effect of tutoring on the students who signed up.
          • Selection bias, the difference between the two groups that would have existed even without tutoring.

          Selection bias is rarely zero. Students who volunteer for extra help may be more motivated, have more involved parents, or have more free time after school. Each of these could raise their scores whether or not they attend. The sign can also go the other way. If teachers steer struggling students into tutoring, the participants start behind and the comparison understates the effect.

          Adding control variables helps only if the variables that drive both participation and scores are measured. Motivation, family support, and a teacher’s private judgment are usually not. This is why comparisons between self-selected groups are unreliable.

          Distinctions

          Why a correlation need not be an effect

          Two variables are correlated when knowing one tells you something about the other. That is a statement about a pattern in a data set. A causal claim is stronger, because it says that changing the first would change the second. Most disagreements about education research come from confusing the two, and the confusion has a small number of recurring sources.

          • A common cause. Something drives both the treatment and the outcome. In the job-training data used in Module 2, trainees earned about $8,500 less in 1978 than adults in a national survey, while the randomized experiment on the same trainees put the effect at a gain of about $1,800 (Dehejia & Wahba, 1999; LaLonde, 1986). The raw comparison has the wrong sign, because trainees were recruited from people with weak earnings histories.
          • Reverse causation. The outcome may be driving the supposed cause. A district that places struggling readers in its smallest classes will produce data in which small classes go with low scores. The arrow runs from reading difficulty to class size, not the other way, and the first condition in the previous section is what fails.
          • Who is in the data. An association can be manufactured by the way a sample was formed. Berkson (1946) showed that two unrelated conditions can appear correlated among hospital patients, because having either one raises the chance of being admitted. Selecting on something that both the treatment and the outcome affect has the same consequence (Hernán et al., 2004). Studying only students who completed a program, or only schools that stayed open, can produce a pattern that is absent in the population those students and schools came from.
          • Chance. Compare enough variables and some will line up. Messerli (2012) reported a correlation of 0.791 between chocolate consumption per person and Nobel laureates per 10 million people across 23 countries. The author presented the finding as hypothesis-generating and noted that differences in national wealth may lie behind it.

          Grouping can also reverse the direction of an association, a pattern named after Simpson (1951). Bickel et al. (1975) reported that Berkeley admitted about 44 percent of its 8,442 male graduate applicants in 1973 and about 35 percent of its 4,321 female applicants. Department by department the pattern largely disappeared, because women had applied in greater numbers to departments that admitted few applicants of either sex. The university-wide figure and the department figures are both correct, and they answer different questions. Whenever an aggregate comparison and a within-group comparison disagree, the useful question is which grouping matches the decision at hand.

          A common causeUDYU moves bothReverse causationDYthe arrow runs backwardWho is in the dataSDYselecting on S links themChanceDYno path at allD treatment, Y outcome, U an unmeasured cause, S being in the sample
          Four ways a correlation can appear without an effect. In the first panel an unmeasured cause moves the treatment and the outcome, so they move together without either causing the other. In the second the arrow runs from the outcome to the treatment. In the third the treatment and the outcome both affect who enters the sample, and studying only those units ties them together. In the fourth nothing connects them and the pattern is chance. Only the second can be ruled out by timing alone.
          A picture of the argument

          Reading a causal diagram

          A causal diagram shows assumptions about how variables might be connected (Pearl, 1995). Each variable is a node and each arrow means that one variable may directly cause another. The figures above already used this language. What a diagram adds is a way to see which other variables sit between the treatment and the outcome, and what each of them should be asked to do.

          A missing arrow is also a claim. Leaving out an arrow from motivation to test scores says that motivation has no effect on them. The diagram is therefore an argument that readers can challenge, not a result that the data produced.

          Any variable on a route between the treatment and the outcome plays one of three roles.

          • Confounder. It points into both the treatment and the outcome. Prior achievement may push students into tutoring and also raise their scores. This opens a back-door path, a route from treatment to outcome that begins with an arrow into the treatment. Comparing students with similar prior achievement closes it, so this is a variable to adjust for.
          • Mediator. It lies on the way from treatment to outcome. Tutoring may raise the hours a student practices, and practice may raise scores. The mediator carries part of the effect, so adjusting for it removes that part. To estimate the total effect of tutoring, leave it out. How much of an effect runs through a mediator is a different question that needs stronger assumptions.
          • Collider. Two arrows point into it. Tutoring and motivation may both affect whether a student enrolls in an advanced course. The route through a collider is closed until it is adjusted for. Adjusting for it, or selecting students on it, links tutoring to motivation where no link existed, the same mechanism as in the selection example above (Hernán et al., 2004). Leave it out.

          Random assignment simplifies the diagram. A coin flip has no arrows pointing into it, so no back-door path exists and nothing needs adjusting to remove confounding. A variable measured before assignment can still be added to sharpen the estimate, which the Intermediate lesson takes up, while variables that tutoring itself changes stay out. Outside experiments the same diagram is how researchers decide what must be measured, and Cinelli et al. (2024) catalog good and bad controls in this vocabulary.

          ConfounderXprior achievementDYAdjust: yesX causes both D and Y,so it opens a back-door routeMediatorMhours practicedDYAdjust: noM carries part of the effect,adjusting removes that partColliderCUadvanced coursemotivationDYAdjust: noC is caused by D and U,adjusting links D to UD tutoring, Y test score. Dashed arrows come from an unmeasured cause.
          Three roles a variable can play between treatment and outcome. A confounder causes both the treatment and the outcome, so comparing students with the same value of it removes a spurious route. A mediator sits on the path the treatment works through, so adjusting for it takes away part of the effect being estimated. A collider is caused by two other variables, so adjusting for it, or selecting students on it, creates a link that was not there before. The same adjustment helps in the first panel and harms in the other two.
          Distinctions

          Does no correlation mean no effect?

          No. Inferring no effect from no correlation is weaker than it looks. A difference near zero is consistent with a real causal effect in at least four ways.

          • Effects that cancel across students. A program that raises scores for students who start behind and lowers them for students who start ahead can average out to almost nothing. The average effect is near zero while few individual effects are. Module 7 returns to effects that differ across students.
          • Effects that cancel across paths. A longer school day may raise achievement through added instructional time and lower it through fatigue. The total can be small even when both paths are large, and a design that recovers only the total will report the small number.
          • Measurement and range. Error in recording the treatment or the outcome pulls an observed correlation toward zero. So does studying a narrow slice of schools in which the treatment barely varies, since an effect cannot show up in data where the cause hardly moves.
          • Adjusting for the wrong variable. Controlling for something the treatment itself changes removes part of the very effect being estimated, which is one reason adjustment variables are usually chosen from what was measured before the program began.

          Reading no correlation as no effect therefore assumes that the associations in the data mirror the causal structure, and that nothing cancels along the way. In the causal graph literature that assumption carries a name, faithfulness (Spirtes et al., 1993). It is an assumption rather than a finding, and it usually goes unstated. A null result is informative when the design had the precision to detect an effect of the size that would matter, so the honest version of such a claim reports that precision, not just the absence of statistical significance.

          One consequence is a rule for writing. If the question is causal, say so, name the design that answers it, and state the assumptions it rests on. Hernán (2018) argues that substituting the word association while still drawing a causal conclusion hides the assumption rather than avoiding it.

          +8+40-4Effect on test scores, points+8Started behind40% of students-4Started ahead60% of students+0.8All studentsthe average effect
          An average near zero, and nothing like zero underneath. The program raises scores by 8 points for the 40 percent of students who start behind and lowers them by 4 points for the 60 percent who start ahead. The average effect is 0.8 points, which a study of ordinary size would report as indistinguishable from zero. Almost no student in this example has an effect near 0.8. An average is a summary of the effects, not a description of any of them.
          The solution

          What a lottery does

          Now suppose the school has more applicants than seats and holds a lottery. Whether a student receives tutoring is decided by chance alone. Chance cannot know who is motivated, so motivated and unmotivated students are equally likely to win a seat. The same holds for every other characteristic, measured or not, including the score each student would have earned without tutoring.

          Because of this, on average across possible lotteries, the two groups have the same distribution of potential outcomes. The losers’ average score therefore estimates what the winners’ average would have been without tutoring, and the difference in means estimates the average effect without selection bias (Imbens & Rubin, 2015).

          Two cautions follow from the words “on average”. First, any single lottery produces groups that differ a little by chance, so a single estimate will miss the true effect by some amount. That chance error shrinks as the groups grow, and standard errors measure its typical size. Second, randomization balances the groups at the moment of assignment. What happens afterwards, such as students leaving the study, can undo that balance.

          Worked example · simulated data

          Self-selection versus a lottery

          The code below creates 2,000 students. Each has an unobserved motivation level that raises test scores. Tutoring raises every student’s score by exactly 5 points, so the true effect is known.

          In the first scenario, motivated students are more likely to join. The difference in means is 10.52 points, roughly double the truth. Of that, 5.52 points is selection bias. Joiners have an average motivation of 0.55 standard deviations against −0.48 for non-joiners, a gap that a real study would not observe.

          In the second scenario, a lottery offers tutoring to exactly half the students. Average motivation is now 0.05 among those offered and 0.01 among the rest, close to equal. The difference in means is 5.80 points. That is not exactly 5, and the reason is chance. Repeating the lottery 500 times gives estimates that average 4.97 with a standard deviation of 0.44, and the middle 95 percent fall between 4.14 and 5.82. The single lottery happened to land near the upper end, which a standard error would warn us about.

          The second script supports the later section on what randomization promises. It repeats the assignment of one fixed sample of 60 students 5,000 times, with and without an adjustment for a pretest.

          Run it yourself in R: code and output
          Limits

          What randomization does and does not promise

          Randomization balances the groups on average over many possible assignments, and it does so for every characteristic, measured or not. It does not promise that the one assignment a study happens to draw is balanced. Deaton and Cartwright (2018) describe a chance imbalance of this kind, which epidemiologists call realized confounding. An estimate can be unbiased across assignments and still be far from the truth in the single trial that was run, so unbiasedness alone says little about how much to trust one result. What randomization does supply is a basis for judging how large the chance error is, which is what a standard error reports.

          The second script in the code panel under the worked example shows this with one fixed sample of 60 students and a true effect of 3 points. Each student has an unmeasured ability that raises scores, and a pretest that measures ability with some noise. The script then repeats the random assignment 5,000 times on the same 60 students. The estimates average 3.02, so they are unbiased, yet their standard deviation is 2.33 and 40 percent of them are more than 2 points from the truth. A regression that also uses the pretest, chosen before seeing any outcome, averages 3.01 with a standard deviation of 1.69, and 24 percent of its estimates are more than 2 points off. Adjusting for a variable that predicts the outcome does not replace randomization. It reduces the chance error that randomization leaves behind.

          A tempting response is to test whether the groups differ on the pretest and to worry only if the test is significant. When assignment was carried out as designed, the null hypothesis of no difference is true by construction, so a significant result is a chance finding and says nothing about whether randomization worked (Deaton & Cartwright, 2018). Such a test is appropriate only when there is reason to doubt the assignment itself, for example a failed random number generator or participants who undermined the allocation. In the simulation the test rejected at the 5 percent level in 5 percent of the assignments, as it should. The flagged assignments nonetheless had a mean absolute error of 3.96 points for the unadjusted estimate, against 1.78 for the rest, because a large gap on a variable that predicts the outcome carries over to the estimate. The useful quantity is the size of the gap, measured by a standardized difference rather than a p-value (Deaton & Cartwright, 2018), and the useful response is to plan the adjustment in advance.

          One further limit concerns where the result applies. A randomized trial estimates the effect for the students, schools, and version of the program it studied. Whether the effect would be the same elsewhere is a separate question that the randomization does not answer, and it needs knowledge about how the program works and how the settings differ.

          Assumptions

          What has to be true

          Random assignment removes selection bias, but a credible experiment rests on a few further conditions.

          • The assignment really is random. A principal who quietly swaps a favored student into the program breaks the lottery. Good studies document how assignment was done and check that the groups look similar on measured characteristics.
          • No interference between students. One student’s assignment should not change another student’s outcome. If tutored students share notes with friends in the control group, the comparison understates the effect. Rubin calls the combination of this condition and a single version of the treatment the stable unit treatment value assumption (Imbens & Rubin, 2015).
          • Outcomes are observed for everyone, or missingness is unrelated to assignment. If the program keeps weaker students enrolled who would otherwise have left, the tested groups are no longer comparable. The Expert lesson returns to this problem.
          • Students receive what they were assigned. When some winners never attend, the difference in means estimates the effect of the offer, not of tutoring itself. That is often the policy-relevant quantity, but it must be labeled correctly.
          Case · published study

          Class size in Tennessee

          Tennessee’s Project STAR is among the best-known education experiments. Beginning with students who entered kindergarten in 1985, participating schools randomly assigned students and teachers to small classes of 13 to 17 students, regular classes of 22 to 25, or regular classes with a full-time teacher’s aide. Each school had to have at least one class of each type, and assignment happened within schools, so comparisons between class types are not confounded by differences between schools or neighborhoods. Over four years the experiment involved about 11,600 students in 80 schools.

          Krueger (1999) reanalyzed the data. Because randomization took place within schools, his regressions include school fixed effects. Students in small classes scored about 5 to 7 percentile points higher on the Stanford Achievement Test than students in regular classes, effect sizes of 0.20 in kindergarten, 0.28 in first grade, 0.22 in second grade, and 0.19 in third grade. Adding student and teacher characteristics barely changed these estimates, which is what random assignment predicts. A full-time aide made little difference, about 1 to 2 percentile points, significant in only one grade.

          He also checked whether the randomization was carried out as designed. For 1,581 students in 18 schools he recovered the original assignment sheets and found that only 0.3 percent of students were not enrolled in the class type they were assigned in kindergarten. Within schools, free-lunch status, race, and age were unrelated to class type. That combination, an estimate based on the original lottery plus explicit checks on what could have gone wrong, shows how any experiment should be reported.

          Cautions

          What can still go wrong

          • Attrition.Students who leave the study before the outcome is measured can make the remaining groups differ, especially if the program affects who stays.
          • Noncompliance.Winners who do not attend, or losers who find their way in, blur the contrast between the groups.
          • Spillovers.Students, teachers, or schools in the control group may be affected by the program, for example when a tutored student’s classmates also benefit.
          • Small samples.With few students or few schools, chance imbalances are large and the estimate is imprecise.
          • Generalization.An experiment estimates the effect for the students, schools, and version of the program it studied. Scaling up can change all three.
          Design

          Blinding and allocation concealment

          Two safeguards have similar names and protect against different problems (Schulz & Grimes, 2002). Allocation concealment keeps the people who enroll students from knowing the next assignment before they enroll. Without it, a teacher who can see that the next seat is tutoring might steer the neediest student toward it, and the groups would differ by selection before the program had any effect. Blinding keeps students, the people who deliver the program, or the people who measure the outcome from knowing who received what after assignment. It limits behavior that follows from knowing, such as control students seeking help elsewhere, and measurement that tilts toward the expected result, such as a rater who scores an essay more kindly when the rater knows the student was tutored.

          Concealment can almost always be done, for example by running the lottery centrally, away from the school. Blinding often cannot, because students know whether they are tutored. Even then the outcome assessment can usually be blinded, for instance by having essays scored by people who do not know the group. Objective outcomes, such as a district exam scored by machine, need less protection than ratings by adults who know the assignment.

          A report should say who was blinded and how, not only that the study was blinded, because terms such as double blind mean different things to different readers (Schulz & Grimes, 2002). Deaton and Cartwright (2018) note that randomization does not guarantee an unbiased estimate if differences that arise after assignment affect the two groups, and they recommend blinding participants, investigators, outcome assessors, and analysts where possible. They add that many social trials cannot be blinded, so a report should explain why the lack of blinding is unlikely to bias the result.

          Practice

          How to read a randomized experiment

          1. What was randomized, and at what level? Students, classrooms, and schools lead to different analyses. The Advanced lesson explains why.
          2. Do the groups look alike at baseline? A balance table should compare measured characteristics. Large differences suggest a problem with assignment.
          3. How many were lost, and from which group? Look for attrition rates by group, not only overall.
          4. Did students receive what they were assigned? If not, check whether the main estimate is for the offer or for participation.
          5. How precise is the estimate? Read the confidence interval, not only whether the result is statistically significant.
          6. For whom does it hold? Ask how the study’s students and schools compare with the ones you care about.
          Cautions

          Common misreadings

          • “It was randomized, so the estimate equals the true effect.”Randomization removes selection bias but not chance error. One lottery can still miss by a meaningful amount.
          • “The groups differ on one baseline variable, so randomization failed.”With many variables, some will differ by chance. A pattern of large differences is the warning sign.
          • “Only compare students who completed the program.”Dropping students based on what happened after assignment brings selection bias back.
          • “No significant effect means no effect.”An imprecise study may be unable to detect a meaningful effect. Look at the interval.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.Chapters on experiments written for education researchers. Start here.
          • Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. https://doi.org/10.1080/01621459.1986.10478354The fundamental problem of causal inference.
          • Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350The potential-outcomes view of causal effects, published in an education journal.
          • Krueger, A. B. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532. https://doi.org/10.1162/003355399556052The Tennessee class-size experiment.
          • Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. https://doi.org/10.1017/CBO9781139025751The standard graduate textbook on experiments and potential outcomes.
          • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.The standard statement of the three conditions, and of what an effect is.
          • Hill, A. B. (1965). The environment and disease: Association or causation? Proceedings of the Royal Society of Medicine, 58(5), 295–300. https://doi.org/10.1177/003591576505800503The nine viewpoints for judging an association, in the author’s own words.
          • Rosner, D. (2015). Criteria for action in population health: The Hill criteria a half century later. The Milbank Quarterly, 93(2), 259–262. https://doi.org/10.1111/1468-0009.12121A short reassessment of Hill’s list fifty years on, including what he did not call it.
          • LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604–620.The job-training comparison whose raw gap has the wrong sign.
          • Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062. https://doi.org/10.1080/01621459.1999.10473858The sample of that experiment used in Module 2, with the matching estimates.
          • Berkson, J. (1946). Limitations of the application of fourfold table analysis to hospital data. Biometrics Bulletin, 2(3), 47–53. https://doi.org/10.2307/3002000How selecting on admission to hospital can create an association from nothing.
          • Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A structural approach to selection bias. Epidemiology, 15(5), 615–625. https://doi.org/10.1097/01.ede.0000135174.63482.43Selection bias set out in causal graphs, including selecting on a common effect.
          • Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2), 238–241. https://doi.org/10.1111/j.2517-6161.1951.tb00088.xThe original note on associations that change with grouping.
          • Bickel, P. J., Hammel, E. A., & O’Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404. https://doi.org/10.1126/science.187.4175.398The best documented case of an association that reverses when groups are separated.
          • Messerli, F. H. (2012). Chocolate consumption, cognitive function, and Nobel laureates. New England Journal of Medicine, 367(16), 1562–1564. https://doi.org/10.1056/NEJMon1211064The chocolate and Nobel correlation, offered by its author as hypothesis-generating.
          • Spirtes, P., Glymour, C., & Scheines, R. (1993). Causation, prediction, and search. Springer-Verlag. https://doi.org/10.1007/978-1-4612-2748-9Where the faithfulness assumption is stated, in the causal graph framework.
          • Hernán, M. A. (2018). The C-word: Scientific euphemisms do not improve causal inference from observational data. American Journal of Public Health, 108(5), 616–619. https://doi.org/10.2105/AJPH.2018.304337Why writing association where you mean cause does not make a claim safer.
          • Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4), 669–688. https://doi.org/10.1093/biomet/82.4.669The article that introduced causal diagrams for empirical research.
          • Imbens, G. W. (2020). Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics. Journal of Economic Literature, 58(4), 1129–1179. https://doi.org/10.1257/jel.20191597A balanced comparison of the potential outcome and diagram approaches for applied work.
          • Cinelli, C., Forney, A., & Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research, 53(3), 1071–1104. https://doi.org/10.1177/00491241221099552A short guide to which control variables help and which harm, with examples.
          • Deaton, A., & Cartwright, N. (2018). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210, 2–21. https://doi.org/10.1016/j.socscimed.2017.12.005What randomization does and does not guarantee, and why a single trial can still mislead.
          • Schulz, K. F., & Grimes, D. A. (2002). Blinding in randomised trials: Hiding who got what. The Lancet, 359(9307), 696–700. https://doi.org/10.1016/s0140-6736(02)07816-9Blinding and allocation concealment, who can be blinded, and how to report it.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 1 · Intermediate

          Estimation and inference in experiments.

          Randomization makes the difference in means an unbiased estimate of the average effect. This lesson addresses how uncertain the estimate is, how a pretest can make it more precise without introducing bias, how a blocked design should be analyzed, and how an effect can be tested without relying on large-sample approximations.

          Before thisFoundations: why random assignment worksPotential outcomes and selection bias.
          After thisAdvanced: cluster-randomized trials and powerWhen whole schools are assigned.
          By the end

          What you will be able to do

          1. Define the sample and population average treatment effects and say which one a given analysis targets.
          2. Compute the Neyman standard error and explain why it is conservative.
          3. Use regression adjustment for baseline covariates, including the interacted estimator of Lin (2013).
          4. Decide which variables belong in an adjustment model and which should stay out.
          5. Analyze a blocked experiment in a way that matches its design.
          6. Carry out a randomization test of the sharp null hypothesis.
          7. Describe how pre-specification and standardized balance reporting protect an experiment against analytic flexibility.
          The target

          Which average effect?

          Write \(Y_i(1)\) and \(Y_i(0)\) for student i's potential outcomes with and without the program. Two averages are common targets. The sample average treatment effect is the mean of \(Y_i(1) - Y_i(0)\) over the students actually in the experiment. The population average treatment effect is the same mean over a larger population from which the students are imagined to have been sampled.

          The distinction matters for uncertainty. For the sample effect, the only source of randomness is which students happened to be assigned to treatment. For the population effect, there is the additional randomness of which students entered the study. Most school experiments use convenience samples of volunteering schools, so the sample effect is often the more defensible target, and claims about a population need an argument beyond the experiment (Imbens & Rubin, 2015).

          Uncertainty

          The Neyman standard error

          With \(n_1\) treated and \(n_0\) control students, the difference in means is \(\hat\tau = \bar Y_1 - \bar Y_0\). Neyman’s variance estimator is

          \[ \widehat{V}(\hat\tau) = \frac{s_1^2}{n_1} + \frac{s_0^2}{n_0}, \]

          where \(s_1^2\) and \(s_0^2\) are the outcome variances in each group. The exact variance of \(\hat\tau\) across possible assignments has a third term that depends on how much individual effects vary. That term cannot be estimated, because we never see both potential outcomes for any student, and it always reduces the variance. Leaving it out makes the Neyman estimator conservative for the sample effect, too large on average unless every student has the same effect (Imbens & Rubin, 2015).

          The same formula is the heteroskedasticity-robust standard error from a regression of the outcome on a treatment indicator, so standard software matches it when robust standard errors are requested, exactly for the HC2 variant and closely for the others. The conventional regression standard error assumes equal variances in the two groups and can be too small or too large, depending on which group has the larger variance.

          Case · worked by Imbens and Rubin

          A Neyman analysis of teacher incentives

          Imbens and Rubin (2015) illustrate Neyman’s approach with an experiment by Duflo, Hanna, and Ryan (2012) in single-teacher schools in rural India. In 57 of 113 schools, chosen at random, teachers’ pay was tied to their attendance over each month. Teachers in all schools received time-stamped cameras to photograph the class at the start and end of each day, and officials made unannounced visits. Imbens and Rubin set aside six schools with missing data, leaving 107. The outcome here is the share of unannounced visits on which the school was open.

          QuantityControl schoolsIncentive schools
          Schools5453
          Share of visits with school open (mean)0.580.80
          Standard deviation0.190.13

          Source is Imbens and Rubin (2015), Tables 6.1 to 6.3.

          The estimated effect is 0.80 − 0.58 = 0.22, so incentive schools were open on 22 percentage points more of the visits. The Neyman standard error is \(\sqrt{0.19^2/54 + 0.13^2/53} = 0.031\), and, working from the unrounded estimate of 0.215, the 95 percent interval runs from 0.15 to 0.28. Two details stand out. The control schools’ variance is more than twice the incentive schools’, which is why the unequal-variance formula is the safe default. And the same experiment shows that an effect on the immediate target, teachers showing up, need not carry through equally to every downstream outcome, since effects on students’ writing scores were smaller and less precise.

          Precision

          Adjusting for a pretest

          Much of the variation in end-of-year scores is predictable from last year’s scores. Removing that predictable part leaves less noise, and so a more precise estimate. The usual approach is a regression of the outcome on the treatment indicator and the centered pretest, often called ANCOVA.

          Critics of this practice pointed out that the adjustment can be biased in small samples and can even reduce precision when effects vary. Lin (2013) reexamined that critique and showed that a simple change removes the second concern. Include the interaction of treatment with the centered covariate and use robust standard errors,

          \[ Y_i = \alpha + \tau D_i + \beta (X_i - \bar X) + \gamma D_i (X_i - \bar X) + \varepsilon_i. \]

          Because the covariate is centered at its full-sample mean, \(\tau\) remains the average effect. Lin shows that, in large samples, this estimator is at least as precise as the unadjusted difference in means, and that robust standard errors are valid or conservative. The small-sample bias does not disappear, but it shrinks as the sample grows (Lin, 2013).

          One rule keeps the adjustment valid. Covariates must be measured before assignment, or be unaffected by it. Adjusting for a variable that the program changed, such as attendance during the program year, can introduce bias.

          Lin built a simulation to show when the interaction matters. The covariate was chosen to predict the treatment effect far better than the outcome, which is the setting where the usual adjustment can backfire.

          Share assigned to treatmentUnadjustedUsual regression adjustmentAdjustment with interaction
          0.759317180
          0.50524646
          0.2514318098

          Asymptotic standard deviations multiplied by 1,000, with 1,000 subjects. The covariate was built to be related much more to the treatment effect than to the outcomes, conditions unfavorable to adjustment. Source is Lin (2013), Table 1.

          With half the sample treated, the usual adjustment and the interacted version are equally precise, a case Lin describes as two wrongs making a right. With unequal groups, the usual adjustment is less precise than no adjustment at all, while the interacted version is more precise than both. Lin also notes that the strongest remaining reasons to prefer unadjusted estimates are transparency and protection against searching over covariate specifications, which a pre-analysis plan addresses.

          Choosing covariates

          Which variables belong in the model

          The covariate rule in the previous section can be stated more fully with the three roles from the Foundations lesson. A variable measured before assignment cannot have been changed by it, so adjusting for it is safe and the only question is whether it helps. A variable measured after assignment may have been changed by the program, and adjusting for it can change what the estimate means.

          The second script in the code panel under the worked example puts numbers on this. It creates 400 students, assigns half to tutoring at random, and draws four further variables. A pretest, measured before assignment, predicts scores. Hours practiced rise with tutoring, and scores rise with hours. Unmeasured motivation raises scores and also makes students more likely to enroll in an advanced course, which tutoring encourages as well. The total effect of tutoring is 4 points, a direct effect of 1 plus 1.5 extra hours at 2 points each.

          Adjusting forMean estimateStandard deviation
          Nothing4.040.90
          Pretest4.010.66
          Pretest and hours practiced (mediator)1.010.76
          Pretest and course enrollment (collider)2.030.64

          Simulated data, 500 repeated experiments of 400 students. The true total effect is 4. R, Python and Stata produce these numbers.

          With no adjustment the estimate is centered on 4, the true total effect, as random assignment guarantees. Adding the pretest leaves it centered there and lowers the standard deviation from 0.90 to 0.66, the precision gain described in the previous section. Adding hours practiced moves the estimate to about 1, the direct effect alone. That can be a legitimate question, but it is no longer the total effect of the program on scores, and with real data it would also need the assumption that nothing unmeasured affects both hours and scores. Adding course enrollment moves the estimate to about 2, even though in this simulation a student’s course choice changes nobody’s score. The shift comes entirely from adjusting for a variable that tutoring and motivation both influence.

          Three habits follow. Write the covariate list before looking at outcomes, which the pre-specification section returns to. Include only variables fixed before assignment. Treat variables measured afterward as outcomes in their own right, or analyze them in a separate and clearly labeled mediation analysis. In the observational designs of Module 2, adjusting for pre-treatment variables also serves to remove confounding, which is a stronger demand on the covariate list.

          Design

          Blocking: analyze as you randomize

          Many school experiments randomize within blocks, for example by assigning half the applicants in each school. Blocking guarantees balance on whatever defines the blocks and removes between-block variation from the comparison. The analysis should reflect that design. Including block indicators in the regression, or averaging within-block differences, gives standard errors that match the design. Ignoring the blocks leaves the estimate valid but makes the standard error too large, which gives up the precision gain that blocking provides.

          When the share treated differs across blocks, a regression with block indicators weights blocks by both their size and the variance of treatment within them. If effects differ across blocks, that weighting can differ from the average effect across all students. Averaging the block-specific differences with weights proportional to block size targets the average effect directly (Athey & Imbens, 2017).

          Worked example · simulated data

          Four estimators, one experiment

          The first script creates 200 students in 10 schools and assigns 10 of the 20 students in each school to a program. The effect varies with the pretest, and the sample average effect is 4.14 points. In this assignment, the difference in means is 4.83 with a Neyman standard error of 1.55. The ANCOVA and Lin estimates are both 4.75, with robust standard errors near 1.04. Adding school indicators gives 4.74 with a standard error of 0.80.

          The second half re-randomizes the same students 500 times, which shows how each estimator behaves across the experiments that could have happened. All four are centered near 4.1. The difference in means has a true standard deviation of 1.48, and its Neyman standard error averages 1.55, slightly conservative as the theory predicts. Adjusting for the pretest cuts the true standard deviation to about 0.80, which is the same precision as more than tripling the sample. The standard errors differ, however. Without school indicators they average 1.04, well above the true 0.80, because the analysis ignores the blocking. With school indicators they average 0.80, matching the design.

          The second script supports the earlier section on what to adjust for. It repeats a smaller experiment 500 times with a pretest, a mediator and a collider available, and shows that only the pretest is safe to add.

          Run it yourself in R: code and output
          Testing

          Randomization inference

          Standard errors rely on large-sample approximations. With small experiments there is an alternative that relies only on the design. Fisher’s sharp null hypothesis states that the program had no effect on any student, so each student’s observed score would be the same under any assignment. If that is true, we can recompute the difference in means for every possible lottery, or a large random sample of them, holding the scores fixed. The p-value is the share of those differences at least as large in magnitude as the one observed (Imbens & Rubin, 2015).

          The third script has 24 students, 12 assigned to a reading program that adds 10 points, and three students with unusually high scores. The difference in means is 12.50 points. The normal approximation gives p = 0.056. Randomization inference with 5,000 draws gives p = 0.071, a little larger. The test is exact for the sharp null and can use any test statistic. Using the difference in mean ranks, which is less sensitive to the three outliers, gives p = 0.032 here. The choice of statistic must be made before seeing the data, or the test loses its validity.

          The test has two limits. The sharp null is stronger than the hypothesis that the average effect is zero. And a randomization test answers whether there is any effect, while a confidence interval for its size needs further assumptions, for example that the effect is the same for everyone.

          Credibility

          Pre-specification and multiple outcomes

          An experiment gives the researcher many choices after the data arrive, such as which covariates to adjust for, which outcomes to emphasize, and which subgroups to report. Each choice is defensible on its own, but trying several and reporting the most favorable inflates the chance of a false positive. Writing down the primary outcome, the covariates, and the estimator before seeing outcomes limits this concern. Where many outcomes are tested, reporting results for all of them, or adjusting for multiple comparisons, keeps the error rate under control.

          A balance table also deserves care. Testing dozens of baseline variables will flag a few by chance. Reporting standardized differences, the gap in group means divided by a standard deviation so that variables on different scales can be compared, and adjusting for prespecified covariates regardless of whether they look imbalanced, is better practice than choosing covariates because a balance test happened to reject.

          The Oregon Health Insurance Experiment shows how this can be done in a large study. Before looking at outcomes for the treatment group, the researchers archived a detailed analysis plan, a practice standard in medical trials but rare in social policy experiments at the time. Because they examined many outcomes, they grouped them into domains, summarized each domain with a standardized treatment effect, and reported p-values adjusted for the number of outcomes within each domain (Finkelstein et al., 2012). Analyses that were not in the plan were labeled as such.

          Reporting

          What to report

          1. The unit and method of randomization, including blocks.
          2. A balance table with standardized differences.
          3. Attrition overall and by group.
          4. The unadjusted difference in means alongside the prespecified adjusted estimate.
          5. Standard errors that match the design, with the method named.
          6. Confidence intervals and effect sizes in meaningful units.
          Cautions

          Limits of these tools

          • Students are not always the unit of assignment.When classrooms or schools are randomized, student-level standard errors are far too small. The Advanced lesson covers this.
          • Adjustment cannot fix a broken lottery.Covariates improve precision in a valid experiment. They do not remove bias from a compromised one.
          • Small samples weaken large-sample formulas.With small groups, randomization inference is a useful check.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. https://doi.org/10.1017/CBO9781139025751Neyman’s variance, Fisher’s exact test, and stratified experiments.
          • Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique. Annals of Applied Statistics, 7(1), 295–318. https://doi.org/10.1214/12-AOAS583Why the interacted regression with robust standard errors is safe.
          • Athey, S., & Imbens, G. W. (2017). The econometrics of randomized experiments. In A. V. Banerjee & E. Duflo (Eds.), Handbook of economic field experiments (Vol. 1, pp. 73–140). North-Holland. https://doi.org/10.1016/bs.hefe.2016.10.003A practical survey, including blocking and clustering.
          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.Experiments in education, with worked examples.
          • Duflo, E., Hanna, R., & Ryan, S. P. (2012). Incentives work: Getting teachers to come to school. American Economic Review, 102(4), 1241–1278. https://doi.org/10.1257/aer.102.4.1241The teacher-incentive experiment used in the worked example.
          • Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J. P., Allen, H., Baicker, K., & Oregon Health Study Group (2012). The Oregon health insurance experiment: Evidence from the first year. Quarterly Journal of Economics, 127(3), 1057–1106. https://doi.org/10.1093/qje/qjs020A pre-registered social experiment with many outcomes.
          • Cinelli, C., Forney, A., & Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research, 53(3), 1071–1104. https://doi.org/10.1177/00491241221099552Which controls help and which harm, with the diagram vocabulary used in this section.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 1 · Advanced

          Cluster-randomized trials and power.

          Many education programs operate on whole classrooms or schools, so whole classrooms or schools are randomized. This changes how much information the study contains, how it must be analyzed, and how many schools it needs. This lesson covers all three.

          Before thisIntermediate: estimation and inferenceNeyman standard errors and covariate adjustment.
          After thisExpert: noncompliance, attrition, and boundsWhen the experiment does not go to plan.
          By the end

          What you will be able to do

          1. Explain why the intraclass correlation determines how much information a clustered sample contains.
          2. Compute a design effect and an effective sample size.
          3. Analyze a cluster-randomized trial with valid standard errors.
          4. State at which level standard errors should be clustered and what changes when there are few clusters.
          5. Calculate a minimum detectable effect size and identify which design choices change it.
          6. Choose the number of students per school for a given budget.
          7. Use pretests and blocking to improve the precision of a cluster trial.
          The setting

          Why randomize schools?

          A new mathematics curriculum, a principal coaching program, or a school-wide attendance policy cannot be given to some students in a school and withheld from others. Even when it could, teachers who use a new method with half their class may use it with the other half too, which would contaminate the comparison. So researchers randomize at the level where the program operates, assigning whole classrooms, schools, or districts. Raudenbush (1997) set out the statistical analysis and optimal design of these cluster-randomized trials for education research.

          The logic of randomization is unchanged. Chance decides which schools receive the program, so treated and control schools are comparable on average. What changes is the amount of independent information. Students in the same school share teachers, peers, and neighborhoods, so their outcomes are correlated. A thousand students in 40 schools contain less information than a thousand students sampled independently, by an amount that depends on the intraclass correlation.

          Key quantity

          The intraclass correlation

          Split the variance of student outcomes into a between-school part \(\tau^2\) and a within-school part \(\sigma^2\). The intraclass correlation is the between-school share,

          \[ \rho = \frac{\tau^2}{\tau^2 + \sigma^2}. \]

          With m students per school, the variance of a school-randomized estimate is larger than that of a student-randomized estimate with the same total sample by a factor equal to the design effect,

          \[ \text{DEFF} = 1 + (m - 1)\rho. \]

          With 25 students per school and \(\rho = 0.15\), the design effect is 4.60. A trial of 1,000 students then has the precision of about 1,000 / 4.60 ≈ 217 independent students. Hedges and Hedberg (2007) compiled intraclass correlations for achievement from national samples to help researchers plan such trials, and they also report how much pretests reduce the variance at each level.

          Their compilation gives planning values. Across grades K to 12, the average intraclass correlation for achievement was about 0.22 in all schools, 0.19 in low-SES schools, and 0.09 in low-achieving schools, similar in reading and mathematics. Except in low-achieving schools, these exceed the 0.05 to 0.15 range often assumed. Pretests typically explained over three quarters of the between-school variance and over half of the within-school variance. Demographic characteristics could explain up to half of the between-school variance, but added little once a pretest was included.

          Low ICCS1S2S3S4High ICCS1S2S3S4Each column is one school. Dashes mark school means.
          What the intraclass correlation measures. Simulated scores for four schools in each panel. On the left, school means sit close together and most variation is within schools, a low ρ. On the right, schools differ far more than students within them, a high ρ. In the right panel a new student from an already-sampled school adds little information, which is why the design effect grows with both ρ and the number of students per school.
          Analysis

          Analyzing a cluster trial

          Three analyses give valid inference when schools are randomized.

          • School means. Average the outcome within each school and compare treated and control school means. This turns the study into a simple experiment with as many units as schools.
          • Student-level regression with cluster-robust standard errors. The point estimate uses all students, and the standard errors allow arbitrary correlation within schools. With few schools, use small-sample corrections and t critical values with degrees of freedom based on the number of schools.
          • Multilevel models. A two-level model with a random school intercept, as in Raudenbush (1997), gives efficient estimates when its assumptions hold and makes it easy to include student and school covariates.

          The analysis to avoid is a student-level regression with conventional standard errors. It treats 1,000 correlated students as 1,000 independent pieces of information and so reports far more certainty than the data contain.

          Recent research

          When should standard errors be clustered?

          A common rule says to cluster whenever outcomes are correlated within groups, and that clustering is harmless when unnecessary. Abadie, Athey, Imbens, and Wooldridge (2023) show that neither claim holds in general. In their framework the need to cluster comes from the design, either from how units were sampled or from how treatment was assigned, not from correlation in the outcomes. If students are sampled at random and treatment is assigned to individual students at random, conventional robust standard errors are appropriate even when outcomes are strongly correlated within schools. If treatment is assigned to whole schools, as in a cluster-randomized trial, clustering by school is required, because treatment is assigned to whole schools rather than to individual students.

          Unnecessary clustering has a cost. Cluster standard errors are conservative when most clusters in the population are observed and treatment effects vary across clusters, and they can be conservative by a wide margin. In a census sample of 2,632,838 workers across the 50 states, the District of Columbia, and Puerto Rico, their regression of log earnings on an individual indicator for some college gave a robust standard error of 0.0012 and a state-clustered one of 0.0269, more than 20 times larger. Their proposed causal cluster variance gave 0.0035, between the two.

          For a school-randomized education trial, the conclusion is simple, because assignment is clustered. Cluster at the level of assignment. The harder cases are observational studies and experiments that assign students within schools, where the answer depends on how schools entered the sample and how treatment varied within them.

          Worked example · simulated data

          What the wrong standard error does

          The first code example simulates 40 schools with 25 students each, an intraclass correlation of 0.15, and outcomes in standard-deviation units. In one trial with a true effect of 0.20, the estimate is 0.050. The naive standard error is 0.064, while the cluster-robust and school-mean standard errors are 0.129 and 0.130, about twice as large, close to the square root of the design effect.

          The second part runs 400 trials with no effect at all. When the true effect is zero, a 5 percent test should reject in about 5 percent of trials. With naive standard errors the null is rejected in 41.2 percent of trials. The estimates actually vary with a standard deviation of 0.148, while the naive standard error averages 0.063. With cluster-robust standard errors and school means, the rejection rates are 7.2 percent and 7.0 percent, close to 5 percent though slightly high, as is common with 40 clusters and no small-sample correction. A study that ignores clustering would report false positives in many null trials.

          Run it yourself in R: code and output
          Planning

          Minimum detectable effect size

          Before a trial, researchers ask how large an effect it could reliably detect. With J schools, a share P treated, and m students per school, the minimum detectable effect size in standard-deviation units is approximately

          \[ \text{MDES} = M_{J-2}\sqrt{\frac{\rho(1 - R_2^2)}{P(1-P)J} + \frac{(1-\rho)(1 - R_1^2)}{P(1-P)Jm}}, \]

          where \(M_{J-2}\) is the sum of two t quantiles, about 2.8 for 80 percent power and a 5 percent two-sided test with many schools, and \(R_2^2\) and \(R_1^2\) are the shares of school-level and student-level variance explained by covariates (Raudenbush, 1997).

          The formula shows where precision comes from. The first term shrinks only with the number of schools. The second shrinks with the number of students, but it is already small when \(\rho\) is large. Adding students within schools quickly stops helping, while adding schools keeps helping.

          Worked example · calculation

          Planning a trial

          The second code example computes the formula for several designs. With 25 students per school and \(\rho = 0.20\), 40 schools can detect an effect of 0.438 standard deviations. Quadrupling students to 100 per school only lowers that to 0.415. Doubling schools to 80 lowers it to 0.306. A pretest that explains half the variance at both levels brings 40 schools down to 0.310, roughly the same gain as doubling the number of schools, usually at lower cost.

          The code then checks the formula by simulation. With a true effect equal to the computed minimum of 0.438, 400 simulated trials reject the null 79.0 percent of the time, close to the intended 80 percent.

          Kraft (2020) proposes that, for randomized trials with standardized achievement outcomes, an effect of 0.05 standard deviations is small and 0.20 is large, so many realistic effects fall well below 0.30 and trials with a few dozen schools are often underpowered for them. Planning with realistic intraclass correlations and covariate R² values from sources such as Hedges and Hedberg (2007) helps avoid this.

          Design

          Design choices that help

          • Pretests.School-level pretest means can explain much of the between-school variance, which is the term that usually dominates precision.
          • Blocking or pairing schools.Randomizing within pairs or blocks of similar schools guarantees balance on the blocking variables and can improve precision.
          • More schools, fewer students per school.When the intraclass correlation is moderate, extra schools add more precision than extra students. Optimal designs weigh this against the cost of recruiting each school (Raudenbush, 1997).
          • Unequal allocation rarely helps.Precision is greatest when half the schools are treated, unless treatment schools cost much more than control schools.
          Design

          How many students per school?

          Raudenbush (1997) treats the design of a cluster trial as a budget problem. If enrolling a school costs \(C_2\) and each additional student costs \(C_1\), the number of students per school that minimizes the variance of the treatment effect is

          \[ n_{\text{opt}} = \frac{\sigma}{\tau}\sqrt{\frac{C_2}{C_1}}, \]

          where \(\tau^2\) and \(\sigma^2\) are the between- and within-school variances. Sampling many students per school makes sense when variation lies mostly within schools and schools are expensive to recruit. The remaining budget then sets the number of schools.

          Intraclass correlationStudents per school, no covariateSchools, no covariateVariance, no covariateStudents per school, with pretestSchools, with pretestVariance, with pretest
          0.0514210.022619170.0091
          0.109260.030413220.0116
          0.206310.04269270.0152

          Total budget of 500 units, a cost of 1 per student and 10 per school, and outcomes standardized to unit variance. The pretest explains 73 percent of between-school and 48 percent of within-school variance, values that Raudenbush takes from Bloom (1995). Source is Raudenbush (1997), Tables 2 and 3.

          The table shows the main lesson. A pretest that explains most of the between-school variance cuts the variance of the estimate by more than half in each row, so a design with a pretest needs fewer schools for the same precision. In the three rows shown, the design without the covariate was only 36 percent to 40 percent as efficient as the design with it (Raudenbush, 1997, Table 4). Choosing covariates that explain variation between schools is as important as choosing the number of schools.

          Cautions

          When schools are few

          Cluster-robust standard errors are justified when the number of clusters is large. When the number of schools is small they tend to be too small. Remedies include small-sample corrections, t critical values with degrees of freedom based on the number of schools, the wild cluster bootstrap covered in the difference-in-differences module (Cameron et al., 2008), and randomization inference at the school level, which remains exact however few schools there are.

          Very small trials, with a handful of schools per arm, are better seen as pilot studies. They can test whether a program can be delivered, but chance differences between a few schools can easily be as large as any plausible effect.

          Reporting

          What to report

          1. The unit of randomization and the number of clusters in each arm.
          2. The intraclass correlation estimated in the study, with and without covariates.
          3. The analysis method and how it accounts for clustering.
          4. The minimum detectable effect size the design could achieve, computed before the trial.
          5. Effect sizes with confidence intervals in standard-deviation units and in natural units.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185. https://doi.org/10.1037/1082-989X.2.2.173Analysis and optimal design of cluster trials. The source of the MDES logic here.
          • Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706Realistic intraclass correlations for planning education trials.
          • Athey, S., & Imbens, G. W. (2017). The econometrics of randomized experiments. In A. V. Banerjee & E. Duflo (Eds.), Handbook of economic field experiments (Vol. 1, pp. 73–140). North-Holland. https://doi.org/10.1016/bs.hefe.2016.10.003Clustered and stratified experiments from an econometric perspective.
          • Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3), 414–427. https://doi.org/10.1162/rest.90.3.414Inference with few clusters, including the wild cluster bootstrap.
          • Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. M. (2023). When should you adjust standard errors for clustering?. Quarterly Journal of Economics, 138(1), 1–35. https://doi.org/10.1093/qje/qjac038A design-based answer to when and at what level to cluster.
          • Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189x20912798Benchmarks for judging whether an effect size is small or large in education research.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 1 · Expert

          Noncompliance, attrition, and bounds.

          Real experiments in schools rarely go exactly to plan. Some students offered a program never attend, and some students are missing when outcomes are measured. This lesson shows what can still be learned without abandoning the protection of random assignment, and what has to be assumed to learn more.

          Before thisAdvanced: cluster-randomized trialsDesign and power.
          Next moduleMatching and weightingAdjusting for observed differences when assignment is not random.
          By the end

          What you will be able to do

          1. Interpret the intention-to-treat effect and explain why it remains unbiased under noncompliance.
          2. Classify students as compliers, never-takers, always-takers, and defiers.
          3. Estimate the complier average causal effect and state the assumptions behind it.
          4. Explain why as-treated and per-protocol comparisons are biased.
          5. Compute Lee bounds when attrition differs between groups.
          6. Examine subgroup effects without inflating false positives.
          7. Read a published experiment for how it handles switching, attrition and noncompliance.
          The safe estimate

          Intention to treat

          Write \(Z_i\) for random assignment, the offer of a program, and \(D_i\) for whether the student actually participated. The intention-to-treat (ITT) effect compares outcomes by \(Z\), ignoring \(D\),

          \[ \text{ITT} = E[Y \mid Z = 1] - E[Y \mid Z = 0]. \]

          Because \(Z\) is random, the ITT is unbiased for the effect of offering the program, whatever students then do. That is often what a district wants to know, since it can offer a program but cannot compel attendance. The ITT is also the starting point of any credible analysis of an imperfect experiment. The complier effect below is built from it.

          Framework

          Four compliance types

          Angrist, Imbens, and Rubin (1996) classify units by what they would do under each assignment.

          • Compliers participate if offered and not otherwise.
          • Never-takers do not participate whatever the offer.
          • Always-takers participate whatever the offer, for example by finding a seat in a similar program elsewhere.
          • Defiers do the opposite of their assignment.

          We cannot tell any individual’s type, because we see each student under only one assignment. A student who was offered tutoring and attended could be a complier or an always-taker. But randomization means the mix of types is the same, on average, in both arms, and that is enough to identify the effect for compliers.

          Would take the treatment if offered?YesNoWould take itif not offered?NoYesCompliertakes it only if offeredNever-takertakes it neither wayAlways-takertakes it either wayDefierdoes the opposite (assumed absent)
          The four types. Each unit has two hidden answers, one per assignment, so its type is never observed directly. Randomization makes the mix of types the same in both arms on average, and monotonicity assumes the defier cell is empty. The complier cell is the group whose average effect the ratio identifies.
          Identification

          The complier average causal effect

          Two assumptions do the work, beyond random assignment itself.

          • Exclusion. The offer affects outcomes only through participation. For never-takers and always-takers, whose participation does not change, the offer has no effect.
          • Monotonicity. There are no defiers. The offer never discourages anyone from participating.

          Under these assumptions, the whole ITT comes from compliers. Dividing by the share of compliers, which is the difference in participation rates between arms, gives the complier average causal effect (CACE),

          \[ \text{CACE} = \frac{E[Y \mid Z=1] - E[Y \mid Z=0]}{E[D \mid Z=1] - E[D \mid Z=0]}. \]

          The division also requires a nonzero first stage, which the data show directly. This is the Wald estimator, and it is the same as two-stage least squares using the offer as an instrument for participation. The instrumental variables module develops the idea in settings where the instrument is not a planned lottery. In an experiment with one-sided noncompliance, where controls cannot get the program, there are no always-takers and the CACE is also the average effect on those who participated.

          The CACE is a local effect. Compliers may differ from never-takers in ways that change how much they benefit, so the CACE need not describe the effect for students who would never attend.

          Cautions

          Two tempting mistakes

          Two analyses look natural and are both biased.

          • As treated compares participants with everyone who did not participate. Non-participants include the never-takers from the offered group, who are self-selected.
          • Per protocol compares participants in the offered group with the whole control group. The offered participants are compliers and always-takers, while the control group contains every type, including never-takers.

          Both analyses condition on participation, which students chose, and so lose the protection that random assignment gave.

          Worked example · simulated data

          An offer that not everyone takes up

          The first script offers tutoring to half of 1,000 students. About 60 percent of students would attend if offered, and those who would not attend score about 6 points lower to begin with. The program raises compliers’ scores by 6 points.

          In the offered group, 58.0 percent attended, which is the first stage. The ITT is 3.11 points with a standard error of 0.71. Dividing by 0.580 gives a CACE of 5.36, within sampling error of the true 6 (its standard error is roughly 0.71 / 0.58 ≈ 1.2). The as-treated comparison gives 10.53 and the per-protocol comparison 9.04, both far too large, because the students who chose not to attend were weaker students.

          Run it yourself in R: code and output
          Missing outcomes

          When the program changes who is measured

          Attrition does not bias the comparison when who goes missing is unrelated to the outcomes students would have had. The problem arises when the program affects who is measured. A mentoring program that keeps at-risk students enrolled will have more of them sitting the final test than the control group does. Comparing tested students then compares unlike groups, and the program can look less effective than it is.

          Reweighting by baseline characteristics helps only if missingness depends on measured variables. When that cannot be defended, bounds offer a transparent alternative. They give a range of effects consistent with the data under weaker assumptions.

          Method

          Lee bounds

          Lee (2009) assumes monotonic selection. Assignment can only make a student more likely to be measured, never less. Then the treated group’s measured students include all the students who would be measured either way, plus some who are measured only because of the program. The extra share is

          \[ p = \frac{q_1 - q_0}{q_1}, \]

          where \(q_1\) and \(q_0\) are the response rates in each arm. We do not know which treated students are the extra ones. In the worst case they are the highest scorers, and in the best case the lowest. Trimming the top share p of treated outcomes gives a lower bound, and trimming the bottom share gives an upper bound, for the effect on students who would be measured whatever their assignment.

          The second script uses 2,000 students. The program keeps more weaker students in the testing pool, so 95.9 percent of treated and 90.1 percent of control students are tested. The naive comparison gives 2.65, below the true 3.00. The trimming share is 0.060, so 58 of 959 treated scores are trimmed, and the Lee bounds are 1.51 to 3.85. The truth lies inside, and the width of the interval shows how much uncertainty the attrition adds.

          Beyond averages

          Heterogeneous effects

          Randomization also supports estimating effects for subgroups defined by baseline characteristics, such as students who started below grade level. Each subgroup is a smaller experiment, so estimates are noisier, and testing many subgroups inflates false positives. Athey and Imbens (2017) recommend prespecifying a small number of subgroups and testing for differences between them directly, rather than noting that one subgroup’s effect is significant and another’s is not. Newer machine-learning methods search for heterogeneity systematically, but they need sample splitting for valid inference.

          Case · published study

          Class size revisited

          Project STAR departed from its design in three ways that this lesson has discussed. At the start of first grade, students in regular classes were randomly reassigned between classes with and without aides. In second and third grade, 9 to 11 percent of students switched class types, often because of behavior problems or parent complaints. And about half of the students present in kindergarten were missing from the sample in at least one later year.

          Krueger (1999) handled switching by estimating effects of students’ initial assignment, an intention-to-treat analysis. These estimates were only slightly smaller than those based on the class students actually attended. He then used initial assignment as an instrument for the number of students in each class.

          GradeOLS, actual class size2SLS, instrumented by initial assignment
          Kindergarten−0.62−0.71
          First−0.85−0.88
          Second−0.59−0.67
          Third−0.61−0.81

          Percentile points per additional student. All models control for school effects and student and teacher characteristics. Source is Krueger (1999), Table VII.

          The 2SLS estimates imply that 10 fewer students raises the average percentile score by about 7 to 9 points. For attrition, he imputed each departing student’s most recent score and found estimates similar to those in the observed sample. He also found that before kindergarten began, 10.4 percent of students assigned to small classes, 12.2 percent of those assigned to classes with aides, and 14.3 percent of those assigned to regular classes had withdrawn. Assuming, to be extreme, that the extra withdrawals from regular classes would all have scored at the top of the distribution raises the regular-class average by only 1 to 2 percentile points. The case shows the ITT as the primary result and the bounding logic of this lesson applied to a well-known experiment.

          Case · published study

          Following an experiment into adulthood

          Chetty et al. (2011) linked the STAR records for 11,571 students to U.S. tax records and located 95 percent of them as adults. A long follow-up is only as good as its attrition, so they first checked that being found did not depend on assignment. Within schools, match rates for small and regular classes differed by less than 0.02 percentage points, and demographic controls revealed no difference. Because assignment was random and follow-up was nearly complete and balanced, the comparisons remain experimental two decades later.

          Students assigned to small classes were 1.8 percentage points more likely to be enrolled in college at age 20, against a mean of 26.4 percent, and scored higher on a summary index of adult outcomes such as home ownership and retirement savings. Differences in earnings at age 27 were not significant, but the authors stress that they were imprecisely estimated, so the data neither confirm nor rule out meaningful effects. Students randomly assigned to a kindergarten teacher with more than 10 years of experience earned about $1,093 more at 27. The authors caution that experience is correlated with other teacher traits, so this shows that the teacher mattered, not that adding experience would.

          The study also illustrates fade-out. Effects of class size and of classroom quality on test scores became statistically insignificant by eighth grade, yet effects reappeared in adult outcomes. Classroom quality did affect behavioral measures in fourth and eighth grade, such as effort and initiative, which predicted earnings. The authors call this evidence suggestive. For this lesson, the case adds two points. An ITT on a short-run outcome is not the whole story of an experiment, and a long-run follow-up keeps its causal footing only if attrition is low and unrelated to assignment.

          Case · published study

          An imperfect experiment analyzed carefully

          In 2008 Oregon reopened a capped Medicaid program for low-income adults and chose whom to invite by lottery from about 90,000 people on a waiting list. Finkelstein et al. (2012) analyzed the first year. A total of 35,169 people in 29,664 households were selected. Selection gave the household the chance to apply, but many did not apply or were found ineligible, so the experiment has one-sided noncompliance of the kind this lesson describes.

          The analysis follows the lesson’s steps. The intention-to-treat estimates compare everyone selected with everyone not selected. Selection raised the probability of having Medicaid by about 25 percentage points, the first stage. Dividing intention-to-treat effects by that first stage gives local average treatment effects of insurance for compliers, those who gained coverage because they won the lottery, roughly four times the size of the ITT effects. In the first year, the treatment group had higher health care use, including primary and preventive care and hospital stays, lower out-of-pocket spending and medical debt, and better self-reported physical and mental health.

          Three details show how much care an imperfect experiment needs. Whole households were selected, so people in larger households had better odds of selection, and every analysis controls for household size. Survey outcomes came from mail surveys with an effective response rate of 50 percent, so the authors paired them with hospital discharge and credit report records, which do not depend on anyone answering a survey. And virtually all of the analysis was prespecified in an archived plan before outcomes for the treatment group were examined (Finkelstein et al., 2012).

          Reporting

          What to report

          1. The ITT with its standard error, as the primary result.
          2. Participation rates in each arm, and the CACE with the assumptions it needs.
          3. Attrition rates by arm and baseline characteristics of those lost.
          4. Bounds when attrition differs between arms.
          5. Prespecified subgroup analyses, with tests of differences between subgroups.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. https://doi.org/10.1080/01621459.1996.10476902Compliance types and the complier effect.
          • Lee, D. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies, 76(3), 1071–1102. https://doi.org/10.1111/j.1467-937X.2009.00536.xSharp bounds under differential attrition.
          • Krueger, A. B. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114(2), 497–532. https://doi.org/10.1162/003355399556052Initial assignment, switching, and attrition in Project STAR.
          • Athey, S., & Imbens, G. W. (2017). The econometrics of randomized experiments. In A. V. Banerjee & E. Duflo (Eds.), Handbook of economic field experiments (Vol. 1, pp. 73–140). North-Holland. https://doi.org/10.1016/bs.hefe.2016.10.003Noncompliance and heterogeneity in randomized experiments.
          • Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. https://doi.org/10.1017/CBO9781139025751Noncompliance chapters with full derivations.
          • Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J. P., Allen, H., Baicker, K., & Oregon Health Study Group (2012). The Oregon health insurance experiment: Evidence from the first year. Quarterly Journal of Economics, 127(3), 1057–1106. https://doi.org/10.1093/qje/qjs020Intention-to-treat and complier effects in a large policy lottery.
          • Chetty, R., Friedman, J. N., Hilger, N., Saez, E., Schanzenbach, D. W., & Yagan, D. (2011). How does your kindergarten classroom affect your earnings? Evidence from Project STAR. Quarterly Journal of Economics, 126(4), 1593–1660. https://doi.org/10.1093/qje/qjr041Long-run outcomes of Project STAR linked to tax records, with checks on follow-up attrition.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 2 · Foundations

          Comparing like with like.

          Most school programs are not assigned by lottery. Students join, teachers refer, or principals choose. Matching and weighting methods try to recover a fair comparison by comparing participants with non-participants who look the same on the characteristics that drove participation. This lesson explains when that works and what it cannot fix.

          Before thisRandomized experiments, FoundationsPotential outcomes and selection bias.
          After thisIntermediate: propensity scoresMatching on many characteristics at once.
          By the end

          What you will be able to do

          1. Explain confounding and why a raw comparison of participants and non-participants is biased.
          2. State the two assumptions behind matching, no unmeasured confounding and overlap.
          3. Separate the design stage of a study from the analysis stage.
          4. Distinguish the average effect on participants from the average effect on everyone.
          5. Compute a stratified estimate by hand.
          6. Explain why regression adjustment becomes fragile when groups do not overlap.
          7. Explain why matching cannot remove bias from characteristics nobody measured.
          The problem

          Who joins a program is not random

          A district offers a summer learning program, and teachers encourage struggling students to attend. At the end of the next year, attendees score lower than non-attendees. That does not show that the program hurt. The attendees started lower, and the comparison mixes the program’s effect with the difference in where students started.

          A characteristic that affects both who receives a program and the outcome is called a confounder. Prior achievement is the classic confounder in education, but family income, attendance, special education status, and English learner status often play the same role. Without random assignment, every comparison must address confounders in some way.

          The idea

          Compare students who started in the same place

          If attendance depended only on prior achievement and lunch status, a fair comparison is available. Take students with low prior achievement who receive free lunch, and compare those who attended with those who did not. Inside that group, the confounders are held fixed. Do the same for every combination of characteristics, and then average the results.

          This is the logic behind every method in this module. Stratification, matching, and weighting differ in how they find comparable students, but they rest on the same idea. Within groups of students who are alike on the confounders, who attended is treated as if it were random.

          Assumptions

          Two assumptions

          • No unmeasured confounding. Among students with the same measured characteristics, attending is unrelated to the outcomes they would have had. Other names for this include selection on observables, conditional ignorability, and unconfoundedness (Rosenbaum & Rubin, 1983; Imbens, 2004). It fails if something unmeasured, such as motivation or a parent’s push, affects both attendance and scores.
          • Overlap. Every kind of student has some chance of attending and some chance of not attending. If every student in some group attended, there is no one to compare them with, and any estimate for that group rests on extrapolation.

          The first assumption cannot be tested directly, because it is about outcomes we never observe. It has to be argued from knowledge of how participation was decided, and it can be probed indirectly, as described below. The second can be checked directly, by looking at whether each kind of participant has comparable non-participants.

          lower0higherPrior achievement (standardized)Attenders, drawn upwardNon-attenders, drawn downwardattenders only
          Overlap, and where it runs out. Simulated data. In the middle of the distribution both groups are present, so attenders have comparable non-attenders. In the shaded region at the top of the distribution every student attended. For them there is no one to compare with, and any estimate there rests on extrapolation.
          Practice

          Design the comparison before looking at outcomes

          Stuart (2010) describes every study of a program’s effect as having two stages. In the design stage, the analyst uses only background characteristics and who participated to build comparable groups, by stratifying, matching, weighting, or dropping students with no counterparts, and then checks how comparable the groups are. Only after the design is fixed does the analysis stage compare outcomes. Imbens (2015) adds a step in between, testing the design on a variable measured before the program began, which the program cannot have changed. The Expert lesson returns to that test.

          The separation keeps the analyst from adjusting the design, even unintentionally, until the estimate looks right. Hiding outcomes while the comparison group is built removes that opportunity. A randomized experiment enforces the same separation automatically, because students are assigned before any outcome exists. An observational study has to impose it by discipline (Imbens & Rubin, 2015; Stuart, 2010).

          The target

          Effect for whom?

          Two averages are common. The average treatment effect on the treated (ATT) is the average effect for students who actually attended. The average treatment effect (ATE) is the average effect if every student in the population attended. When effects differ across students, the two can differ. A district deciding whether to keep offering a voluntary program usually cares about the ATT. A district considering making it mandatory cares about the ATE.

          The choice affects how the strata are averaged. For the ATT, each stratum’s comparison is weighted by the number of participants in it. For the ATE, it is weighted by the total number of students in it.

          Module 1 calls the quantity a study aims at its estimand. Fixing it before choosing a method is what makes the ATE or ATT choice a decision rather than a byproduct of the software’s defaults.

          Worked example · simulated data

          Stratifying by hand

          The code creates 3,000 students in three prior-achievement bands, about half receiving free or reduced-price lunch. Students in the low band and students receiving free lunch are much more likely to join. The program raises every joiner’s score by 3 points.

          The raw comparison gives −4.30 points, so the program appears harmful. The reason is visible in the data. 52 percent of joiners are in the low band, against 17 percent of non-joiners. Within each of the six strata, the comparisons range from 1.97 to 3.60, scattered around the truth by chance. Weighting them by the number of joiners in each stratum gives 2.82, close to the true 3.

          Run it yourself in R: code and output
          Evidence

          Why not just run a regression?

          Regression adjustment relies on the same assumption of no unmeasured confounding as matching, so it is not a different research design. The two differ in how they treat participants who have no close counterparts. Imbens (2015) shows the problem with LaLonde’s (1986) job-training data, as assembled by Dehejia and Wahba (1999). The 185 trainees earned on average 1.5 thousand dollars in 1975, while the 2,490 comparison men drawn from the Panel Study of Income Dynamics earned 19.1 thousand. The experiment that produced the trainees puts the program’s effect on 1978 earnings at roughly 2 thousand dollars.

          A linear regression of 1978 earnings on 1975 earnings, fitted to the comparison men, predicts that trainees would have earned 6.88 thousand dollars without the program. A regression on the logarithm of 1975 earnings, an equally defensible choice, predicts 2.81 thousand. The two predictions differ by 4.07 thousand, about twice the effect being studied, and the data cannot say which is right because almost no comparison men had earnings as low as the trainees’.

          Imbens (2015) also shows why. The linear regression is a weighted average of the comparison men’s outcomes, and one man who earned 156.7 thousand dollars in 1975 receives a weight of −12.05, although no trainee earned more than 25.14 thousand. Whether that man earned 100 or 300 thousand dollars in 1978 shifts the prediction for trainees by almost a thousand dollars. Matching on prior earnings would give him no weight at all. The practical lesson is that regression is dependable when the groups already overlap and fragile when they do not, and which case applies can be seen before any outcome is examined.

          Next step

          When there are many characteristics

          Stratification works with two or three categorical characteristics. With ten characteristics, some continuous, the number of strata explodes and most contain only participants or only non-participants. Rosenbaum and Rubin (1983) showed a way out. The propensity score, the probability of participating given all the measured characteristics, is enough to balance them. Students with the same propensity score have, on average, the same distribution of the characteristics that went into it, whether or not they participated. So instead of matching on ten characteristics, one can match on a single number. The Intermediate lesson develops this idea.

          Case · published study

          Holding children back in kindergarten

          Whether a struggling kindergartner should repeat the year is hard to settle by experiment, and a raw comparison is misleading, because retained children differ sharply from promoted children before the decision. Hong and Raudenbush (2006) used the Early Childhood Longitudinal Study, Kindergarten cohort, a nationally representative survey. Their analytic sample had 471 retained children and 10,255 promoted children in 1,080 schools that allowed retention, with many measured characteristics of children, families, teachers, and schools.

          Their design mimics a two-stage experiment. Schools were first classified as high-retention or low-retention and balanced on school characteristics with propensity scores. Within each type of school, children were then grouped into strata of similar estimated probability of retention, and retained and promoted children were compared within strata. This also let the authors ask a question a single-level analysis cannot, whether a school’s retention rate affects children who are not retained.

          In high-retention schools the raw reading gap between retained and promoted children was about 16 points on the test scale. Within propensity strata it shrank to 8.86 points (standard error 1.38). In low-retention schools the adjusted gap was 8.18 points (0.94), almost three-fifths of a standard deviation of the reading outcome. Math estimates were −5.56 and −4.79. The authors concluded that retained children would probably have learned more that year had they been promoted, and they found no benefit of a high retention rate for children at low risk of retention.

          They then asked how strong an unmeasured confounder would have to be to change the conclusion. For high-retention schools, it would need to be stronger than the strongest measured confounder other than the pretest. For low-retention schools, a confounder as strong as that measured one could make the reading estimate indistinguishable from zero. Reporting this difference, rather than a single verdict, is the habit this module recommends.

          Cautions

          Common misreadings

          • “We matched on everything, so the estimate is causal.”Matching balances what was measured. Unmeasured confounders remain.
          • “Matching is better than regression.”Both rest on the same assumption. Matching makes the comparison and overlap more visible, which is its main advantage.
          • “Adding more variables is always safer.”Variables affected by the program, or that affect only participation and not outcomes, can make estimates worse.
          • “The matched groups look the same, so the design is valid.”Balance on measured characteristics is necessary, not sufficient.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313A clear review of matching methods. Start here.
          • Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. https://doi.org/10.1080/00273171.2011.568786An accessible introduction to propensity scores for behavioral researchers.
          • Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41The paper that introduced the propensity score.
          • Hong, G., & Raudenbush, S. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101(475), 901–910. https://doi.org/10.1198/016214506000000447Kindergarten retention, a careful education application.
          • Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 86(1), 4–29. https://doi.org/10.1162/003465304323023651A review of estimators under no unmeasured confounding.
          • Imbens, G. W. (2015). Matching methods in practice: Three examples. Journal of Human Resources, 50(2), 373–419. https://doi.org/10.3368/jhr.50.2.373A step-by-step practical strategy with three worked data sets.
          • LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604–620.The job-training data that became the standard test bed for these methods.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 2 · Intermediate

          Propensity scores, matching, and balance.

          This lesson turns the idea of comparing like with like into a working analysis. You will estimate a propensity score, match participants to non-participants, check whether the matched groups are balanced, and check whether every participant has a credible comparison.

          Before thisFoundations: comparing like with likeConfounding, overlap, and stratification.
          After thisAdvanced: weighting and doubly robust estimationUsing every comparison student, weighted.
          By the end

          What you will be able to do

          1. State the balancing property of the propensity score and the assumptions it requires.
          2. Estimate a propensity score with logistic regression and choose its covariates.
          3. Carry out nearest-neighbor matching with replacement and a caliper.
          4. Assess balance with standardized mean differences rather than significance tests.
          5. Diagnose limited overlap and decide what to do about it.
          6. Read a published propensity score analysis for its design checks, trimming and robustness across estimators.
          Theory

          The propensity score

          Let \(X_i\) be student i's measured characteristics and \(D_i\) an indicator for participation. The propensity score is \(e(X_i) = \Pr(D_i = 1 \mid X_i)\). Rosenbaum and Rubin (1983) proved two results that make it useful.

          • Balancing. Conditional on the propensity score, the distribution of \(X\) is the same for participants and non-participants.
          • Sufficiency. If potential outcomes are independent of participation given \(X\), they are also independent given \(e(X)\).

          Together, these mean that a comparison of participants and non-participants with the same propensity score is as good as a comparison of students identical on all of \(X\), provided there is no unmeasured confounding and the scores lie strictly between 0 and 1.

          The true score is unknown and must be estimated, usually with logistic regression. A good propensity model is judged by the balance it produces in the matched groups, not by how accurately it predicts participation.

          Theory in practice

          What the theory guarantees, and what it does not

          Rosenbaum and Rubin (1983) defined a balancing score as any function of the covariates that leaves participants and non-participants with the same covariate distribution once it is held fixed. They showed that the covariates themselves are the finest balancing score and the propensity score is the coarsest, and that any balancing score finer than the propensity score also works. If participation is unrelated to potential outcomes given the covariates, then pair matching, subclassification, or regression on a balancing score all give unbiased estimates of the average effect.

          Their illustration came from a study of coronary bypass surgery versus drug therapy with 74 covariates. Drawing on Cochran’s (1968) finding that five subclasses remove at least 90 percent of the bias for many continuous distributions, they split patients at the quintiles of an estimated propensity score. Before subclassification, the median F statistic comparing the two groups on a covariate was 10.9. Within the subclasses it was 0.2, less imbalance than random assignment within subclasses would typically produce.

          Two limits follow from the same paper. The theorems hold for the true score, and an estimated score balances covariates only as far as its model is adequate, which is why balance has to be checked. And the guarantee concerns measured covariates only. Abadie and Imbens (2016) add a detail that matters for standard errors. Matching on an estimated rather than the true propensity score makes the estimate of the average treatment effect more precise in large samples, so ignoring the estimation step gives conservative standard errors for the ATE. For the ATT, however, the correction can go either way, so standard errors that ignore it may be too small or too large.

          Design

          Choosing covariates

          Covariates belong in the propensity model when they affect the outcome and may affect participation. Prior achievement, demographic characteristics, attendance, and school characteristics are typical. Three kinds of variable should be left out.

          • Variables measured after participation began, or that the program could affect. Adjusting for them blocks part of the effect or introduces bias.
          • Variables that affect only participation, such as distance to the program site. They add no bias reduction and push propensity scores toward 0 and 1, which worsens overlap and precision.
          • Variables chosen because they improve the result. As in experiments, choices made after seeing outcomes undermine credibility. Specify the propensity model while blind to outcomes where possible (Stuart, 2010).
          Method

          Nearest-neighbor matching

          For the ATT, each participant is paired with the non-participant whose estimated propensity score is closest. Several choices follow.

          • With or without replacement. Matching with replacement lets one non-participant serve as the match for several participants. It reduces bias when good comparisons are scarce, at some cost in precision, and the analysis must count how often each control is used.
          • One or several matches. Using several nearest neighbors lowers variance but uses less similar matches.
          • A caliper. A maximum allowed distance, commonly 0.2 standard deviations of the logit of the propensity score (Austin, 2011). Participants without a match within the caliper are dropped, which changes the estimand to the effect for the participants who could be matched.

          Abadie and Imbens (2006) derived the large-sample properties of matching estimators, including a bias that arises because matches are never exact, and gave variance estimators that account for controls being used more than once.

          Diagnostics

          Checking balance

          Thresholds differ across the literature, and it helps to know where they come from. Imbens and Rubin (2015) suggest that a normalized difference above 0.25 signals that regression adjustment will be sensitive to its specification (Imbens & Wooldridge, 2009). Stuart (2010) reports Rubin’s (2001) guidelines that standardized differences should be below 0.25 and variance ratios between 0.5 and 2 before regression adjustment can be trusted. The stricter 0.1 is common in medicine and is the safer target for a matched sample. Whatever the threshold, Imbens (2015) stresses that the normalized difference, unlike a t statistic, does not grow with the sample size, so it measures how hard the adjustment problem is rather than how confident we are that groups differ.

          The standardized mean difference for a covariate is the difference in means between groups divided by a pooled standard deviation. Values below 0.1 in absolute value are widely treated as adequate balance (Austin, 2011; Stuart, 2010). Compare it before and after matching.

          Significance tests are a poor guide here. After matching, the sample is smaller, so a t-test will find fewer significant differences even if balance has not improved. Balance is a property of the matched sample, not a hypothesis about a population. Check variance ratios and the balance of squares and interactions of key covariates too, because equal means can hide different distributions.

          If balance is poor, revise the propensity model, for example by adding a squared term or an interaction, and match again. Repeat until balance is acceptable, without looking at outcomes. This iteration is legitimate precisely because it does not use the outcome.

          -0.5-0.2500.250.5Prior scoreFamily incomeParent educationDays absentSchool sizeAgeStandardized mean differencebefore matchingafter matching0.1
          A balance plot. Simulated example. Each row is a covariate, with its standardized mean difference before matching (hollow) and after (filled). Matching pulls every covariate inside the 0.1 guideline shown by the dashed lines. A plot like this, not a table of t-tests, is the standard check.
          Worked example · simulated data

          Matching tutoring participants

          The first script simulates 2,000 students. Students with lower prior scores, those receiving free lunch, and those with more absences are more likely to join tutoring, and 797 join. The effect is 3 points, or 4 for students receiving free lunch, so the true average effect on joiners is 3.66. The outcome also depends on the square of the prior score, which a linear regression would miss.

          The estimated propensity coefficients, −0.93 on prior score, 0.61 on free lunch, and 0.56 on absences, are close to the values used to generate the data. Before matching, the standardized differences are −1.00, 0.57, and 0.71, which is severe imbalance, and the raw difference in means is −5.71. After nearest-neighbor matching with replacement, the differences fall to 0.04, −0.05, and 0.10. Absences sit right at the conventional threshold, which would prompt a careful analyst to refine the model.

          The matching estimate is 3.39, and a linear regression on the three covariates gives 3.31. Both are somewhat below the true 3.66. Matching used only 392 distinct controls, one of them 13 times, because few non-participants look like the participants with the highest scores. Only 2 matches fall outside a caliper of 0.2 standard deviations.

          Run it yourself in R: code and output
          Diagnostics

          Checking overlap

          The second script tabulates the estimated propensity scores for both groups. Below 0.1 there are 214 non-participants and only 12 participants. Above 0.9 there are 31 participants and only 5 non-participants. Those 31 participants are matched over and over to the same few controls, and any estimate for them leans heavily on those few students.

          Crump, Hotz, Imbens, and Mitnik (2009) proposed a simple rule of thumb, keeping students with estimated scores between 0.1 and 0.9. Here that keeps 1,738 of 2,000 students. Trimming improves precision and credibility, but it changes the question. The estimate then describes students in the region of overlap, and the report should say who was dropped.

          The trimming rule adapts to the data. In the lottery analysis described below, the optimal cut-off removed 173 of 496 players. Applied to the experimental part of LaLonde’s data, where randomization had already produced comparable groups, the same procedure set the cut-off at 0.1299 and dropped only 7 of 445 men (Imbens, 2015). A trimming rule that removes many units is itself a finding about the data.

          Alternatives

          Other matching methods

          • Exact matching on key variables.Match within school or grade, then on the propensity score within those groups.
          • Mahalanobis distance matching.Matches on the covariates directly, and works well with a few continuous covariates.
          • Subclassification.Divide the propensity score into five or more strata and compare within each, as in the Foundations lesson.
          • Full and optimal matching.Use every student, forming matched sets that minimize total distance.
          Case · published analysis

          A full analysis from start to finish

          Imbens (2015) works through a complete analysis of 496 Massachusetts lottery players, 237 who won large prizes paid over 20 years and 259 who won small one-time prizes, to estimate how a large prize changes labor earnings. Lottery numbers are random, but who bought more tickets and who answered the survey were not, so the groups differ. Winners bought 4.57 tickets a week on average and losers 2.19, a normalized difference of 0.90. Age and years of schooling also differed by more than 0.30.

          The design stage estimated a propensity score by a stepwise logistic regression that kept 4 chosen covariates, 4 more linear terms, and 10 squared or interaction terms. The variance-based trimming rule of Crump et al. (2009) then set the cut-off at 0.0891, dropping 86 players with lower scores and 87 with scores above 0.9109. The trimmed sample of 323 was better balanced, with the normalized difference in tickets falling from 0.90 to 0.51.

          Before estimating the effect on later earnings, Imbens tested the design by estimating the effect of winning on earnings in the years before winning, which a lottery cannot cause. The three estimates ranged from −1,160 to −390 dollars, none significant at the 10 percent level. Only then did he estimate the effect on average earnings over the six years after winning. Estimators using five propensity score blocks gave −5,070 to −5,740 dollars a year, with standard errors of 1,400 to 2,000, and the preferred estimate was about −5,400 (standard error about 1,400). With a simpler linear propensity model, the same six estimators spread over a range more than twice as wide. The argument rests on agreement across reasonable estimators, not on any single estimate.

          Reporting

          What to report

          1. The covariates in the propensity model and why each was included.
          2. The matching method, ratio, replacement, and caliper.
          3. A balance table with standardized differences before and after matching.
          4. The overlap plot or table and the number of students dropped.
          5. The estimand, ATT or ATE, and for whom it holds after any trimming.
          6. A standard error that accounts for the matching.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41The balancing and sufficiency results.
          • Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313Matching choices and diagnostics.
          • Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. https://doi.org/10.1080/00273171.2011.568786Calipers, balance thresholds, and practical advice.
          • Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74(1), 235–267. https://doi.org/10.1111/j.1468-0262.2006.00655.xBias and variance of matching estimators.
          • Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1), 187–199. https://doi.org/10.1093/biomet/asn055The 0.1 to 0.9 trimming rule.
          • Imbens, G. W. (2015). Matching methods in practice: Three examples. Journal of Human Resources, 50(2), 373–419. https://doi.org/10.3368/jhr.50.2.373Normalized differences, trimming, blocking, and a full lottery example.
          • Abadie, A., & Imbens, G. W. (2016). Matching on the estimated propensity score. Econometrica, 84(2), 781–807. https://doi.org/10.3982/ECTA11293Why estimating the propensity score changes the standard error.
          • Imbens, G. W., & Wooldridge, J. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47(1), 5–86. https://doi.org/10.1257/jel.47.1.5A broad review of estimation under unconfoundedness.
          • Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24(2), 295–313. https://doi.org/10.2307/2528036Why five subclasses remove most of the bias from a covariate.
          • Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2(3–4), 169–188. https://doi.org/10.1023/A:1020363010465Balance guidelines for trusting regression adjustment.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 2 · Advanced

          Weighting and doubly robust estimation.

          Matching leaves out comparison students who are not anyone’s nearest neighbor. Weighting keeps them and gives every student a weight so that the groups come to resemble each other. This lesson covers inverse probability weighting, weights chosen to balance covariates directly, overlap weights, and augmented estimators that combine a weighting model with an outcome model and stay consistent if either one is right.

          Before thisIntermediate: propensity scores and matchingBalance and overlap diagnostics.
          After thisExpert: sensitivity to unobserved confoundingWhat if a confounder was not measured?
          By the end

          What you will be able to do

          1. Explain why nearest-neighbor matching is biased in large samples and how bias correction addresses it.
          2. State inverse probability weights for the ATE and the ATT.
          3. Recognize extreme weights and apply trimming or truncation.
          4. Explain how entropy balancing and stable balancing weights choose weights that balance covariates directly.
          5. Compute overlap weights and say which population they target.
          6. Check balance and the weights themselves in the weighted sample, beyond means.
          7. Write the augmented inverse probability weighted estimator and explain double robustness.
          8. Choose between regression, weighting, and doubly robust estimators in a given study.
          Theory

          How good is nearest-neighbor matching?

          Matching with a fixed number of matches per student is easy to explain. Abadie and Imbens (2006) derived its large-sample properties. Because matches are never exact, each matched pair carries a small bias from the remaining covariate difference. That bias shrinks as the sample grows, but only at a rate of \(N^{-1/k}\), where k is the number of continuous covariates. With one continuous covariate the bias disappears faster than the standard error, and with two it is of the same order. With three or more it disappears more slowly, so in large samples the bias outweighs the sampling error and confidence intervals centered on the estimate are off target. Even when matching is consistent, it does not reach the efficiency bound (Abadie & Imbens, 2006), and the usual bootstrap does not give valid standard errors (Imbens, 2015).

          The remedy is bias-corrected matching. Regress the matched comparison students’ outcomes on their covariates, and use the fitted regression to adjust each matched outcome for its remaining covariate difference. Because the matched groups already overlap, this regression mostly interpolates rather than extrapolates. Abadie and Imbens (2011) showed that the corrected estimator is consistent at the usual square-root-of-N rate whatever the number of continuous covariates, and that its asymptotic variance equals that of simple matching.

          Evidence

          What bias correction buys

          Abadie and Imbens (2011) applied both estimators to the 185 job-training trainees and the Panel Study of Income Dynamics comparison group of 2,490 men in LaLonde’s data, where the experiment gives a benchmark of 1.80 thousand dollars (95 percent interval 0.49 to 3.10). Simple matching drifts away from the benchmark as more matches per trainee are used, because the added matches are worse. The bias-corrected version barely moves.

          Matches per traineeSimple matchingBias-corrected matching
          12.07 (1.13)2.42 (1.13)
          41.62 (0.91)2.51 (0.90)
          160.47 (0.85)2.48 (0.83)
          64−0.11 (0.75)2.26 (0.71)

          Earnings in thousands of 1978 dollars, standard errors in parentheses. The experimental estimate is 1.80 (0.67). Source is Abadie and Imbens (2011), Table 2.

          A simulation built to mimic these data tells the same story. With 64 matches per trainee, nominal 95 percent intervals around simple matching covered the true effect in 9 percent of samples, while bias-corrected intervals covered it in 92 percent. In the actual data, a linear regression on the full sample gave 0.84 and a quadratic one 3.26, showing again how sensitive regression adjustment without matching is when the groups barely overlap.

          Method

          Inverse probability weighting

          A participant with propensity score 0.2 belongs to a kind of student who usually does not participate. Only one in five students with that score joined, so each participant stands in for about five such students and receives a weight of 5. A non-participant with the same score receives a weight of 1/(1 − 0.2) = 1.25. Weighting each participant by \(1/e(X)\) and each non-participant by \(1/(1 - e(X))\) creates a pseudo-population in which participation is unrelated to \(X\). The weighted difference in means then estimates the ATE.

          \[ \hat\tau_{ATE} = \frac{\sum D_i Y_i / \hat e_i}{\sum D_i / \hat e_i} - \frac{\sum (1-D_i) Y_i / (1-\hat e_i)}{\sum (1-D_i)/(1-\hat e_i)}. \]

          Dividing by the sum of the weights in each group, as above, is called normalizing. It is usually more stable than the unnormalized version. For the ATT, participants get weight 1 and non-participants get \(\hat e/(1 - \hat e)\), which reshapes the control group to look like the participants. Hirano, Imbens, and Ridder (2003) showed that weighting by a flexibly estimated propensity score reaches the efficiency bound, and in large samples is more precise than weighting by the true score.

          Problem

          Extreme weights

          A participant with an estimated score of 0.01 receives a weight of 100. If that student’s outcome is unusual, it can move the whole estimate. Extreme weights are the weighting version of the overlap problem. They signal that some students have almost no comparable counterparts.

          Common responses include trimming the sample to scores between 0.1 and 0.9 (Crump et al., 2009), which changes the estimand to the effect for the students who remain, and truncating weights at a maximum, which introduces some bias in exchange for lower variance. Neither is a substitute for looking at the distribution of weights and reporting it.

          Method

          Entropy balancing

          Propensity weights balance covariates only to the extent that the propensity model is adequate, and that has to be checked afterwards. Hainmueller (2012) sets the balance conditions first. To estimate the ATT, entropy balancing chooses weights for the comparison students that exactly match the participants’ covariate means, and variances or higher moments if requested, while staying as close as possible to equal weights. The solution has the form \(w_i \propto \exp(\lambda' X_i)\), with \(\lambda\) found by a few Newton steps.

          Balance on the targeted moments is guaranteed by construction, so the balance table becomes a check that the algorithm converged rather than a diagnostic of the model. Balance on moments that were not targeted, such as squares or interactions, still needs checking. When overlap is poor, entropy balancing can fail to converge or produce very uneven weights, which is useful information.

          Recent methods

          Weights that target balance directly

          Inverse probability weights balance covariates only on average over repeated samples, and only if the propensity model is right. In a given data set they may leave imbalances and contain a few huge weights. Zubizarreta (2015) proposed stable balancing weights, the weights with the smallest variance among all non-negative weights that bring every chosen covariate mean within a tolerance δ of its target. The researcher specifies what balance is needed, not a model. Squares, interactions, or quantile indicators can be added as further constraints to balance whole distributions. Because the problem is a convex optimization, the dual variables report how much tightening each constraint would increase the variance of the weights.

          The variance of the weights matters because, roughly, a weighted mean’s variance is inflated by a factor of one plus the squared coefficient of variation of the weights. In the simulation design of Kang and Schafer (2007), with propensity and outcome models fitted to the wrong transformations of the covariates, logistic weights had a mean coefficient of variation of 9.29. The root mean squared error of the Horvitz-Thompson estimator was 235 and of a doubly robust estimator 118. Stable balancing weights with δ of 0.01 standard deviations or less had a coefficient of variation near 0.53, and both errors fell to between 1.7 and 1.8 (Zubizarreta, 2015).

          Chattopadhyay, Hase, and Zubizarreta (2020) place these methods, together with entropy balancing (Hainmueller, 2012), in a family they call the balancing approach, as opposed to the modeling approach of fitting a propensity model and inverting it. The two are closer than they look, since balancing weights implicitly estimate a propensity score by minimizing a loss that imposes balance while penalizing the dispersion of the weights. Their review recommends three practices for any weights. Measure balance of each group against the target population, not only against each other. Report the effective sample size, \((\sum w_i)^2 / \sum w_i^2\) (Kish’s approximation), to show how much information survives weighting. And prefer non-negative weights, so that weighted means stay within the range of the observed outcomes. Unless the assignment model is known, they recommend the balancing approach.

          Weighting

          Three weights, three targets

          Li, Morgan, and Zaslavsky (2018) show that the common propensity score weights belong to one family. Each treated unit receives \(h/e\) and each comparison unit \(h/(1-e)\), where \(e\) is the propensity score and \(h\) is a tilting function that chooses the target population. Setting \(h = 1\) gives weights for the ATE, \(h = e\) gives weights for the ATT, and \(h = e(1-e)\) gives overlap weights. Under overlap weights a treated student receives \(1-e\) and a comparison student receives \(e\). Students whose scores are near 0.5, who could plausibly have gone either way, count the most, and students whose scores are near 0 or 1 count the least.

          \[ w_1(x) = \frac{h(x)}{e(x)}, \qquad w_0(x) = \frac{h(x)}{1 - e(x)}, \qquad h(x) = e(x)\,\bigl(1 - e(x)\bigr) \;\Rightarrow\; w_1 = 1 - e(x), \; w_0 = e(x). \]

          Three properties make this attractive. The weights are bounded between 0 and 1, so no single student can dominate. Among weights in this family they have the smallest asymptotic variance when the outcome has constant variance. And when the propensity score comes from a logistic regression fitted by maximum likelihood, the weighted means of every covariate in that model are exactly equal in the two groups (Li et al., 2018).

          The cost is a change of question. The overlap population is the group of students with a real chance of either treatment, and it is defined by the weights, not by the policy question. If the district must decide for everyone, overlap weights answer a different question, namely the effect for students on whom the data are informative. Often that is the only effect the data can support, and the report should say so.

          The second script in the code panel under the worked example compares the three weights on 2,000 simulated students. Tutoring is offered mostly to students with high scores on two baseline measures, so overlap is weak, and the effect of tutoring is \(2 + x_1\), so each target has its own true value.

          In one sample, the largest standardized difference between the groups is 1.16 before weighting. Weights for the ATE reduce it to 0.07 and weights for the ATT to 0.12, which exceeds the 0.1 guideline, while overlap weights make it 0.00. The effective sample size, summed over the two groups, is 969 for ATE weights, 1,221 for ATT weights, and 1,307 for overlap weights. Over 200 samples the ATE estimate averages 2.01 for a target of 2.00 with a standard deviation of 0.73. The overlap estimate averages 1.99 for the same target of 2.00 with a standard deviation of 0.27. The ATT estimate averages 2.48 for a target of 2.52. Here the overlap population happens to have the same average effect as the whole sample, so the gain in precision comes at no cost in this example. With another effect pattern the targets would differ.

          Diagnostics

          Check balance in the weighted sample

          Weighting is only as good as the balance it produces, and the balance has to be checked in the weighted sample. Austin and Stuart (2015) reviewed studies that used inverse probability of treatment weighting and found that, in the most recent year, a majority did not formally examine whether the weighting had balanced the covariates. The basic check is the weighted standardized difference of each covariate, with the same 0.1 guideline used for matching.

          Means are not the only thing to compare. Austin (2009) proposed diagnostics that go beyond means, because two groups can have equal means and still differ in spread or shape. Compare the ratio of the variances of each continuous covariate, which should be close to 1, and look at squares and interactions of the covariates most strongly related to the outcome. Comparing the distribution of the propensity score between groups is not a substitute, because it does not show whether the covariates themselves are balanced (Austin & Stuart, 2015).

          The weights need their own check. Stabilized weights should average close to 1, and very large values point to students with propensity scores near 0 or 1, where the data hold little information (Austin & Stuart, 2015). Truncating the weights, for example at their 1st and 99th percentiles, is one response. Overlap weights are another, with the change of target described above. Report the effective sample size so that readers can see how much information remains after weighting.

          Method

          Doubly robust estimation

          Regression adjustment relies on a correct model for the outcome. Weighting relies on a correct model for participation. The augmented inverse probability weighted (AIPW) estimator uses both. Fit outcome models \(\hat\mu_1(X)\) and \(\hat\mu_0(X)\) for participants and non-participants, then correct their predictions with weighted residuals,

          \[ \hat\tau_{AIPW} = \frac{1}{n}\sum_i \Big[\hat\mu_1(X_i) - \hat\mu_0(X_i) + \frac{D_i (Y_i - \hat\mu_1(X_i))}{\hat e_i} - \frac{(1-D_i)(Y_i - \hat\mu_0(X_i))}{1 - \hat e_i}\Big]. \]

          If the outcome models are right, the residual terms average to zero whatever the propensity scores are. If the propensity model is right, the weighted residuals correct whatever error the outcome models make. The estimator is consistent if either model is correct, which is the property called double robustness (see Glynn & Quinn, 2010, for an introduction). When both are right, it is also efficient. The same structure underlies the doubly robust difference-in-differences estimators in the DiD module.

          Evidence

          Two chances to get it right

          Bang and Robins (2005) describe double robustness as giving the analyst two chances, instead of one, to make a valid inference. They also show that the estimator can be computed with ordinary regression software. Fit the propensity score, then add one extra covariate to the outcome regression, the inverse of each person’s estimated probability of receiving the treatment they actually received. Averaging the predictions of this augmented regression under treatment and under control gives the doubly robust estimate. Their simulation shows the pattern.

          EstimatorPropensity modelOutcome modelMean biasVariance
          Inverse probability weightingCorrectNot used−0.010.21
          Inverse probability weightingWrongNot used0.860.15
          Outcome regressionNot usedCorrect0.000.07
          Outcome regressionNot usedWrong−1.560.07
          Doubly robustCorrectCorrect0.000.09
          Doubly robustCorrectWrong−0.090.28
          Doubly robustWrongCorrect0.000.08
          Doubly robustWrongWrong0.920.15

          The true effect is 2. Samples of 500, 1,000 replications. Source is Bang and Robins (2005), Table 2, lower panel.

          When either model was right, the doubly robust estimate was close to the truth, and when the outcome model was right it was nearly as precise as outcome regression alone. Its variance was about three to four times as large when only the propensity model was right, and when both models were wrong it was about as badly biased as weighting with the wrong propensity model (0.92 against 0.86). The authors argue that the small loss of efficiency when both models are correct is worth the added protection, which supports routine use. In practice no model is exactly correct, but if either is nearly correct, the bias of the doubly robust estimate should be small. Comparing the weighting, regression, and doubly robust estimates is a sensible informal check, since large disagreements signal that at least one model is badly wrong.

          Worked example · simulated data

          Weighting the tutoring data

          The code uses the tutoring design from the Intermediate lesson with a new sample of 2,000 students, so the true ATE is 3.49 and the true ATT is 3.62. Weighting for the ATT gives 3.52, and entropy balancing gives 3.53, with the control means matching the joiners’ means exactly. AIPW with an outcome model that includes the squared prior score gives 3.34 for the ATE.

          Weighting for the ATE gives 6.61, far from the truth. The reason is a single participant whose estimated propensity is very low, so that student receives a weight of 126.8. Dropping that single student moves the estimate to 3.27. The ATE asks what would happen if every student joined, and for students who almost never join, the data contain very little information. Trimming to scores between 0.1 and 0.9 keeps 1,739 students and gives 3.09, against a true effect of 3.52 for those kept.

          The second script compares weights for the ATE, the ATT, and overlap on a design with weak overlap. The section on three weights and three targets discusses it.

          Run it yourself in R: code and output
          Monte Carlo · simulated data

          Double robustness in action

          The second half of the code repeats the analysis on 200 new samples. A wrong propensity model omits absences, and a wrong outcome model omits absences and the squared prior score.

          EstimatorBiasSD
          Regression, linear in the covariates0.350.25
          IPW, propensity model right−0.020.82
          IPW, propensity model wrong−1.020.61
          AIPW, propensity right, outcome wrong0.000.53
          AIPW, propensity wrong, outcome right0.020.30
          AIPW, both wrong−0.960.42

          The pattern follows the theory. AIPW is nearly unbiased when either model is right, and badly biased when both are wrong. IPW with the right model is unbiased but the noisiest estimator, with a standard deviation of 0.82. The linear regression has the smallest variance but a bias of 0.35 from its missing squared term. Double robustness offers two chances to get the model right, not a guarantee.

          Practice

          Choosing an estimator

          • Good overlap, many covariates.AIPW with flexible models for both parts is a strong default. With machine-learning models, use cross-fitting, which predicts each student’s propensity score and outcome from models fit on other parts of the sample. The Intermediate level of Machine learning for causal inference and the Expert level of Difference-in-differences develop it.
          • ATT with a large pool of controls.Matching or entropy balancing make the comparison transparent and easy to explain.
          • Poor overlap.Trim, use overlap weights, or redefine the target population. No estimator can create information about students with no counterparts.
          • Always.Report the distribution of weights, the effective sample size, and the balance achieved, and compare several estimators. Large disagreements point to model dependence.
          Inference

          Standard errors

          Weighted estimators need standard errors that reflect the estimation of the weights. Robust sandwich standard errors from a weighted regression treat the weights as fixed and are usually conservative for IPW estimates of the ATE. The bootstrap, re-estimating the propensity model in each resample, is a practical alternative for weighting estimators. For AIPW, the variance of the bracketed term in the formula above, divided by n, gives a valid standard error when both models are correct or are estimated flexibly with cross-fitting. When students are nested in schools, resample or cluster by school.

          For matching estimators the bootstrap is not valid in general, because the estimators are not smooth functions of the data (Imbens, 2015). Abadie and Imbens (2006) give an analytic variance estimator that uses within-group matching to estimate each unit’s conditional variance. When matching is on an estimated propensity score, the adjustment of Abadie and Imbens (2016) accounts for the first step. For the ATE, ignoring it errs on the conservative side.

          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Hirano, K., Imbens, G. W., & Ridder, G. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4), 1161–1189. https://doi.org/10.1111/1468-0262.00442Efficiency of weighting with an estimated propensity score.
          • Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25–46. https://doi.org/10.1093/pan/mpr025Entropy balancing.
          • Glynn, A. N., & Quinn, K. M. (2010). An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18(1), 36–56. https://doi.org/10.1093/pan/mpp036A readable introduction to AIPW and double robustness.
          • Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1), 187–199. https://doi.org/10.1093/biomet/asn055Limited overlap and trimming.
          • Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 86(1), 4–29. https://doi.org/10.1162/003465304323023651A review comparing regression, matching, and weighting.
          • Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74(1), 235–267. https://doi.org/10.1111/j.1468-0262.2006.00655.xBias and variance of matching with a fixed number of matches.
          • Abadie, A., & Imbens, G. W. (2011). Bias-corrected matching estimators for average treatment effects. Journal of Business & Economic Statistics, 29(1), 1–11. https://doi.org/10.1198/jbes.2009.07333Bias-corrected matching, with the LaLonde application.
          • Zubizarreta, J. R. (2015). Stable weights that balance covariates for estimation with incomplete outcome data. Journal of the American Statistical Association, 110(511), 910–922. https://doi.org/10.1080/01621459.2015.1023805Stable balancing weights.
          • Chattopadhyay, A., Hase, C. H., & Zubizarreta, J. R. (2020). Balancing vs modeling approaches to weighting in practice. Statistics in Medicine, 39(24), 3227–3254. https://doi.org/10.1002/sim.8659A practical review comparing balancing and modeling approaches.
          • Kang, J. D. Y., & Schafer, J. L. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22(4), 523–539. https://doi.org/10.1214/07-STS227The simulation design used to stress-test weighting estimators.
          • Bang, H., & Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4), 962–972. https://doi.org/10.1111/j.1541-0420.2005.00377.xDoubly robust estimation as an augmented regression, with simulations under each pattern of model failure.
          • Li, F., Morgan, K. L., & Zaslavsky, A. M. (2018). Balancing covariates via propensity score weighting. Journal of the American Statistical Association, 113(521), 390–400. https://doi.org/10.1080/01621459.2016.1260466The family of balancing weights, overlap weights, and the population they target.
          • Austin, P. C., & Stuart, E. A. (2015). Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661–3679. https://doi.org/10.1002/sim.6607Practical advice on weighting, including balance checks and stabilized weights.
          • Austin, P. C. (2009). Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Statistics in Medicine, 28(25), 3083–3107. https://doi.org/10.1002/sim.3697Balance diagnostics beyond means, including variance ratios and higher moments.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 2 · Expert

          Sensitivity to unobserved confounding.

          Every method in this module assumes that the measured characteristics capture everything that drives both participation and outcomes. That assumption cannot be tested. This lesson shows how to ask instead how strong an unmeasured confounder would have to be to change the conclusion, and what studies that compared observational estimates with experiments suggest about when the assumption holds.

          Before thisAdvanced: weighting and doubly robust estimationEstimators under no unmeasured confounding.
          Next moduleInstrumental variablesA design that tolerates unmeasured confounding. It explains noncompliance and compliers from the beginning, so the Expert level of Randomized experiments is useful background but not required.
          By the end

          What you will be able to do

          1. Write the omitted variable bias of a regression estimate in terms of partial R² values.
          2. Compute and interpret the robustness value of Cinelli and Hazlett (2020).
          3. Benchmark a hypothetical confounder against observed covariates, and recognize the limits of doing so.
          4. Use a pre-program outcome as a pseudo-outcome to probe a design, and say what the result can and cannot show.
          5. Summarize what within-study comparisons suggest about when adjustment for observed covariates works.
          6. Report a sensitivity analysis that a skeptical reader can evaluate.
          Theory

          Omitted variable bias, rescaled

          Suppose the correct regression includes an unmeasured confounder Z alongside the treatment and the observed covariates, but we estimate the regression without it. The classic formula says the bias equals the effect of Z on the outcome times the imbalance of Z across treatment groups. Both quantities are in units that are hard to reason about for something unmeasured.

          Cinelli and Hazlett (2020) rewrite the size of the bias in terms of two partial R² values. \(R^2_{Y\sim Z|D,X}\) is the share of the outcome’s residual variance that Z explains, and \(R^2_{D\sim Z|X}\) is the share of the treatment’s residual variance that Z explains,

          \[ |\widehat{\text{bias}}| = \text{se}(\hat\tau)\sqrt{\text{df}}\,\sqrt{\frac{R^2_{Y\sim Z|D,X}\;R^2_{D\sim Z|X}}{1 - R^2_{D\sim Z|X}}}. \]

          Every quantity except the two partial R² values comes from the regression output. A researcher can then ask how much variance a confounder would need to explain, a question that can be answered by comparison with the variance that measured characteristics explain.

          Summary measure

          The robustness value

          The robustness value is the smallest strength of association, equal on both sides, that a confounder would need to have with the treatment and with the outcome to reduce the estimate to zero. With \(f = t/\sqrt{\text{df}}\), where t is the estimate’s t statistic,

          \[ RV = \tfrac{1}{2}\left(\sqrt{f^4 + 4f^2} - f^2\right). \]

          A robustness value of 0.10 means that a confounder explaining 10 percent of the residual variance of both treatment and outcome would be enough to eliminate the estimate, while a confounder weaker than that on both sides would not. A second version replaces f with a value reduced by the critical t value, giving the strength needed to make the estimate no longer statistically significant (Cinelli & Hazlett, 2020).

          Interpretation

          Benchmarking against what was measured

          A robustness value has meaning only relative to plausible confounders. Cinelli and Hazlett (2020) propose benchmarking. Compute how much the strongest observed covariates explain, and ask whether an unmeasured confounder as strong as, or several times as strong as, a key covariate would change the conclusion. In education, prior achievement is often among the strongest predictors of outcomes, so a confounder as strong as prior achievement is a demanding benchmark.

          Benchmarking has a known weakness. It is reassuring only if the observed covariates are at least as strong as the unobserved ones. If an important confounder like motivation is plausibly stronger than anything measured, benchmarks against weak covariates give false comfort, as the worked example shows.

          Diagnostics

          Testing the design on an outcome the program cannot change

          Unmeasured confounding cannot be tested directly, but Imbens (2015) describes a way to probe the design. Pick a variable measured before the program, most usefully an earlier value of the outcome, treat it as if it were the outcome, and run the whole analysis, adjusting for the remaining covariates. The true effect on this pseudo-outcome is zero. An estimate far from zero makes the assumption less plausible, and an estimate near zero makes it more plausible, although neither settles it.

          The test passed in two of Imbens’s three examples. For the lottery players, effects on pre-lottery earnings ranged from −1,160 to −390 dollars and none was significant. In the experimental part of LaLonde’s data they ranged from −80 to 220 dollars. With the Current Population Survey comparison group, however, the estimated effect on 1975 earnings was about −1.2 thousand dollars (standard errors about 0.3), and on average 1974 and 1975 earnings, adjusting only for the six covariates other than earnings, about −6.1 to −6.4 thousand dollars (standard errors under 0.7). The final estimates for that comparison were close to the experimental benchmark, but without the experiment an analyst would have had no basis for trusting them. More years of prior earnings would have been needed to make the case.

          The logic rests on an extra assumption, that the process linking past outcomes to participation is stable over time. The relevance of the test depends on how plausible that assumption is. Several lagged outcomes make the test more informative, since a variable closely related to the outcome gives it more power (Imbens, 2015).

          Worked example · simulated data

          A confounder nobody measured

          The code adds unobserved motivation to the tutoring design from the previous lessons, in a new sample of 2,000 students. Motivated students are more likely to join and score higher. A regression on the treatment, the observed covariates, and the squared prior score gives 6.00 (standard error 0.30), against a true effect on joiners of about 3.62. The estimate looks extremely precise, with t = 20.18, and that precision is the problem. A small standard error offers no protection against bias from an unmeasured confounder.

          The robustness value is 0.361 for reducing the estimate to zero and 0.333 for losing statistical significance. A confounder would need to explain about 36 percent of the residual variance of both participation and scores to eliminate the estimate. Benchmarking against free lunch status looks reassuring, because free lunch explains only 4.3 percent of residual outcome variance and 0.6 percent of residual participation variance, implying a bias of just 0.22.

          In the simulation we can measure motivation. It explains 24.7 percent of residual outcome variance and 9.6 percent of residual participation variance, well beyond the free lunch benchmark though below the robustness value. So it does not reduce the estimate to zero, but it accounts for most of the gap. Adjusting for it gives 3.84, so the bias it caused was about 2.16, roughly ten times the 0.22 implied by the free lunch benchmark. The robustness value described correctly what it would take to eliminate the estimate, but the benchmark against free lunch status understated how much bias was plausible, because free lunch is a much weaker predictor than motivation here.

          Run it yourself in R: code and output
          Evidence

          What within-study comparisons show

          A within-study comparison asks whether an observational method can reproduce an experimental benchmark using the same treated group. LaLonde (1986) compared estimates of a job-training program’s effect from a randomized experiment with estimates that replaced the experimental controls with survey respondents, and found that many standard non-experimental methods were far from the experimental answer. Dehejia and Wahba (1999) revisited a subsample of the same data and found that propensity score methods came much closer, which started a long debate about how much depended on the sample and specification.

          Shadish, Clark, and Steiner (2008) designed a study for this question. They randomly assigned undergraduate participants either to be randomized between mathematics and vocabulary training or to choose between them. Adjusting the self-selected group with a rich set of pretests and measures of preferences and motivation removed most of the bias in the observational estimates. Their result suggests that the covariates matter more than the choice of estimator, and that measures of the specific reasons people select into a program are especially valuable.

          Later reanalyses refine that debate. Imbens (2015) and Abadie and Imbens (2011) obtain estimates close to the experimental benchmark from LaLonde’s non-experimental comparisons, but only with matching or weighting plus a correction for remaining covariate differences, and in Imbens (2015) after trimming to the region of overlap. Regression on the full sample can differ widely with its specification. In the PSID data a linear regression gave 0.84 and a quadratic one 3.26, against an experimental estimate of 1.80. The within-study evidence therefore supports careful design, not any one estimator.

          Practice

          Designing for credibility

          • Understand the selection process.Interview the people who decide participation and measure what they use.
          • Measure pretests of the outcome.Prior values of the outcome often absorb much of what makes participants different.
          • Find a local comparison group.Comparisons within the same schools or districts share much of their unmeasured context.
          • Combine designs.Matching combined with difference-in-differences can remove unmeasured differences that stay constant over time, provided outcomes would have trended in parallel.
          • Report sensitivity.State the robustness value, the benchmarks used, and why the unmeasured confounders you worry about are or are not stronger than them.
          Reporting

          What to report

          1. The estimate under no unmeasured confounding, with its standard error.
          2. The partial R² of treatment with the outcome and the robustness values.
          3. Benchmarks against the strongest observed covariates, including multiples of their strength.
          4. A plain-language statement of which unmeasured confounders are plausible and how strong they might be.
          5. The result of any pseudo-outcome test, with the covariates used.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Cinelli, C., & Hazlett, C. (2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1), 39–67. https://doi.org/10.1111/rssb.12348Partial R² sensitivity analysis and the robustness value.
          • Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103(484), 1334–1344. https://doi.org/10.1198/016214508000000733A randomized test of whether observational adjustment works.
          • LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604–620.The original within-study comparison.
          • Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062. https://doi.org/10.1080/01621459.1999.10473858Propensity scores applied to LaLonde’s data.
          • Hong, G., & Raudenbush, S. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101(475), 901–910. https://doi.org/10.1198/016214506000000447A careful observational study in education that includes a sensitivity analysis for an unmeasured confounder.
          • Imbens, G. W. (2015). Matching methods in practice: Three examples. Journal of Human Resources, 50(2), 373–419. https://doi.org/10.3368/jhr.50.2.373Pseudo-outcome tests of unconfoundedness in three data sets.
          • Abadie, A., & Imbens, G. W. (2011). Bias-corrected matching estimators for average treatment effects. Journal of Business & Economic Statistics, 29(1), 1–11. https://doi.org/10.1198/jbes.2009.07333Bias-corrected matching on the LaLonde within-study comparison.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 3 · Foundations

          Using a nudge to learn about a treatment.

          Sometimes we cannot assign a treatment, but something random pushes some people toward it. Examples include a school admissions lottery, a randomly timed reminder, and a rule that applies to some birthdays and not others. Instrumental variables use that push to estimate the treatment’s effect, even when the people who take the treatment differ in ways nobody measured.

          Before thisRandomized experiments, FoundationsRandom assignment. Noncompliance and compliers are explained from scratch in this lesson.
          After thisIntermediate: two-stage least squaresCovariates, continuous treatments, and standard errors.
          By the end

          What you will be able to do

          1. Explain what an instrument is with a school example.
          2. State the three conditions an instrument must meet.
          3. Compute the Wald estimate from two differences.
          4. Explain why the estimate describes compliers rather than everyone.
          5. Spot common threats to an instrument in published studies.
          The problem

          When choice and outcomes share a cause

          Does attending a charter school raise test scores? Comparing charter students with other students mixes the school’s effect with the reasons families chose it. Families who apply to charter schools may be more engaged in their children’s education, and engagement raises scores whatever school a child attends. Matching on measured characteristics helps only if engagement is measured, which it rarely is.

          Many popular charter schools have more applicants than seats and admit students by lottery. Winning does not guarantee attendance, and losing does not always prevent it. But the lottery itself is random. That random push toward attendance is the raw material of an instrumental variables analysis.

          Intuition

          When an offer is not the same as treatment

          Suppose a district randomly offers free after-school tutoring to half of its struggling readers. Because the offer is random, offered and non-offered students are alike on average, in motivation as well as in measured characteristics. But an offer is not a session. Some offered students never attend, and some students without an offer find a seat anyway, perhaps through a sibling. When the treatment students actually receive differs from the one they were assigned, there is noncompliance.

          Say 70 percent of offered students attend tutoring and 20 percent of the others do, and offered students score 1.5 points higher on average. Comparing scores by offer gives the intention-to-treat (ITT) effect. It is still a fair comparison, because the offer was random. But the 1.5 points is the effect of the offer, diluted by the many students whose attendance the offer did not change.

          Comparing students who attended with those who did not would not remove the dilution. Students who choose to attend may be more motivated, so that comparison brings back the bias the random offer removed. Instead, divide the ITT by the share of students whose attendance the offer changed, 70 percent − 20 percent = 50 percentage points. The result, 1.5 / 0.50 = 3 points, is the average effect of tutoring for students who attend because they were offered it. These students are called compliers.

          The division rests on conditions that the rest of this lesson develops. Two matter most here. The offer must actually change attendance, which is relevance. And the offer must affect scores only through attendance, which is exclusion. If offered families also received free reading books, part of the 1.5 points would come from the books rather than the tutoring, and the ratio would overstate tutoring’s effect. The Expert level of Randomized experiments treats noncompliance in more depth. It is optional background that this module does not require.

          Share who attend tutoringOffered70%Not offered20%50 points of attendance the offer changed, the compliers
          The two attendance rates. The offer moves attendance from 20 percent to 70 percent. The 50-point difference is the share of students whose attendance the offer changed, and dividing the 1.5-point score gap by it spreads the offer’s effect over exactly those students, giving 3 points.
          The idea

          Two comparisons and a ratio

          Compare lottery winners with lottery losers in two ways. First, how much more likely were winners to attend? This is the first stage. Second, how much higher were winners’ scores? This is the reduced form. Both comparisons are clean, because the lottery was random.

          If winning raised attendance by 60 percentage points and raised average scores by 2.4 points, and winning matters only because it changes attendance, then the 2.4 points come entirely from the 60 percent of students whose attendance the lottery changed. Spread over those students, the effect of attending is 2.4 / 0.60 = 4 points. This ratio is the Wald estimator, and it is the simplest instrumental variables estimate.

          Assumptions

          Three conditions for an instrument

          • Relevance. The instrument changes the treatment. A lottery that barely affects attendance gives a ratio with a tiny denominator, and the estimate becomes unstable. This condition can be checked with the data.
          • Independence. The instrument is as good as randomly assigned. A real lottery satisfies this by design. A proposed instrument such as distance to the nearest charter school does not, because families choose where to live.
          • Exclusion. The instrument affects the outcome only through the treatment. A lottery win could violate this if winning itself, say by lifting a family’s expectations, raised scores even for winners who did not attend.

          Independence and exclusion cannot be fully tested. They must be argued from how the instrument arose. Angrist, Imbens, and Rubin (1996) add a fourth condition, monotonicity, which says the instrument never pushes anyone away from the treatment. A lottery win does not make any family less likely to attend.

          relevanceeffect?exclusion rules this path outZDYULottery (instrument)Attendance (treatment)Scores (outcome)Confounderno arrow in: independence
          The three conditions as arrows. Relevance is the arrow from the lottery to attendance, and it can be checked in the data. Independence means no arrow points into the lottery, which random assignment supplies by design. Exclusion removes the direct path from the lottery to scores, so the lottery can move scores only through attendance. The confounder may affect both attendance and scores, and the instrument works around it rather than removing it.
          Interpretation

          Whose effect is it?

          Some families would attend the charter school whatever the lottery said, perhaps by winning a seat later. Some would never attend, even after winning. The lottery changes behavior only for the rest, called compliers. The Wald estimate is the average effect for compliers, the local average treatment effect (Imbens & Angrist, 1994).

          This is a real limitation. If compliers benefit more or less than other students, the estimate does not describe the effect for everyone. It does describe the students whose attendance a policy like expanding charter seats would change, which is often the policy-relevant group.

          In the vocabulary of Module 1, the estimand here is the effect for compliers, and a study should say so before reporting a number.

          History

          The Vietnam draft lottery

          Angrist, Imbens, and Rubin (1996) illustrated the framework used in this module with a lottery that decided who could be drafted into the U.S. military. Each birth date was given a random number, and men born in 1950 were called up to number 195. They used the lottery to ask whether military service affected later civilian mortality among white men born in 1950 and 1951. Draft eligibility was random, but service was not. Some eligible men avoided service, and some ineligible men volunteered.

          Their analysis made the module’s vocabulary precise. Men who served only if drafted are compliers, men who never served whatever their number are never-takers, and men who served whatever their number, such as volunteers, are always-takers. Eligibility raised the probability of serving by 15.9 percentage points, so compliers were 15.9 percent of the population. Dividing the effect of eligibility on mortality by that share gave an estimated effect of service of 0.56 percentage points for compliers, about a 25 percent increase in the probability of death. The authors stressed that this number is credible only if eligibility affected mortality through service alone, and they examined how the estimate would change if, for example, draft-eligible men had stayed in school to avoid the draft.

          Worked example · simulated data

          A charter-school lottery

          The code simulates 6,000 applicants. Unobserved family motivation raises scores, makes some families attend whatever happens, and makes others unlikely to attend at all. Attending raises scores by 4 points.

          68.6 percent of winners and 8.8 percent of losers attended, so the first stage is 0.598. The naive comparison of attenders with non-attenders gives 5.25, too large because attenders are more motivated. Winners score 2.52 points higher than losers on average. Dividing by the first stage gives a Wald estimate of 4.21, close to the true 4.

          Run it yourself in R: code and output
          Case · published study

          Boston’s charter schools

          Many Boston charter schools are oversubscribed and admit students by lottery. Abdulkadiroğlu, Angrist, Dynarski, Kane, and Pathak (2011) used those lotteries to compare applicants who received an offer with applicants who did not, and used the offer as an instrument for charter attendance. They also studied Boston’s pilot schools, which have some of the autonomy of charter schools but operate within the district.

          Angrist, Hull, and Walters (2023) use lotteries at Massachusetts urban charter middle schools, mostly in Boston, to lay out the ingredients of such an analysis. Each school’s lottery has its own odds, so comparisons are made within risk sets, groups of applicants who faced the same lotteries. A charter offer raised charter enrollment by nearly 60 percentage points, a strong first stage. The two-stage least squares (2SLS) estimate, a regression version of the Wald ratio developed at the Intermediate level, implies that a year of charter attendance raised math scores by 0.45 standard deviations. Non-charter students in the sample scored 0.32 standard deviations below the state mean, so a year in a charter would be predicted to move a typical applicant above the state average. The corresponding OLS estimate was 0.33, smaller than the lottery estimate despite worries that charter applicants are positively selected.

          The design also sets the study’s scope. Its estimates describe applicants whose attendance depended on the lottery at oversubscribed schools. Schools without excess demand, and families who never applied, are outside it.

          The original study reports the same pattern in more detail. For middle school applicants, a charter offer raised time in a charter by about one year, and the 2SLS estimates imply that each year in a charter raised English language arts scores by about 0.25 standard deviations and math scores by about 0.42, with standard errors near 0.07. At the high school level the math effect was similar, about 0.35 to 0.36 per year, and the English effect was about 0.27 once baseline scores were controlled. Lottery estimates for Boston’s pilot schools were small and mostly insignificant, with some significant negative effects. The authors also ran an observational analysis that controlled for students’ earlier test scores. For schools in the lottery study it gave estimates close to the lottery estimates, but for charter schools without usable lotteries the observational estimates were only about half as large. The lottery schools, which the authors describe as following a No Excuses model, appear to be among the more effective charters, a reminder that a local effect describes the schools and students that generated it.

          Cautions

          Common threats to an instrument

          • A weak first stage.When the instrument barely moves the treatment, small biases in the reduced form are blown up by a small denominator.
          • Direct effects of the instrument.Distance to a college may affect earnings through local labor markets, not only through schooling.
          • Non-random instruments.Policy rules adopted because of local conditions are not as good as random.
          • Defiers.If the instrument pushes some people away from treatment, the ratio no longer has a clean interpretation.
          Practice

          How to read an IV study

          1. What is the instrument, and why is it as good as random?
          2. How strong is the first stage? Look for the first-stage estimate and its F statistic.
          3. Could the instrument affect the outcome in any other way? Think through each channel.
          4. Who are the compliers? Ask whether their effect is the one you care about.
          5. Does the reduced form tell the same story? It is the most transparent result and should be reported.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.Instrumental variables chapters written for education researchers. Start here.
          • Abdulkadiroğlu, A., Angrist, J. D., Dynarski, S. M., Kane, T. J., & Pathak, P. A. (2011). Accountability and flexibility in public schools: Evidence from Boston's charters and pilots. Quarterly Journal of Economics, 126(2), 699–748. https://doi.org/10.1093/qje/qjr017The Boston charter lottery study.
          • Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. https://doi.org/10.1080/01621459.1996.10476902The conditions behind IV and the complier interpretation.
          • Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828An accessible graduate treatment of IV.
          • Angrist, J., Hull, P., & Walters, C. (2023). Methods for measuring school effectiveness. In E. A. Hanushek, S. Machin, & L. Woessmann (Eds.), Handbook of the economics of education (Vol. 7, pp. 1–60). Elsevier. https://doi.org/10.1016/bs.hesedu.2023.03.001Lottery, regression discontinuity, and value-added methods for school effectiveness.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 3 · Intermediate

          Two-stage least squares.

          The Wald ratio handles one binary instrument and one binary treatment. Real studies have covariates, treatments measured in hours or years, and sometimes several instruments. Two-stage least squares generalizes the ratio to all of these. This lesson shows how it works, how to get its standard errors right, and how to report it.

          Before thisFoundations: using a nudgeThe Wald estimator and the three conditions.
          After thisAdvanced: LATE and compliersWhat 2SLS estimates when effects differ.
          By the end

          What you will be able to do

          1. Write the structural, first-stage, and reduced-form equations of an IV model.
          2. Compute 2SLS by hand and with software, with covariates.
          3. Explain why a manual second stage gives the wrong standard errors.
          4. Explain the role of covariates and of multiple instruments.
          5. Report the first stage, the reduced form, and the 2SLS estimate together.
          The model

          Three equations

          Let Y be the outcome, D the treatment, Z the instrument, and X covariates. The equation of interest is

          \[ Y_i = \alpha + \beta D_i + \gamma' X_i + u_i, \]

          where D is correlated with u, so ordinary least squares is biased. The first stage regresses the treatment on the instrument and covariates, and the reduced form regresses the outcome on them,

          \[ D_i = \pi_0 + \pi Z_i + \delta' X_i + v_i, \qquad Y_i = \rho_0 + \rho Z_i + \phi' X_i + e_i. \]

          With one instrument, the IV estimate is the ratio \(\hat\beta = \hat\rho / \hat\pi\), the Wald logic with covariates held fixed.

          Method

          How two-stage least squares works

          The name describes a recipe. In the first stage, regress D on Z and X and keep the fitted values \(\hat D\). These fitted values contain only the variation in D that comes from the instrument and the covariates, so they are uncorrelated with u. In the second stage, regress Y on \(\hat D\) and X. The coefficient on \(\hat D\) is the 2SLS estimate.

          The same covariates must appear in both stages. Leaving a covariate out of the first stage but keeping it in the second introduces bias, because the first-stage residual is then correlated with that covariate.

          Inference

          Getting the standard error right

          Running the second stage by hand gives the right coefficient and the wrong standard error. The residuals from that regression are \(Y - \hat\beta \hat D - \hat\gamma' X\), built with the fitted treatment. The correct residuals use the actual treatment, \(Y - \hat\beta D - \hat\gamma' X\). Dedicated IV routines use the correct residuals automatically. In R these include fixest::feols with the IV formula, in Python linearmodels.IV2SLS or pyfixest, and in Stata ivregress 2sls or ivreg2.

          As with any regression, heteroskedasticity-robust standard errors are a sensible default, and standard errors should be clustered at the level at which the instrument varies, such as the school or the lottery.

          Worked example · simulated data

          The effect of each hour of tutoring

          The code simulates 3,000 students. A randomly sent text-message nudge encourages families to use free tutoring. Unobserved motivation raises both hours and scores, and each hour truly adds 0.25 points.

          The first stage shows the nudge adding 6.25 hours, with a first-stage F of 1,238, a very strong instrument. OLS gives 0.578 points per hour, more than double the truth, because motivated students both attend more and score higher. 2SLS gives 0.255 with a standard error of 0.042, and the reduced form divided by the first stage, 1.595 / 6.254, reproduces it exactly. The standard error from a manual second stage is 0.044. The difference is small here, but it can be large in either direction, and it is never correct.

          Run it yourself in R: code and output
          Case · published study

          Quarter of birth and the return to schooling

          Angrist and Krueger (1991) built an instrument from school rules. Children typically had to turn six by January 1 of the school year in which they started, and could leave school on reaching the legal dropout age, often 16. A child born in the first quarter of the year therefore starts school older and reaches the dropout age having completed less schooling. If birth date affects earnings only through schooling, quarter of birth is an instrument for years of education.

          They first showed that the first stage reflects the mechanism they proposed. Men born in the first quarter had about a tenth of a year less schooling than men born in the fourth quarter. The pattern was strong for high school graduation and much weaker or absent for college and graduate degrees, which compulsory schooling laws do not touch. Comparing states with dropout ages of 16 and of 17 or 18 in a difference-in-differences design, they estimated that the laws kept about 4 percentage points more 16-year-olds enrolled in 1960.

          Worked numbers

          The Wald estimate in published data

          The simplest IV estimate divides the difference in mean outcomes by the difference in mean treatment between instrument groups. Angrist and Krueger (1991) report it for two census cohorts.

          Men, censusDifference in log weekly wageDifference in years of schoolingWald estimateOLS
          Born 1920–1929, 1970−0.00898−0.12560.0715 (0.0219)0.0801
          Born 1930–1939, 1980−0.01110−0.10880.1020 (0.0239)0.0709

          Differences compare men born in the first quarter with men born in the other three quarters. Standard errors in parentheses. Source is Angrist and Krueger (1991), Table III.

          For the older cohort, 0.00898 / 0.1256 = 0.0715, a return of about 7 percent per year of schooling, close to the OLS estimate of 8 percent. For the younger cohort the Wald estimate is larger than OLS, but neither difference is statistically significant. Their two-stage least squares models then used quarter of birth interacted with year of birth as 30 instruments, with year-of-birth controls, and gave similar answers. The Expert lesson explains why heavily over-identified versions of these models later became a cautionary example.

          Design

          The role of covariates

          When the instrument is randomly assigned, covariates are not needed for identification. Including baseline covariates such as prior scores still reduces residual variance and so improves precision, just as in an experiment. When the instrument is random only within groups, such as lotteries held separately at each school, indicators for those groups are required. Without them, the instrument is confounded by differences between groups. The Boston charter study, for example, conditions on the specific lotteries each student entered (Abdulkadiroğlu et al., 2011).

          Covariates should be measured before the instrument or be unaffected by it. Adjusting for a variable that the instrument changes, such as attendance during the program year, breaks the design.

          Extension

          Several instruments

          With more instruments than treatments, 2SLS combines them through the first-stage fitted values. This can increase precision, and it allows a test of whether different instruments give the same answer, called an overidentification test. Passing the test is weak evidence. If all the instruments share the same flaw, they will agree and still be wrong. When effects differ across people, different instruments can legitimately give different answers, because each identifies the effect for its own compliers.

          Many weak instruments are worse than one strong one. Bound, Jaeger, and Baker (1995) showed that when the quarter-of-birth instrument of Angrist and Krueger (1991) is interacted with many other variables, the resulting set of weak instruments pulls 2SLS toward OLS. The Expert lesson takes this up.

          Reporting

          What to report

          1. The first-stage estimate, its standard error, and the first-stage F statistic.
          2. The reduced-form estimate, which is free of any weak-instrument problem.
          3. The 2SLS estimate alongside OLS, with standard errors that match the design.
          4. The argument for independence and exclusion, including checks of balance across instrument values.
          5. A description of the compliers, covered in the Advanced lesson.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/97814008298282SLS, covariates, and standard errors in practice.
          • Angrist, J. D., & Krueger, A. B. (1991). Does compulsory school attendance affect schooling and earnings?. Quarterly Journal of Economics, 106(4), 979–1014. https://doi.org/10.2307/2937954Quarter of birth as an instrument for schooling.
          • Bound, J., Jaeger, D. A., & Baker, R. M. (1995). Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. Journal of the American Statistical Association, 90(430), 443–450. https://doi.org/10.1080/01621459.1995.10476536Why many weak instruments are dangerous.
          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.IV in education research, with worked examples.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 3 · Advanced

          Local average treatment effects and compliers.

          When everyone benefits equally, IV estimates the effect of the treatment for everyone. When effects differ, as they almost always do in schools, IV estimates the effect for a particular group, the compliers. This lesson explains that result, shows how to describe the compliers with the data, and discusses when a local effect is the right answer to the question.

          Before thisIntermediate: two-stage least squaresEstimation and standard errors.
          After thisExpert: weak instrumentsWhen the first stage is not strong.
          By the end

          What you will be able to do

          1. State the LATE theorem and its four assumptions.
          2. Explain why monotonicity cannot be dropped and judge whether it is plausible for a given instrument.
          3. Estimate the shares of compliers, always-takers, and never-takers.
          4. Estimate the average characteristics and potential outcomes of compliers.
          5. Explain how LATE relates to the ATE and the ATT.
          6. Interpret 2SLS with covariates or multiple instruments as a weighted average of complier effects.
          Theory

          The LATE theorem

          Imbens and Angrist (1994) and Angrist, Imbens, and Rubin (1996) showed that with a binary instrument and a binary treatment, under four assumptions, the Wald ratio equals the average effect among compliers,

          \[ \frac{E[Y \mid Z=1] - E[Y \mid Z=0]}{E[D \mid Z=1] - E[D \mid Z=0]} = E[Y_i(1) - Y_i(0) \mid \text{complier}]. \]

          The assumptions are independence of the instrument, exclusion, relevance, and monotonicity, meaning there are no defiers. The derivation is short. Always-takers and never-takers have the same treatment under either instrument value, so by exclusion they contribute nothing to the reduced form. Without defiers, all of the reduced form comes from compliers, and the first stage equals their share.

          The result changed how economists and education researchers interpret IV. An IV estimate is not automatically the effect of the treatment in general. It is the effect for the people the instrument moves, and different instruments move different people.

          Theory

          Why monotonicity matters

          Imbens and Angrist (1994) showed why the no-defiers assumption cannot be dropped. Without it, the reduced form mixes two groups, people the instrument pushes into treatment and people it pushes out. Their effects enter with opposite signs, so the reduced form can be zero, or even negative, when the treatment helps every single person. Assuming a constant effect, or that some value of the instrument makes treatment impossible, also solves the problem, but both are strong. Monotonicity is weaker, and it is the assumption the LATE result needs.

          Their examples show how to judge it. In the draft lottery, it is plausible that a man who would serve with a high lottery number would also serve with a low one. Now suppose applicants to a program are screened by one of two officials, assigned as good as randomly, and one official admits more applicants than the other. The official’s identity may well satisfy exclusion. But monotonicity requires that the stricter official admit only applicants the more lenient one would also admit. If the officials weigh several criteria differently, that is unlikely, and the IV estimate need not be a complier average. Education settings with teachers, counselors, or judges who make placement decisions raise the same question.

          With an instrument that takes several values, such as a lottery number or a distance, the IV estimate is a weighted average of LATEs between adjacent values of the instrument, with weights that are nonnegative and sum to one, as long as the instrument shifts treatment in the same direction across its range.

          Description

          Counting the compliance types

          Individual compliers are never identified, but their share is. Under monotonicity, people treated when Z = 0 are always-takers, so their share is \(E[D \mid Z=0]\). People untreated when Z = 1 are never-takers, with share \(1 - E[D \mid Z=1]\). Compliers make up the rest, which equals the first stage. Because the instrument is random, these shares apply to the whole sample.

          Description

          Describing compliers

          The same logic gives the mean of any baseline characteristic X among compliers,

          \[ E[X \mid \text{complier}] = \frac{E[XD \mid Z=1] - E[XD \mid Z=0]}{E[D \mid Z=1] - E[D \mid Z=0]}. \]

          Replacing X with Y gives compliers’ mean outcome with treatment, and replacing D with \(1-D\) and reversing the sign gives their mean outcome without it. Reporting these quantities lets readers judge whether the compliers resemble the population they care about. Angrist and Pischke (2009) describe these calculations and recommend them as standard practice.

          Worked example · simulated data

          Who complies?

          The code simulates 8,000 applicants to a program. Students with higher prior scores are more likely to be always-takers. The effect is 8 points for always-takers, 4 for compliers, and would be 1 for never-takers.

          The estimated shares are 0.147 always-takers, 0.259 never-takers, and 0.594 compliers, close to the true 0.152, 0.256, and 0.592. The Wald estimate is 4.18, near the compliers’ true 4. The average effect on everyone who attended is 5.38, and on all students 3.84. The three estimands differ because effects differ across types, and IV recovers only one of them.

          The compliers’ mean prior score is estimated at −0.049, against a true −0.064, compared with −0.001 for all students and 0.286 for always-takers. Their mean outcome is estimated at 53.63 with the program and 49.45 without it, a difference that matches the Wald estimate by construction.

          Run it yourself in R: code and output
          Extensions

          2SLS as a weighted average

          The clean complier interpretation holds exactly for a binary instrument and a binary treatment without covariates (Imbens & Angrist, 1994). Angrist and Imbens (1995) extended it to a treatment measured in units, such as years of schooling. Under independence and monotonicity, the IV estimand is an average causal response, a weighted average of the effects of each additional unit, \(E[Y_j - Y_{j-1}]\), among people whom the instrument moves from below j units to j or more. The weight on unit j is proportional to the share of people the instrument moves across it, which equals the difference between the two groups’ cumulative distributions of treatment at j.

          Those weights can be estimated, which turns an abstract result into a description of who drives the estimate. For quarter of birth, the schooling distributions of men born in the first and fourth quarters do not cross, as monotonicity requires, and their difference is largest between 8 and 12 years. At most about 2 percent of men were induced to complete 11th grade, and few were moved at college levels. The quarter-of-birth estimate is therefore mostly a return to high school years for men at risk of dropping out (Angrist & Imbens, 1995).

          Two further results matter in practice. With several mutually exclusive instruments, 2SLS is a weighted average of the separate Wald estimates, giving more weight to instruments with larger first stages. With discrete covariates and a saturated first stage, it averages covariate-specific effects with weights proportional to the variance of the predicted treatment within each covariate cell (Angrist & Imbens, 1995). Abadie (2003) showed how to go further. A weighting function, κ, makes any characteristic of compliers estimable as a weighted average over the whole sample, and allows a regression of the outcome on treatment and covariates to be fitted for compliers alone. Unlike the usual 2SLS with covariates, this gives the best linear approximation to the compliers’ average response function, even if that function is not linear.

          Under monotonicity the weights are never negative, so the estimate is a genuine average of effects for some population. That population is defined by the instrument, which is why reporting the characteristics of compliers matters.

          Interpretation

          Is a local effect useful?

          Critics argue that LATE answers a question chosen by the instrument rather than by the researcher. Defenders reply that compliers are often exactly the policy-relevant group. Charter lottery compliers are the students who would attend if more seats were offered, which is the margin a policy to expand charters would affect.

          A useful habit is to state the policy question first and then ask whether the instrument’s compliers match it. When they do not, extrapolating requires assumptions about how effects vary, and those assumptions should be stated rather than left implicit.

          Diagnostics

          Checking the assumptions

          • Independence.Compare baseline characteristics across instrument values, as in an experiment’s balance table.
          • Exclusion.Look for groups whose treatment cannot change, such as students ineligible for the program, and check that the instrument does not affect their outcomes.
          • Monotonicity.Check that the first stage has the same sign in every subgroup. A reversed sign in some group suggests that monotonicity fails, although the same sign everywhere does not prove that it holds.
          • Relevance.Report the first-stage F. The Expert lesson explains why the usual rule of thumb is not enough.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Imbens, G. W., & Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475. https://doi.org/10.2307/2951620The local average treatment effect.
          • Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. https://doi.org/10.1080/01621459.1996.10476902Compliance types and the assumptions in potential-outcomes notation.
          • Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. https://doi.org/10.1515/9781400829828Complier characteristics and 2SLS weights.
          • Abdulkadiroğlu, A., Angrist, J. D., Dynarski, S. M., Kane, T. J., & Pathak, P. A. (2011). Accountability and flexibility in public schools: Evidence from Boston's charters and pilots. Quarterly Journal of Economics, 126(2), 699–748. https://doi.org/10.1093/qje/qjr017An education study where compliers are the policy margin.
          • Angrist, J. D., & Imbens, G. W. (1995). Two-stage least squares estimation of average causal effects in models with variable treatment intensity. Journal of the American Statistical Association, 90(430), 431–442. https://doi.org/10.1080/01621459.1995.10476535The average causal response for treatments measured in units.
          • Abadie, A. (2003). Semiparametric instrumental variable estimation of treatment response models. Journal of Econometrics, 113(2), 231–263. https://doi.org/10.1016/S0304-4076(02)00201-4κ weighting for complier characteristics and response functions.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 3 · Expert

          Weak instruments and robust inference.

          An instrument that barely moves the treatment produces estimates that are biased toward OLS and confidence intervals that are too narrow. The familiar rule that a first-stage F above 10 is safe turns out to be too lenient for the t-test most papers report. This lesson shows the problem by simulation and presents inference that remains valid however weak the instrument is.

          Before thisAdvanced: LATE and compliersWhat IV estimates.
          Next moduleRegression discontinuitySharp designs first, then fuzzy designs as IV at a cutoff.
          By the end

          What you will be able to do

          1. Explain why 2SLS is biased toward OLS when instruments are weak.
          2. Interpret the first-stage F statistic and its common thresholds.
          3. Carry out an Anderson–Rubin test and explain why it is robust.
          4. Explain why many weak instruments are especially dangerous.
          5. Choose an inference strategy for a just-identified IV study.
          6. Report how far an interval widens if the instrument has a small direct effect, and the direct effect that would erase the result.
          7. Use an effective F statistic when errors are heteroskedastic or clustered, and say what it does not settle.
          Theory

          Why weak instruments mislead

          2SLS divides the reduced form by the first stage. When the true first stage is small, its sampling error is large relative to its value, and the estimated denominator sometimes lands near zero. The ratio then has a heavy-tailed, non-normal distribution, and the usual standard error, built on a normal approximation, misstates the uncertainty.

          There is also a bias. The fitted values in the first stage partly reflect noise, and that noise is correlated with the treatment’s endogenous component. In finite samples, 2SLS therefore inherits some of OLS’s bias, and the share inherited grows as the instrument weakens. With many instruments, each adds more noise to the fitted values, so 2SLS drifts further toward OLS (Bound et al., 1995; Staiger & Stock, 1997).

          History

          A cautionary tale from education

          Angrist and Krueger (1991) argued that quarter of birth affects years of schooling through compulsory attendance laws, and their simple comparisons of men born in different quarters were precise enough to be informative (see the Intermediate lesson). Some of their specifications, however, interacted quarter of birth with year and state of birth, producing as many as 180 instruments. Bound, Jaeger, and Baker (1995) showed that these first stages were weak and that, with so many weak instruments, 2SLS estimates drifted toward OLS. In one demonstration, instruments generated at random produced estimates and standard errors much like those from the real quarter-of-birth instruments. The episode helped make the first-stage F statistic a standard part of IV reporting.

          Angrist and Kolesár (2024) draw a narrower lesson from the same history. The weak instrument problem was sharpest for heavily over-identified models. Most applied studies use a single instrument for a single treatment, and that case behaves much better, as the next section explains.

          Diagnostics

          The first-stage F and its thresholds

          Staiger and Stock (1997) proposed treating instruments as weak when the first-stage F statistic falls below 10, a rule that became nearly universal. The threshold was designed to limit the bias of 2SLS relative to OLS, not to guarantee that a 95 percent confidence interval covers the truth 95 percent of the time.

          Lee, McCrary, Moreira, and Porter (2022) studied the second question for the case of one instrument and one treatment. They found that the usual t-test with a 1.96 critical value needs a first-stage F above about 104.7 to have its intended 5 percent error rate in the worst case. For smaller F values, they provide adjusted critical values, the tF procedure, which widen the interval smoothly as the instrument weakens. In heteroskedastic or clustered data, the F statistic should be computed with the same robust variance used for inference (Andrews, Stock, & Sun, 2019).

          Method

          The Anderson–Rubin test

          To test whether the effect equals some value \(\beta_0\), form the adjusted outcome \(Y - \beta_0 D\) and regress it on the instrument and covariates. If \(\beta_0\) is the true effect, the adjusted outcome contains only the error term, which the instrument does not predict. So the test is simply whether the instrument’s coefficient is zero.

          This regression involves no ratio, so the test’s error rate is correct whatever the strength of the instrument. Collecting every \(\beta_0\) that the test does not reject gives the Anderson–Rubin confidence set. When the instrument is strong, the set is close to the usual interval. When it is weak, the set widens, and it can be unbounded, which correctly signals that the data cannot pin down the effect (Andrews et al., 2019).

          Debate

          How worried should a study with one instrument be?

          Lee, McCrary, Moreira, and Porter (2022) and Andrews, Stock, and Sun (2019) set their standards by the worst case. Angrist and Kolesár (2024) ask instead how likely the worst case is. With one instrument and one treatment, the behavior of the usual t-test depends on just two numbers, the strength of the first stage and the endogeneity ρ, the correlation between the errors in the first-stage and outcome equations. If |ρ| is below 0.76, a nominal 5 percent test rejects a true null less than 10 percent of the time however weak the instrument, and if |ρ| is below 0.565 it never over-rejects. The reason is that as the instrument weakens, the median bias of the IV estimate grows, but its standard error grows with it.

          They then calibrate ρ in three well-known studies, Angrist and Krueger (1991), Angrist and Evans (1998), and the class-size estimates of Angrist and Lavy (1999). Estimated |ρ| was at most 0.47 and exceeded 0.4 in only one specification. They argue that this is typical, because endogeneity is capped by how large causal effects can plausibly be. In education, where an effect of half a standard deviation from a large change in class size would be remarkable, they put the cap for class-size studies around 0.5.

          Two further results bear on practice. Screening on the sign of the estimated first stage, which researchers do whenever theory predicts the sign, roughly halves the median bias of the IV estimate and leaves confidence interval coverage almost unchanged. Screening on F > 10, by contrast, makes bias worse. Robust methods also cost precision. With F = 16, Anderson–Rubin intervals are 15 percent to 32 percent longer than conventional ones, and tF intervals 42 percent longer.

          The authors agree on the numbers and differ on which default to recommend. Angrist and Kolesár (2024) favor conventional inference with sign screening when endogeneity is plausibly moderate. Andrews, Stock, and Sun (2019) favor reporting robust inference routinely. A careful study can do both, report the first stage and its sign, the conventional interval, and the Anderson–Rubin set, and explain any difference between them.

          Monte Carlo · simulated data

          Weak, moderate, and strong instruments

          The code draws 500 samples of 500 observations for three instrument strengths. The true effect is 1 and the treatment is strongly endogenous, so OLS is centered near 1.7 to 1.8.

          Strength (n × π²)Mean FShare F > 10Median 2SLSt-test coverageAR coverage
          23.15 percent1.1089.2 percent95.4 percent
          1011.451 percent1.0192.0 percent94.2 percent
          100104.0100 percent1.0193.6 percent93.8 percent

          With a weak instrument, the median 2SLS estimate is pulled from 1 toward OLS, and the t-test interval covers the truth only 89.2 percent of the time. The Anderson–Rubin test covers 95.4 percent, close to its nominal level. With a moderate instrument, half the samples pass the F above 10 rule, yet t-test coverage is still 92.0 percent. With a strong instrument, all three methods agree, and coverage differences of a point or so reflect simulation noise with 500 samples.

          The second script concerns the section on a doubtful exclusion restriction. It gives the instrument a direct effect of 0.2 and compares intervals that allow for it.

          Run it yourself in R: code and output
          Practice

          Do not screen on the F statistic

          A tempting workflow is to report the t-test if F exceeds 10 and abandon the instrument otherwise. That procedure itself distorts inference, because the samples that pass the screen are not representative of the samples one would draw, and in calibrated simulations screening on F above 10 raised the maximal rejection rate of a nominal 5 percent t-test above 70 percent. Andrews, Stock, and Sun (2019) recommend reporting robust inference such as Anderson–Rubin confidence sets in every just-identified study, alongside the F statistic, rather than letting the F statistic decide which inference to report.

          One kind of screening is benign. Angrist and Kolesár (2024) show that discarding estimates whose first stage has the wrong sign, the minimal screen of c = 0, minimizes median bias among all rules of the form t > c, while leaving coverage nearly unchanged. Any positive cut-off, including the conventional F > 10, does worse on bias unless the instrument is extremely weak.

          Sensitivity

          When the exclusion restriction is doubtful

          The exclusion restriction says the instrument affects the outcome only through the treatment. Write the outcome equation as \(Y = X\beta + Z\gamma + \varepsilon\), where \(Z\) is the instrument. The restriction is the claim that \(\gamma\) is exactly zero. Conley, Hansen, and Rossi (2012) call this dogmatic, because a small direct effect, which a defender of the instrument might concede is possible, makes the usual interval wrong, and the error does not shrink as the sample grows.

          Their simplest remedy asks what the data say if \(\gamma\) is allowed to lie anywhere in a stated range. For each value in the range, build the confidence interval for \(\beta\), and take the union. The union has coverage at least as large as the nominal level, so it is conservative. In a just-identified model with a positive direct effect of at most \(\delta\), the union runs as follows, where \(\hat\pi\) is the first-stage coefficient.

          \[ \Bigl[\; \hat\beta - \frac{\delta}{\hat\pi} - 1.96\,\widehat{se}, \;\; \hat\beta + 1.96\,\widehat{se} \;\Bigr] \]

          The interval widens by \(\delta/\hat\pi\) on one side. A strong first stage makes the instrument tolerant of a violation, and a weak one makes it fragile. The authors also describe approaches that put a prior on \(\gamma\) or treat it as small in a local sense, which give narrower intervals under stronger assumptions.

          The second script in the code panel under the worked simulation uses 2,000 students, a true effect of 2, and an instrument that has a direct effect of 0.2 on the outcome, which the analyst does not know. The first stage is 0.51 and the IV estimate averages 2.41 over 300 samples, so the direct effect adds about 0.2/0.51, or 0.4, of bias. A conventional interval is 0.27 wide and contains the truth in none of the samples. Allowing \(\gamma\) between 0 and 0.1 gives a width of 0.47 and coverage of 0.18. Allowing it between 0 and 0.2, which matches the truth, gives 0.67 and 0.93, and between 0 and 0.3 gives 0.87 and 1.00. Coverage is near the nominal level only when the range is about right, and a range that is too narrow still fails.

          In the first sample the estimate is 2.39 with a standard error of 0.07. The direct effect that would bring the lower end of the interval to zero is 1.14, which would account for most of the reduced form of about 1.2. A report can state this breakdown value and let readers judge whether a direct effect of that size is plausible. The method does not prove that the restriction holds. It shows how much of the conclusion depends on it.

          Diagnostics

          An F statistic that allows for clustering

          Schools cluster students and errors often differ in spread, so the usual first-stage F statistic, which assumes constant variance and independent errors, can look strong when the instrument is not. Montiel Olea and Pflueger (2013) propose an effective F statistic that uses a variance estimate allowing for heteroskedasticity, serial correlation, and clustering, together with critical values that account for them. With a single instrument it equals the robust F statistic, and the usual critical values are replaced by theirs. Comparing a robust F statistic with the homoskedastic rule of 10, a common practice, has no theoretical support, as the authors point out.

          The null hypothesis being tested is that the bias of two-stage least squares exceeds 10 percent of a worst-case benchmark. With one instrument and a 5 percent test size, the critical value is 23.1, and 23.1 is a safe rule of thumb for two-stage least squares in general. An effective F above that value is evidence that the bias is below 10 percent of the benchmark. The number is larger than 10 because the benchmark is the worst-case bias over the possible error structures, rather than the bias under homoskedastic errors.

          This threshold answers a question about bias, not about the size of the t-test. As the earlier section reports, a nominal 5 percent t-test needs a much larger F, near 105, to have its intended error rate in the worst case. Neither number should be used to decide which results to report. Report the effective F with the robust inference that does not depend on it, such as Anderson–Rubin confidence sets.

          Evidence

          Weak instruments in published research

          Andrews, Stock, and Sun (2019) examined 230 IV specifications from 17 papers published in the American Economic Review between 2014 and 2018. Thirteen of the 16 papers with a single endogenous regressor reported at least one first-stage F statistic, so researchers are clearly aware of the problem. Yet many reported F statistics fell below 10, and an unusual number sat just above 10, a pattern consistent with selection favoring results that clear the rule of thumb. Some papers computed the F statistic assuming homoskedastic errors while computing standard errors that did not.

          They calibrated simulations to these specifications. Without screening, nominal 5 percent t-tests rejected a true null at most a little under 35 percent of the time in the calibrated designs. When draws with F below 10 were discarded, as a publication filter would, the maximal rejection rate rose above 70 percent. Screening compounded rather than reduced the problem, whatever threshold or version of the F statistic was used.

          Their recommendations for the case of one endogenous regressor are to judge strength with an effective F statistic that is valid with heteroskedastic or clustered errors, and, with a single instrument, to report Anderson–Rubin confidence sets in every case, since they are efficient when the instrument is strong and valid when it is weak. Weak instruments alone, they conclude, are not a reason to discard an application.

          Extension

          Many instruments

          When there are many instruments, the bias toward OLS grows with their number. Remedies include using fewer, stronger instruments, estimators such as limited-information maximum likelihood that are less biased with many instruments, and weak-instrument-robust tests that generalize Anderson–Rubin (Andrews et al., 2019). In education, settings such as judge or examiner designs, where many decision-makers with different tendencies serve as instruments, raise these issues directly.

          Reporting

          What to report

          1. The first-stage F statistic, computed with the same variance estimator as the main inference.
          2. The reduced form, whose t-test is valid regardless of instrument strength.
          3. An Anderson–Rubin confidence set, or tF-adjusted intervals, alongside conventional ones.
          4. The number of instruments and, with many, an estimator robust to many weak instruments.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Andrews, I., Stock, J. H., & Sun, L. (2019). Weak instruments in instrumental variables regression: Theory and practice. Annual Review of Economics, 11, 727–753. https://doi.org/10.1146/annurev-economics-080218-025643A practical review of weak-instrument inference. Start here.
          • Lee, D. S., McCrary, J., Moreira, M. J., & Porter, J. (2022). Valid t-ratio inference for IV. American Economic Review, 112(10), 3260–3290. https://doi.org/10.1257/aer.20211063Why F above 10 is not enough for the t-test, and the tF fix.
          • Staiger, D., & Stock, J. H. (1997). Instrumental variables regression with weak instruments. Econometrica, 65(3), 557–586. https://doi.org/10.2307/2171753Weak-instrument asymptotics and the F above 10 rule.
          • Bound, J., Jaeger, D. A., & Baker, R. M. (1995). Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. Journal of the American Statistical Association, 90(430), 443–450. https://doi.org/10.1080/01621459.1995.10476536The quarter-of-birth critique.
          • Angrist, J. D., & Krueger, A. B. (1991). Does compulsory school attendance affect schooling and earnings?. Quarterly Journal of Economics, 106(4), 979–1014. https://doi.org/10.2307/2937954The original quarter-of-birth study.
          • Angrist, J., & Kolesár, M. (2024). One instrument to rule them all: The bias and coverage of just-ID IV. Journal of Econometrics, 240(2), Article 105398. https://doi.org/10.1016/j.jeconom.2022.12.012Why conventional inference with one instrument is usually reliable, and the value of sign screening.
          • Angrist, J. D., & Evans, W. N. (1998). Children and their parents' labor supply: Evidence from exogenous variation in family size. American Economic Review, 88(3), 450–477.Same-sex sibships as an instrument for family size, one of the three calibration studies.
          • Angrist, J. D., & Lavy, V. (1999). Using Maimonides' rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 114(2), 533–575. https://doi.org/10.1162/003355399556061Maimonides rule class-size estimates, a third calibration study.
          • Conley, T. G., Hansen, C. B., & Rossi, P. E. (2012). Plausibly exogenous. The Review of Economics and Statistics, 94(1), 260–272. https://doi.org/10.1162/rest_a_00139How to report an instrument that may have a small direct effect.
          • Montiel Olea, J. L., & Pflueger, C. (2013). A robust test for weak instruments. Journal of Business & Economic Statistics, 31(3), 358–369. https://doi.org/10.1080/00401706.2013.806694An effective first-stage F with critical values that allow for heteroskedasticity and clustering.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 4 · Foundations

          Comparing just above and just below a cutoff.

          Schools use cutoffs all the time. A test score below a line means summer school, an enrollment above a number means a new class, a rating below a threshold means a warning. Students just on either side of a cutoff are nearly identical, yet one group gets the program and the other does not. Regression discontinuity designs use that fact to build a comparison that, near the cutoff, resembles a lottery.

          Before thisRandomized experiments, FoundationsWhy comparable groups matter.
          After thisIntermediate: local linear estimationBandwidths, kernels, and polynomials.
          By the end

          What you will be able to do

          1. Identify the running variable, cutoff, and treatment in an RD design.
          2. Explain why students near the cutoff are comparable.
          3. Read an RD graph and judge whether a jump is larger than ordinary variation.
          4. State the key assumption, that nothing else jumps at the cutoff.
          5. Explain why the estimate applies to students near the cutoff.
          6. Spot manipulation and other threats in a published study.
          The idea

          A line in the sand

          Suppose a district requires students who score below a cutoff on a spring placement test to attend summer school. Comparing all summer school students with all other students mixes the program with large differences in achievement. Students far below the cutoff are very different from students far above it.

          But compare a student who scored one point below the cutoff with one who scored one point above. Their achievement, motivation, and family backgrounds are, on average, almost the same. Whether they landed just below or just above depended partly on chance, on a lucky guess or a bad morning. Yet one attended summer school and the other did not. Any difference in their later outcomes is plausibly the effect of summer school.

          Thistlethwaite and Campbell (1960), writing in an education journal, introduced this idea to study awards given to students above a test-score cutoff. The score that determines treatment is called the running variable, and the line is the cutoff.

          40506070-20-100 cutoff1020Placement score relative to the cutoffjump = effectSummer school (below cutoff)No summer school
          The design in one picture. Each dot is a student (simulated data). Later outcomes rise with the placement score on both sides, and the dashed line shows how the right-hand trend would continue. The vertical gap between the fitted lines at the cutoff is the estimated effect of summer school, here 6 points. Students far from the cutoff differ in many ways. The comparison that matters is local.
          The picture

          Reading an RD graph

          Every RD study should show a graph. The horizontal axis is the running variable, and the vertical axis is the average outcome within narrow bins of it. Outcomes usually rise smoothly with the running variable, because higher-scoring students tend to score higher next year too. The question is whether the line jumps at the cutoff. A smooth line with a visible break exactly at the cutoff suggests an effect. A graph where the “jump” is no larger than the wiggles elsewhere is not.

          The jump is the estimated effect. To measure it, researchers fit a line to the data on each side of the cutoff and compare where the two lines meet the cutoff. The Intermediate lesson explains how to fit those lines well.

          Assumptions

          What has to be true

          • Nothing else changes at the cutoff. If students below the cutoff also got a different teacher or a letter home, the jump would mix those with summer school. Formally, the average outcome students would have without the program must change smoothly across the cutoff (Hahn, Todd, & Van der Klaauw, 2001).
          • No precise manipulation. If students, teachers, or parents can push scores to one side of the cutoff, those just above may differ systematically from those just below. If a teacher rescores tests that land just below, for instance, the students who are moved will be the ones the teacher favored.

          Lee and Lemieux (2010) explain why these conditions make RD close to a randomized experiment near the cutoff. If students cannot precisely control their score, chance determines which side they land on, and so the two sides are comparable on average in every respect, measured or not.

          Worked example · simulated data

          Summer school at a cutoff

          The code simulates 2,000 students whose placement scores, centered at the cutoff, range from −40 to 40. Students below 0 attend summer school, which truly raises next year’s score by 3 points.

          Comparing all students below with all students above gives −12.41, because the groups started far apart. Narrowing the window helps. Within 10 points of the cutoff the difference is −1.07, within 5 points it is 1.19, and within 2 points it is 2.12, based on only 89 students. Fitting a straight line on each side and comparing them at the cutoff gives 3.21, close to the truth. Narrow windows reduce bias but leave few students, which is the central tradeoff of RD.

          Run it yourself in R: code and output
          Interpretation

          Effect for whom?

          RD estimates the effect for students near the cutoff. Summer school might help a student who scored 30 points below the cutoff much more, or much less, than one who scored 1 point below. The design says nothing directly about them. For a policy question about moving the cutoff slightly, the local effect is exactly what is needed. For a question about the program’s value to the lowest-achieving students, it is not.

          In the vocabulary of Module 1, the estimand here is the effect at the cutoff, and a study should state that target in advance.

          Case · published study

          Summer school and retention in Chicago

          In 1996 Chicago Public Schools ended social promotion. Third graders had to score at least 2.8 grade equivalents in both reading and math on the Iowa Test of Basic Skills, about the 20th percentile nationally, and sixth graders 5.3. Students below the cutoff in June attended a six-week summer program and retested in August, and those who still fell short repeated the grade. Jacob and Lefgren (2004) compared students just below and just above the June reading cutoff among those who had passed math.

          The jump in treatment was large but not complete. Among third graders, 92 percent of those just below the cutoff received summer school or retention, against about 1 percent of those just above, because some students received waivers. That makes this a fuzzy design, one in which the cutoff changes the probability of treatment without fixing it. The authors also argue that students could not have placed themselves just above the cutoff, since they could not control their score so precisely, and they found no sign of bunching in the score distribution after the policy began.

          OutcomeThird gradeSixth grade
          Reading, one year later0.104 (0.025)0.024 (0.027)
          Reading, two years later0.062 (0.026)0.000 (0.030)
          Math, one year later0.136 (0.024)0.077 (0.021)
          Math, two years later0.095 (0.026)0.019 (0.023)

          IV estimates in logits on the Rasch scale, with standard errors in parentheses, for students who passed math and were within 0.2 grade equivalents of the reading cutoff, with student covariates. Source is Jacob and Lefgren (2004), Table 3.

          For third graders, the first-year effect was about 20 percent of a typical year’s learning gain, and roughly 25 to 40 percent of it faded by the second year. For sixth graders, the net effect was essentially zero in reading and small in math. The estimates hold for students near the cutoff, low-achieving students for whom such programs are designed. The authors stress that, with effects likely to differ across students, they cannot say what would happen if every student were treated.

          Case · published study

          School attendance boundaries and house prices

          The cutoff in an RD design does not have to be a test score. Black (1999) used the lines that divide neighboring elementary school attendance zones in suburban Boston districts. Two houses on opposite sides of a boundary share a neighborhood, local taxes, and school district, but send their children to different elementary schools. Comparing their prices shows how much parents pay for a school with higher test scores.

          The data covered 22,679 house sales in 39 school districts with 181 attendance boundaries. A conventional regression across all houses found that a 5 percent increase in a school’s test scores went with a 4.9 percent higher house price. Restricting the comparison to houses within 0.35, 0.20, or 0.15 miles of a boundary, and comparing only houses on the same boundary, shrank the estimate to roughly 2.1 percent, about $3,948 at the mean price of $188,000. Unmeasured neighborhood qualities probably inflated the conventional estimate to more than twice the boundary estimate.

          Black also ran a placebo check. At invented boundaries drawn through the middle of attendance zones, houses on the higher-scoring side sold for no more than houses on the other side, while at the real boundaries they did. The study shows two habits that carry over to every RD analysis, narrowing the comparison toward the cutoff and checking that nothing happens where nothing should.

          Cautions

          Common misreadings

          • “The groups differ a lot, so the design fails.”Students far from the cutoff differ, by design. Only students near it need to be comparable.
          • “The effect applies to all students in the program.”It applies near the cutoff.
          • “A big jump in the graph settles it.”Check whether jumps of similar size appear at other points, and whether students bunch just on one side.
          • “Using all the data is always better.”Data far from the cutoff help with precision but can bias the estimate if the line is curved.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.RD chapters written for education researchers. Start here.
          • Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. https://doi.org/10.1037/h0044319The original RD paper, in an education journal.
          • Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A regression-discontinuity analysis. Review of Economics and Statistics, 86(1), 226–244. https://doi.org/10.1162/003465304323023778Summer school and retention in Chicago.
          • Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355. https://doi.org/10.1257/jel.48.2.281Why RD resembles a local randomized experiment.
          • Black, S. E. (1999). Do better schools matter? Parental valuation of elementary education. Quarterly Journal of Economics, 114(2), 577–599. https://doi.org/10.1162/003355399556070A boundary design for the value parents place on school quality, with a placebo boundary check.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 4 · Intermediate

          Local linear estimation and bandwidths.

          The RD estimate is the gap between two curves at the cutoff. How we fit those curves decides how good the estimate is. This lesson covers the standard approach, local linear regression with a kernel and a bandwidth, and explains why global high-order polynomials, once common, are now discouraged.

          Before thisFoundations: comparing just above and belowThe RD idea and assumptions.
          After thisAdvanced: validity checks and fuzzy designsTesting the assumptions.
          By the end

          What you will be able to do

          1. Define the sharp RD estimand as a difference in limits.
          2. Estimate it with local linear regression and a triangular kernel.
          3. Explain the bias and variance tradeoff in choosing a bandwidth.
          4. Explain why global high-order polynomials are discouraged.
          5. Make an RD graph that shows the jump and the data honestly.
          6. Report RD estimates across a range of bandwidths.
          Theory

          The estimand

          Let R be the running variable, centered so the cutoff is 0, and let treatment apply when \(R < 0\). The sharp RD estimand is the difference between the limits of the average outcome as R approaches 0 from each side,

          \[ \tau_{RD} = \lim_{r \uparrow 0} E[Y \mid R = r] - \lim_{r \downarrow 0} E[Y \mid R = r]. \]

          Under continuity of the potential outcomes at the cutoff, this equals the average effect for students at the cutoff (Hahn et al., 2001). Every RD estimator is a way of estimating these two limits from data on each side.

          Method

          Local linear regression

          Choose a bandwidth h. On each side, fit a weighted straight line to the students within h of the cutoff, giving more weight to those nearer it. The standard weights come from a triangular kernel, which gives weight \(1 - |R|/h\), so a student at the cutoff counts fully and a student at distance h counts not at all. The estimate is the difference between the two lines’ intercepts at the cutoff.

          Imbens and Lemieux (2008) recommend local linear rather than local constant fits. A local mean on each side is biased whenever the outcome slopes with the running variable, because the students used lie on average away from the cutoff. A local line corrects for the slope, and its remaining bias depends mainly on curvature.

          cutoff − hcutoffcutoff + hPosition of a student's scoretriangular kerneluniform kernelfullzeroweight
          How the weights fall off. With a triangular kernel, a student at the cutoff counts fully and the weight declines in a straight line to zero at distance h, matching the formula in the text. A uniform kernel, shown in gold, counts every student inside the bandwidth equally. Students beyond h receive no weight under either choice.
          Tradeoff

          Choosing a bandwidth

          A narrow bandwidth uses students very close to the cutoff, where a straight line is a good approximation, so bias is small. But it uses few students, so variance is large. A wide bandwidth reverses both. Data-driven methods choose the bandwidth that minimizes an estimate of mean squared error, balancing the two. Calonico, Cattaneo, and Titiunik (2014) developed widely used versions, implemented in the rdrobust packages for R, Python, and Stata.

          Because no single bandwidth is correct, good practice reports the estimate at the data-driven bandwidth and shows how it changes at, for example, half and twice that bandwidth. Estimates that are stable across bandwidths are more credible.

          Worked example · simulated data

          Bandwidths and polynomials

          The code simulates 2,000 students with a true effect of 3. The outcome curves differently on the two sides of the cutoff, which is what makes wide windows risky.

          With a bandwidth of 5 the local linear estimate is 4.37, with a standard error of 1.21, based on 259 students. At 10 it is 3.57 (0.81), and at 20 it is 4.10 (0.55). At 40, using every student, it is 6.00 with a standard error of 0.40. The standard error shrinks steadily as the bandwidth grows, but at 40 the curvature has pushed the estimate far from the truth, and its small standard error gives false confidence.

          Run it yourself in R: code and output
          Caution

          Against global high-order polynomials

          An older approach fits a polynomial of order three, four, or five to all the data on each side. Gelman and Imbens (2019) argue against it. High-order polynomials can give large weight to observations far from the cutoff, produce estimates that are sensitive to the chosen order, and yield confidence intervals that are too narrow.

          The second half of the code fits global polynomials of orders 1 to 5. A global line gives 7.74, badly biased. Orders 2 through 4 give 3.41 to 3.45, because the true curve on each side is a quadratic, so a quadratic is enough. Order 5 gives 3.76, and the standard error climbs from 0.38 at order 1 to 1.16 at order 5. In real data the true shape is unknown, so there is no way to know which order would be right. Local linear estimation with a data-driven bandwidth avoids that choice.

          Practice

          Making the RD graph

          • Bin the running variable.Plot the mean outcome in equally spaced bins, without smoothing across the cutoff.
          • Overlay the fitted lines.Show the local linear fits within the bandwidth on each side.
          • Do not choose bins to make the jump look bigger.Data-driven bin choices are available in the RD software packages (Cattaneo, Idrobo, & Titiunik, 2019).
          • Show the whole range.Readers should see how the outcome behaves away from the cutoff too.
          Practice

          Three graphs and an IV connection

          Imbens and Lemieux (2008) organize an RD analysis around three graphs. The first plots mean outcomes in bins of the running variable, computed separately on each side so that no bin straddles the cutoff. If this plot shows no jump, more elaborate estimators are unlikely to find a credible one, and the plot should also be checked for jumps elsewhere that are as large as the one at the cutoff. The second plots predetermined covariates the same way, and for fuzzy designs the treatment rate, which should jump. The third plots the number of observations per bin, where a jump would suggest manipulation.

          They also explain why local linear regression replaced the simple comparison of means within a window. Because the cutoff is a boundary and outcomes usually trend with the running variable, a difference in means has bias proportional to the bandwidth, while a local linear fit has bias proportional to its square. For fuzzy designs they note a useful equivalence. With a uniform kernel and the same bandwidth for outcome and treatment, the fuzzy RD estimate equals two-stage least squares with an indicator for being above the cutoff as the instrument, controlling for separate linear trends on each side. The estimate is then a LATE, the average effect for compliers at the cutoff (Module 3, Instrumental variables, introduces both terms), which narrows external validity further than a sharp design.

          Extensions

          Covariates and clustering

          Baseline covariates are not needed for identification in RD, because the design balances them at the cutoff. Adding them can improve precision, as in an experiment. They must be predetermined, measured before the running variable was realized. When the running variable is discrete, such as a test score with few possible values, or when students are nested in schools, standard errors should account for that structure.

          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Imbens, G. W., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), 615–635. https://doi.org/10.1016/j.jeconom.2007.05.001A practical guide to local linear RD.
          • Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2019). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press. https://doi.org/10.1017/9781108684606A short modern textbook with code.
          • Gelman, A., & Imbens, G. (2019). Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business & Economic Statistics, 37(3), 447–456. https://doi.org/10.1080/07350015.2017.1366909Why high-order polynomials mislead.
          • Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326. https://doi.org/10.3982/ECTA11757Data-driven bandwidths and robust inference.
          • Hahn, J., Todd, P., & Van der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1), 201–209. https://doi.org/10.1111/1468-0262.00183Identification in sharp and fuzzy RD.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 4 · Advanced

          Validity checks and fuzzy designs.

          An RD design is only as good as its claim that students on either side of the cutoff are comparable. This lesson covers the checks that support or undermine that claim, and the fuzzy design, in which the cutoff changes the chance of treatment rather than deciding it.

          Before thisIntermediate: local linear estimation, and Instrumental variables through the Advanced levelBandwidths and kernels, plus compliers, LATE, and 2SLS for the fuzzy design.
          After thisExpert: robust bias-corrected inferenceConfidence intervals that account for bias.
          By the end

          What you will be able to do

          1. Test for manipulation with a density check at the cutoff.
          2. Check that predetermined covariates do not jump at the cutoff.
          3. Run placebo tests at false cutoffs.
          4. Explain why a close contest can fail to be as good as random and test for sorting with a covariate at the cutoff.
          5. Estimate a fuzzy RD as a ratio of two jumps and interpret it as a local complier effect.
          6. Recognize settings with multiple cutoffs or discrete running variables.
          Check 1

          Manipulation and the density test

          If students can precisely control their running variable, the ones who end up just past the cutoff may differ from those who do not. A tell-tale sign is bunching. If scores just above a passing line are more common than scores just below it, someone may be pushing scores over the line. McCrary (2008) proposed testing whether the density of the running variable is continuous at the cutoff, and newer local polynomial density tests refine the idea (Cattaneo et al., 2019).

          The test detects one-directional manipulation. It can miss manipulation that moves some students up and others down by equal amounts, so it supports but cannot establish validity. Institutional knowledge matters too. Who sets the score, can it be retaken, and do graders see the cutoff?

          cutoffbelowaboveRunning variablemissing mass below, pile-up above
          What bunching looks like. Simulated counts of the running variable. A smooth distribution should cross the cutoff without a break. Here the bar just below the line is thin and the bar just above is thick, the signature of scores being pushed over the line. The density test asks whether a break like this is larger than chance.
          Check 2

          Covariate balance at the cutoff

          Characteristics fixed before the running variable was measured, such as last year’s test score, demographics, or prior attendance, should not jump at the cutoff. Run the same RD estimator with each covariate as the outcome. A jump suggests that students on the two sides differ, perhaps through manipulation or another policy using the same cutoff. As in experiments, with many covariates some will show jumps by chance, so look for patterns rather than single results (Lee & Lemieux, 2010).

          Check 3

          Placebo cutoffs

          Estimate the RD at points where nothing happens, such as 10 or 20 points above or below the real cutoff, using data from one side only so the true jump does not contaminate the placebo. Sizable jumps at false cutoffs suggest that ordinary variation in the outcome is as large as the jump at the real cutoff, so the real jump may not be distinguishable from noise.

          Worked example · simulated data

          Running the checks

          The code simulates 4,000 students. The density check compares students within 2 points above the cutoff with those within 2 points below. With honest scores the z statistic is −0.75, consistent with no bunching. If 40 percent of students just below the cutoff could retake the test and land just above, the same check gives 5.66, a clear warning.

          Last year’s score shows a jump of 1.17 at the cutoff with a standard error of 0.91, consistent with balance. Placebo cutoffs at −20, −10, 10, and 20 give jumps between −0.79 and 0.32, none more than 1.2 standard errors from zero (standard errors are about 0.65).

          The second script concerns the section on sorting. It simulates a scholarship cutoff with and without rescoring just below the cutoff and compares covariate checks.

          Run it yourself in R: code and output
          Lessons from elections

          When near the cutoff is not as good as random

          Lee (2008) argued that when candidates cannot control the vote share precisely, winning or losing a close election is as good as random, and so candidates just above and just below 50 percent should look alike on everything fixed beforehand. The condition is testable, and later work tested it. Caughey and Sekhon (2011) studied U.S. House elections from 1942 to 2008 and found that bare winners and bare losers differed on characteristics fixed before the election, including campaign money, previous experience, and ratings by experts. The imbalance grew in the closest races, and almost three-quarters of races decided by less than half a percentage point went to the party that had won the district before. Earlier analyses had missed this because they relied on a model that extrapolated from races far from the cutoff.

          The lesson for education applies to any score. A cutoff for a gifted program, a scholarship, or remediation is only as good as the students’ inability to control which side they land on. Retesting, rescoring after appeals, and teachers who nudge a borderline score upward are forms of sorting, and those who sort tend to differ from those who do not.

          The second script in the code panel under the worked example simulates a scholarship for scores of 60 and above, where prior achievement raises the score and the true effect of the scholarship on the outcome is 3. In one world scores are honest. In the other, 60 percent of students with above-average prior achievement who score between 57 and 60 are rescored to 60.5. Each world is run for 100 samples of 6,000 students. With honest scores the jump in prior achievement at the cutoff averages 0.00 with a quadratic fit on all students and 0.00 with a local linear fit within 4 points, and the test flags a jump in 8 and 4 percent of samples. The effect on the outcome averages 3.05. With rescoring the jump averages 0.45 and 0.67 in units of the standard deviation of prior achievement, and both tests flag it in all 100 samples. The estimated effect averages 5.92, nearly twice the truth, because the students above the cutoff now include some of the most prepared students from below it.

          Both checks caught the problem in this design, which has a large sample and strong sorting. Weaker sorting or a smaller sample would be harder to see, and a failed balance test is a reason to ask how the score was produced before trying another specification. A balanced covariate is reassuring, not proof, since the sorting may concern something unmeasured. Caughey and Sekhon (2011) note that matching cannot repair the problem, and the testable implications of the design remain its main protection.

          Method

          Fuzzy regression discontinuity

          Often the cutoff makes treatment more likely without guaranteeing it. Some students below the summer school cutoff receive waivers, and some above it enroll voluntarily. Then the jump in outcomes understates the effect, because not everyone switches treatment at the cutoff. Hahn, Todd, and Van der Klaauw (2001) showed that dividing the jump in outcomes by the jump in treatment probability identifies an effect,

          \[ \tau_{F} = \frac{\text{jump in } E[Y \mid R] \text{ at } 0}{\text{jump in } \Pr(D = 1 \mid R) \text{ at } 0}. \]

          This is instrumental variables at the cutoff, with being below the cutoff as the instrument. Under monotonicity, it is the average effect for compliers at the cutoff, students whose treatment is decided by which side they fall on. In practice it is estimated with local linear 2SLS, using the same bandwidth and kernel for both jumps. The interpretation rests on the same conditions as any instrument, relevance (a real jump in treatment), exclusion (crossing the cutoff affects outcomes only through treatment), and monotonicity (no one is pushed away from treatment by being below the cutoff), together with continuity at the cutoff. Module 3, Instrumental variables, develops these conditions and the complier interpretation in full, at the Foundations and Advanced levels.

          In the code, the cutoff raises the probability of summer school by 0.738, and outcomes jump by 2.32. The ratio is 3.14, close to the true 3. The sharp comparison of outcomes alone, 2.32, would understate the effect by about a quarter.

          Interpretation

          What an RD estimate averages over

          Lee and Lemieux (2010) give the local randomization intuition a precise form. If each student’s score has a random component they cannot control, every student has some chance of landing near the cutoff. The RD estimate is then a weighted average of individual effects, where each student’s weight is their probability, before the test, of scoring near the cutoff. Students whose expected score is far from the cutoff contribute little. This is why the estimate is local even when the effect is computed from many students.

          The same view explains the validity checks. If scores are imprecisely controlled, predetermined characteristics should be balanced near the cutoff and the density of scores should be smooth across it (McCrary, 2008). A pile-up of students just above the cutoff signals that some had precise control, and the comparison is no longer like an experiment. Lee and Lemieux also caution against treating any one estimator as the answer. Local linear regression and global polynomials should be reported together, across bandwidths, and graphs should be drawn so that they neither invent nor hide a jump.

          Case · published study

          Maimonides’ rule and class size

          Israeli public schools have followed a rule, attributed to the medieval scholar Maimonides, that caps class size at 40. A grade with 40 students has one class, while a grade with 41 must be split into two classes of about 20.5 on average. At 81 students the grade splits into three classes of 27. Angrist and Lavy (1999) used these enrollment cutoffs to study class size in fourth and fifth grade. The rule’s predicted class size is an instrument for actual class size, so the design is a fuzzy RD with several cutoffs.

          Schools did not follow the rule exactly, but closely enough. Predicted class size explained about half of the variation in actual class size, and a one-student increase in predicted size raised actual size by 0.54 to 0.77 students, depending on grade and controls. The median class had 31 pupils.

          EstimateCoefficient on class size
          OLS, class size only0.221
          OLS, controlling for percent disadvantaged−0.031 (0.022)
          2SLS, no enrollment controls−0.16 (0.04)
          2SLS, linear or quadratic enrollment controls−0.26 (0.08) to −0.28 (0.07)
          2SLS, ±3 discontinuity sample−0.45 (0.24)

          Test-score points per student of class size, fifth grade, 1991. Standard errors in parentheses. All 2SLS models control for percent disadvantaged. Source is Angrist and Lavy (1999).

          The table shows why the design matters. Larger classes in Israel were found in larger, often more advantaged schools, so the raw correlation between class size and reading scores is positive. Controlling for the percent of disadvantaged students removes most of it. Using the rule as an instrument reverses the sign. For fifth-grade math, the IV estimate is close to zero without enrollment controls and about −0.23 with a linear control. The authors translate the reading estimate of −0.275 into a policy scale. Cutting class size by 8 students, about the reduction in the Tennessee STAR experiment, would raise average reading scores by about 2.2 points, roughly 0.29 standard deviations of class averages.

          Effects were smaller for fourth-grade reading, around −0.11 to −0.13, and not significant for fourth-grade math. A third-grade cohort tested in 1992, under changed testing conditions, showed no effects. Effects were larger in schools with more disadvantaged students. The authors argue that manipulation was limited because Israeli children attend neighborhood schools and the rule was applied to enrollment counted in September. The case shows that an RD estimate is tied to the grade, place, and margin where the cutoff falls.

          Case · published study

          Birthdays, school entry, and education

          School entry rules create cutoffs in exact date of birth. In California a child had to turn five by December 1 to start kindergarten, and in Texas by September 1. Two girls born a day apart, on either side of the date, start school a year apart. McCrary and Royer (2011) used birth records from both states, which record the mother’s exact date of birth, to compare women born just before and just after the entry date and to estimate effects of their education on fertility and infant health.

          The design is fuzzy. Some parents hold children born before the date back a year, and some start children born after it early or in private school, so the cutoff shifts schooling for compliers only. The authors note that compliance is more common in lower-income families. Among mothers aged 23 or younger, being born just after the date reduced education at motherhood by about 0.24 years in Texas and 0.14 years in California. If the rule shifts schooling by at most one year, that corresponds to roughly one in four young mothers in Texas and one in seven in California having a year less schooling. Women born on either side were similar on predetermined characteristics, as the design requires.

          Women born just after the date were no less likely to become mothers and gave birth at similar ages, and their infants’ health was similar, so education effects on these outcomes were generally small. Two features limit the interpretation, and the authors discuss both. The estimates describe women whose schooling the rule changed, mainly young women at risk of leaving school. And entry rules change age relative to classmates as well as education, so the exclusion restriction requires that being older than one’s peers matters only through schooling. Women born after the date had younger and less educated partners, which the authors attribute partly to their younger peer groups.

          Extensions

          Other complications

          • Multiple cutoffs.Policies like Maimonides’ rule create several cutoffs. Estimates can be pooled after normalizing the running variable, or reported separately.
          • Discrete running variables.Test scores with few distinct values limit how close to the cutoff one can get. Inference should reflect the discreteness.
          • Several running variables.Some programs use two scores, such as reading and mathematics. The cutoff becomes a boundary, and effects may differ along it.
          • Compound treatments.When several policies share one cutoff, the RD estimates their combined effect.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698–714. https://doi.org/10.1016/j.jeconom.2007.05.005The density test for manipulation.
          • Hahn, J., Todd, P., & Van der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1), 201–209. https://doi.org/10.1111/1468-0262.00183Identification of fuzzy RD.
          • Angrist, J. D., & Lavy, V. (1999). Using Maimonides' rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 114(2), 533–575. https://doi.org/10.1162/003355399556061Maimonides’ rule and class size.
          • Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355. https://doi.org/10.1257/jel.48.2.281Validity checks and their logic.
          • Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2019). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press. https://doi.org/10.1017/9781108684606Modern implementation of RD checks.
          • Cattaneo, M. D., & Titiunik, R. (2022). Regression discontinuity designs. Annual Review of Economics, 14, 821–851. https://doi.org/10.1146/annurev-economics-051520-021409A review of the continuity and local randomization frameworks, with validation methods.
          • McCrary, J., & Royer, H. (2011). The effect of female education on fertility and infant health: Evidence from school entry policies using exact date of birth. American Economic Review, 101(1), 158–195. https://doi.org/10.1257/aer.101.1.158A fuzzy RD on exact date of birth, with careful discussion of who the compliers are.
          • Lee, D. S. (2008). Randomized experiments from non-random selection in U.S. House elections. Journal of Econometrics, 142(2), 675–697. https://doi.org/10.1016/j.jeconom.2007.05.004The close-election design and the condition under which near the cutoff is as good as random.
          • Caughey, D., & Sekhon, J. S. (2011). Elections and the regression discontinuity design: Lessons from close U.S. House races, 1942–2008. Political Analysis, 19(4), 385–408. https://doi.org/10.1093/pan/mpr032Evidence that close House elections are not balanced, and the lessons for any score-based design.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 4 · Expert

          Robust bias-corrected inference.

          Choosing the bandwidth that minimizes mean squared error makes the estimate’s bias and standard deviation similar in size. That is good for the point estimate and bad for the confidence interval, which assumes bias is negligible. This lesson explains the problem, the robust bias-corrected intervals that fix it, and the local randomization approach as an alternative.

          Before thisAdvanced: validity checks and fuzzy designsTesting RD assumptions.
          Next moduleDifference-in-differencesComparing changes over time for groups that did and did not adopt a program.
          By the end

          What you will be able to do

          1. Explain why conventional intervals under-cover at MSE-optimal bandwidths.
          2. Describe robust bias-corrected inference and why it widens the interval.
          3. Implement a simple version with a local quadratic fit.
          4. Describe the local randomization framework and when it suits education data.
          5. Read RD output from standard software and report it correctly.
          Theory

          The coverage problem

          A local linear estimator has a bias that grows with the square of the bandwidth and a variance that shrinks as the bandwidth grows. The MSE-optimal bandwidth balances them, so at that bandwidth the bias is of the same order as the standard error. A conventional interval, estimate plus or minus 1.96 standard errors, is centered on a biased estimate, and it covers the truth less often than 95 percent.

          Undersmoothing, using a smaller bandwidth than optimal, makes the bias negligible relative to the standard error, but there is no clear rule for how much smaller, and the cost is a noisier estimate.

          Method

          Robust bias correction

          Calonico, Cattaneo, and Titiunik (2014) estimate the bias with a higher-order local polynomial, subtract it from the estimate, and then, crucially, compute a standard error that includes the extra variability introduced by estimating the bias. Earlier bias-corrected methods ignored that extra variability and still under-covered.

          The resulting interval is centered on the bias-corrected estimate and is wider than the conventional one. Standard practice, implemented in rdrobust, reports the conventional point estimate at the MSE-optimal bandwidth together with the robust bias-corrected confidence interval (Cattaneo et al., 2019). The point estimate may then lie off-center in its interval, which is expected.

          When the bandwidth used to estimate the bias equals the main bandwidth, the robust bias-corrected procedure amounts to fitting a local quadratic with that bandwidth and using its standard error. That special case gives a simple way to see the idea in code.

          Calonico, Cattaneo, and Titiunik (2014) show that the robust interval remains valid at MSE-optimal bandwidths, that its coverage error shrinks faster as the sample grows, and that in large samples it can be shorter than a conventional interval with correct coverage, although in small samples it may be longer. The same intervals can be applied to predetermined covariates as outcomes, which makes them the natural tool for balance tests.

          Monte Carlo · simulated data

          Coverage by simulation

          The code draws 300 samples of 2,000 students with a true effect of 3 and curvature that differs on the two sides of the cutoff. With a bandwidth of 20, the conventional local linear estimate averages 3.66, and its 95 percent interval covers the truth in only 78.3 percent of samples. The local quadratic version averages 3.00 and covers 95.7 percent of the time. The price is width. Its intervals average 3.17 points wide, against 2.17 for the conventional intervals.

          The conventional intervals are narrower because they ignore bias, not because they are more informative. An interval that misses the truth about one time in five is not a 95 percent interval.

          Run it yourself in R: code and output
          Alternative

          The local randomization approach

          A different framework treats a small window around the cutoff as if it were a randomized experiment. Within the window, which side a student landed on is assumed to be as good as random, and the outcome is assumed not to depend on the running variable. The analysis then uses tools from experiments, such as a difference in means and randomization inference (Cattaneo et al., 2019).

          This approach suits running variables with few distinct values, such as test scores reported in whole points, where the continuity-based methods struggle. Its assumption is demanding, so the window is usually narrow and is chosen by checking that predetermined covariates are balanced within it.

          Case · published study

          Teacher evaluation in Washington, DC

          The IMPACT evaluation system in the District of Columbia Public Schools gave each teacher a score built from classroom observations and other measures. Teachers scoring 250 to 349 were rated effective. Those just below 250 were rated minimally effective, told they would be dismissed if not rated effective the following year, and denied a step increase. Those at 350 or above were rated highly effective, earned one-time bonuses of up to $25,000, and qualified for a permanent base-pay increase after a second consecutive highly effective rating. Dee and Wyckoff (2015) compared teachers just above and just below each threshold.

          In IMPACT’s first year the effects were small and insignificant. In its second year, when the policy was more credible, a minimally effective rating raised voluntary attrition by about 11 percentage points. Since roughly 20 percent of teachers just above the threshold left anyway, this is an increase of more than 50 percent. Teachers who received the rating and stayed improved their next IMPACT score by about 12.6 points, 0.27 of a teacher-level standard deviation. At the upper threshold, the prospect of a permanent raise improved performance by about 10.9 points, or 0.24.

          Two features make this a useful Expert case. The running variable is a score that evaluators assign, so manipulation is a real risk, and the authors report density tests (McCrary, 2008) that show no bunching at either threshold. And the performance effect is measured only for teachers who stayed, because the rating itself changed who stayed. The authors present it as the performance of teachers who chose to remain, which is the appropriate reading of an outcome observed only after selection.

          Synthesis

          Continuity and local randomization compared

          Cattaneo and Titiunik (2022) organize the RD literature around the two frameworks met in this lesson. The continuity framework assumes that average potential outcomes change smoothly at the cutoff and estimates the jump with local polynomials, a bandwidth, and robust bias-corrected inference. The local randomization framework assumes that, within a window around the cutoff, the assignment mechanism is known and the potential outcomes do not depend on the score. It then analyzes the window as an experiment.

          In the second framework, choosing the window plays the role that choosing the bandwidth plays in the first. Cattaneo and Titiunik describe a data-driven rule. Starting from a wide window, test whether predetermined covariates are balanced, and shrink the window until balance cannot be rejected in it or in any smaller window inside it. The assumption that potential outcomes do not depend on the score inside the window is stronger than continuity, because the score itself is often related to outcomes, so the window is usually narrow.

          Their review also marks the design’s main limitation. Without further assumptions, RD says nothing about effects away from the cutoff. Methods that extrapolate use a pre-intervention measure of the outcome, covariates under a conditional independence assumption, or multiple cutoffs, and each brings its own assumption. Their summary of good practice is that tuning choices, the bandwidth and the window, should be objective and data driven so that specification searching is limited.

          Reporting

          What to report

          1. The RD graph with binned means and fitted lines.
          2. The point estimate at the MSE-optimal bandwidth, with the bandwidth and kernel.
          3. The robust bias-corrected confidence interval.
          4. Estimates at other bandwidths, such as half and twice the optimal one.
          5. The density test, covariate balance, and placebo cutoffs.
          6. For fuzzy designs, the first-stage jump and its strength.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326. https://doi.org/10.3982/ECTA11757Robust bias-corrected confidence intervals.
          • Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2019). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press. https://doi.org/10.1017/9781108684606Continuity-based and local randomization approaches in practice.
          • Dee, T. S., & Wyckoff, J. (2015). Incentives, selection, and teacher performance: Evidence from IMPACT. Journal of Policy Analysis and Management, 34(2), 267–297. https://doi.org/10.1002/pam.21818RD with teacher evaluation thresholds.
          • Imbens, G. W., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), 615–635. https://doi.org/10.1016/j.jeconom.2007.05.001The foundations of local linear RD.
          • Cattaneo, M. D., & Titiunik, R. (2022). Regression discontinuity designs. Annual Review of Economics, 14, 821–851. https://doi.org/10.1146/annurev-economics-051520-021409A review organized around the continuity and local randomization frameworks.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 6 · Foundations

          Building a comparison for one treated unit.

          Some policies happen in only one place. A single district adopts a new reading program, one state changes its graduation rules, one city launches a scholarship guarantee. There is no group of treated units to average over, and no single untreated unit looks quite like the treated one. Synthetic control builds a comparison by blending several untreated units into one that tracks the treated unit before the policy.

          Before thisDifference-in-differences, FoundationsComparing changes over time.
          After thisIntermediate: estimation and placebo inferenceHow the weights are chosen and how to judge uncertainty.
          By the end

          What you will be able to do

          1. Explain why a single treated unit makes ordinary comparisons hard.
          2. Describe a synthetic control as a weighted average of untreated units.
          3. Read a synthetic control graph and judge its pre-period fit.
          4. State the main assumptions in plain words.
          5. Recognize the method in a published study.
          The problem

          One district, many possible comparisons

          Suppose one district adopts a new reading program in year 13 of a long data series, and 20 similar districts in the state do not. To estimate the program’s effect we need to know what the adopting district’s scores would have been without it. Which comparison should we use?

          The average of the 20 districts is one option, but those districts differ from the adopting district in level and in how fast their scores have been rising. Picking the single most similar district is another option, but no one district matches well, and the choice is easy to make in a way that favors a desired answer. Difference-in-differences against the average removes the gap in levels, but it assumes the adopting district’s scores would have risen at the average district’s pace, which may not be true.

          The idea

          A weighted blend of districts

          Synthetic control, introduced by Abadie and Gardeazabal (2003) and developed by Abadie, Diamond, and Hainmueller (2010), chooses a weight for each untreated district. The weights are zero or positive and add up to one, so the result is a weighted average. They are chosen so that the weighted average follows the adopting district’s scores as closely as possible in the years before adoption. That weighted average is the synthetic control, and its path after adoption is the estimate of what would have happened without the program.

          The estimated effect in each year after adoption is the gap between the adopting district and its synthetic control. If the two lines move together for 12 years and then separate when the program starts, the separation suggests an effect.

          The untreated districts are called the donor pool, because they donate pieces of the comparison. Usually only a few donors receive positive weight, which makes the comparison easy to inspect. A reader can see exactly which districts the comparison is built from and ask whether they are sensible.

          45556575Year 1481216program startsgap = estimated effectAdopting district (solid), synthetic control (dashed), donors (gray)
          The method in one picture. Simulated data. No single gray donor follows the adopting district, but a weighted blend of them, the dashed line, tracks it closely for 12 years. When the program starts the lines separate, and the gap in each later year is the estimated effect for that year.
          Assumptions

          What has to be true

          • A good match is possible. The adopting district must look like some blend of donors before adoption. If its scores are higher, or rising faster, than every donor’s, no weighted average can match them.
          • The match carries forward. A blend that tracked the district for many years should keep tracking it without the program. Long pre-periods make this more believable, because a short run of years is easy to match by chance.
          • Donors are untouched. No donor adopts a similar program or suffers a large shock of its own after year 12, and the program does not spill over to the donors.
          • No anticipation. The district did not change its behavior before the program officially started.

          In the vocabulary of Module 1, the estimand here is the effect on the one district that adopted the program, in each year after adoption. The assumptions above are what allow the data to identify it.

          Worked example · simulated data

          A reading program in one district

          The code simulates 17 years of average reading scores for one adopting district and 20 donors. Scores share a statewide upward trend and a cycle, but each district follows them to a different degree. The program truly raises scores by 3 points in each of the 5 years after adoption.

          Comparing the adopting district with the average donor after adoption gives 1.10, because the adopting district started lower. Difference-in-differences gives 4.42, because the adopting district’s scores were rising faster than the average donor’s. Synthetic control puts weight on six districts, led by district 2 at 0.49 and district 17 at 0.21. Its pre-period fit is close, with a root mean squared error of 0.53 points, and its estimate is 2.99, against a true effect of 3. The year-by-year gaps hover around zero for 12 years and then jump to between 2.3 and 3.6.

          Run it yourself in R: code and output
          The picture

          Reading a synthetic control graph

          Every synthetic control study shows two lines over time, the treated unit and its synthetic control, with a vertical line at adoption. Read it in two steps. First, look at the years before adoption. If the lines do not track each other closely, the comparison is weak and the later gap means little. Second, look at the years after. A gap that opens at adoption and persists suggests an effect. A gap that was already opening before adoption suggests anticipation or a poor match.

          Many studies also plot the gap itself. That graph should hover near zero before adoption and move away from zero after it.

          History

          Terrorism and the Basque economy

          The method was introduced to answer a question with one treated unit. Terrorist activity in the Basque Country began in the late 1960s. Before then the region was among the richest in Spain, third in per-capita GDP among its 17 regions. Abadie and Gardeazabal (2003) asked how the Basque economy would have grown without the conflict.

          No single Spanish region resembled the Basque Country closely enough to serve as a comparison. A weighted combination of two did. The synthetic Basque Country was about 85 percent Catalonia and 15 percent Madrid, weights chosen so that the combination matched the Basque Country’s pre-1970 per-capita GDP and its predictors of growth, such as its investment ratio, sectoral shares, and the schooling of its population. After terrorism began, actual per-capita GDP fell about 10 percentage points below the synthetic region.

          The authors added a second, independent piece of evidence. During the 1998 to 1999 ceasefire, stocks of firms with much of their business in the Basque Country performed better than other stocks as the truce became credible, and worse when it ended. Two different designs pointing the same way made the conclusion more persuasive than either alone.

          Case · published study

          Tobacco control in California

          In 1988 California passed Proposition 99, a large tobacco control program that raised cigarette taxes. Abadie, Diamond, and Hainmueller (2010) built a synthetic California from other states that had not adopted large tobacco programs, chosen to match California’s cigarette sales and related predictors before 1988. Synthetic California was a weighted average of five states, Utah (0.33), Nevada (0.23), Montana (0.20), Colorado (0.16), and Connecticut (0.07). The other 33 states in the donor pool received zero weight. The two lines track closely before 1988 and then separate. The authors estimate that by 2000, annual per-capita cigarette sales in California were about 26 packs lower than they would have been without the program.

          In education, Bifulco, Rubenstein, and Sohn (2017) used synthetic control to evaluate Say Yes to Education in Syracuse, New York, a district-wide program of college scholarships and student supports begun in 2008. With one district treated, there was no group of treated districts to average, which is exactly the situation the method was built for.

          Cautions

          Common misreadings

          • “The lines match before adoption, so the estimate is causal.”A good fit is necessary but not sufficient. A shock that hits only the treated unit at adoption would also open a gap.
          • “The weights show which districts are most similar.”The weights produce a blend that matches. A district with high weight need not resemble the treated unit on its own.
          • “One treated unit means no uncertainty.”The estimate still contains noise. The Intermediate lesson shows how placebo comparisons gauge it.
          • “Synthetic control works with any donor pool.”If the treated unit lies outside the range of the donors, the method cannot match it. The Advanced lesson shows what goes wrong.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746The tobacco control study. Start here.
          • Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the Basque Country. American Economic Review, 93(1), 113–132. https://doi.org/10.1257/000282803321455188The first synthetic control study.
          • Bifulco, R., Rubenstein, R., & Sohn, H. (2017). Using synthetic controls to evaluate the effect of unique interventions: The case of Say Yes to Education. Evaluation Review, 41(6), 593–619. https://doi.org/10.1177/0193841x17742233An education application to a district-wide program.
          • Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. https://doi.org/10.12987/9780300255881An accessible textbook chapter with code.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 6 · Intermediate

          Estimation and placebo inference.

          This lesson states the synthetic control estimator formally, shows how it relates to difference-in-differences, and explains how to judge whether an estimated gap is larger than chance. With one treated unit, the usual standard errors do not apply. Instead, researchers ask how unusual the treated unit’s gap is compared with gaps produced by pretending other units, or other dates, were treated.

          Before thisFoundations: building a comparisonWeights, donors, and the graph.
          After thisAdvanced: feasibility, fit, and robustnessWhen the method can and cannot be trusted.
          By the end

          What you will be able to do

          1. Write the synthetic control estimator and its weight constraints.
          2. Explain how the weights are chosen from pre-period outcomes and predictors.
          3. Compare synthetic control with difference-in-differences.
          4. Run in-space and in-time placebo tests and compute a permutation p-value.
          5. Explain why few pre-period years invite overfitting.
          Method

          The estimator

          Label the treated unit 1 and the donors \(j = 2, \dots, J+1\). The treated unit adopts after period \(T_0\). Synthetic control chooses weights \(w_j \ge 0\) with \(\sum_j w_j = 1\) and estimates the effect in a post-adoption period \(t\) as

          \[ \hat\tau_t = Y_{1t} - \sum_{j=2}^{J+1} w_j Y_{jt}. \]

          Abadie, Diamond, and Hainmueller (2010) choose the weights to make the synthetic unit resemble the treated unit on a set of pre-adoption characteristics, such as averages of the outcome and covariates, with an importance weight for each characteristic chosen to minimize the pre-period prediction error for the outcome. A common simpler choice, used in this module’s code, matches the outcome in every pre-period year. The weights then solve a least-squares problem subject to the two constraints.

          The constraints matter. Because the weights cannot be negative and must sum to one, the synthetic unit cannot extrapolate beyond the donors, and the solution usually puts weight on only a few of them. Abadie (2021) argues that this sparsity and transparency are among the method’s main advantages.

          Connections

          How it relates to difference-in-differences

          Difference-in-differences with one treated unit also compares it with a weighted average of donors, but with equal weights, and it allows a constant gap in levels by subtracting the pre-period difference. Synthetic control instead chooses unequal weights and allows no level gap. DiD relies on parallel trends with the average donor. Synthetic control relies on the chosen blend tracking the treated unit, which can hold when trends differ across donors, as in the Foundations example.

          Synthetic difference-in-differences (Arkhangelsky et al., 2021) combines the two, with unit weights like synthetic control and a level shift like DiD. The Expert lesson on few treated units in the difference-in-differences module works through it.

          Inference

          Placebos in space

          Abadie et al. (2010) propose a permutation approach. Apply the method to each donor in turn, as if it had adopted the program, using the remaining donors as its donor pool. Each placebo produces a gap even though nothing happened, and the collection of placebo gaps shows how large a gap chance alone produces.

          Raw gaps are hard to compare because some units are matched poorly before adoption, and those units produce large gaps afterward for no reason. The standard fix compares the ratio of the root mean squared prediction error (RMSPE) after adoption to the RMSPE before. A large ratio means the gap after adoption is large relative to how well the unit was matched before. If the treated unit’s ratio is the largest of \(J+1\) units, the permutation p-value is \(1/(J+1)\). Some studies also report results after dropping placebos whose pre-period fit is much worse than the treated unit’s.

          -8-4048Year 1481216adoptiontreated unitGap between each unit and its synthetic control
          The permutation test as a picture. Simulated gaps. Each gray line treats one donor as if it had adopted the program, and those placebo gaps hover around zero in every year. The treated unit’s gap, in teal, moves with the placebos before adoption and then leaves the band. An effect is credible when the treated gap stands outside what the placebos produce by chance.
          Inference

          Placebos in time

          An in-time placebo, or backdating test, pretends adoption happened earlier. Fit the weights on the years before the fake date, then check the gap between the fake date and the real one. Because nothing happened in those years, the gap should be near zero. A large gap suggests the synthetic control does not predict well out of sample.

          Worked example · simulated data

          Placebos for the reading program

          The code uses the same data as the Foundations lesson. The adopting district’s estimate is 2.99, with a pre-period RMSPE of 0.53, a post-period RMSPE of 3.04, and a ratio of 5.73. The 20 placebo estimates range from −2.22 to 2.60, so the adopting district’s gap of about 3 is larger than every placebo gap, though not by a wide margin. The adopting district’s ratio is the largest of the 21, giving a permutation p-value of 1/21 = 0.048. Keeping only the 16 placebos whose pre-period RMSPE is at most twice the adopting district’s gives 1/17 = 0.059.

          The in-time placebos show why the length of the pre-period matters. Fitting on years 1 to 8 and predicting years 9 to 12 gives an average gap of 0.33, close to zero. Fitting on only years 1 to 6 gives 2.21. With 20 donors and only 6 years to match, the weights fit noise as well as signal, and the synthetic control predicts poorly afterward.

          Run it yourself in R: code and output
          Case · published study

          Inference for Proposition 99

          Abadie et al. (2010) ran placebo tests for every state in their donor pool. California’s ratio of post- to pre-period prediction error was the largest among the 39 states, so the probability of a ratio that large under a random assignment of the program is 1/39, about 0.026. They also showed the placebo gaps graphically, with California’s line standing apart after 1988, a display that has since become standard.

          Note what this p-value means. It treats the choice of which state adopted as if it were random and asks how unusual California’s result would be under that assumption. It is not a statement about sampling from a population of states, and its smallest possible value depends on the number of donors. Placebo states that a synthetic control cannot fit well before 1988 will show large gaps afterward for that reason alone. Before turning to the ratio, the authors showed the placebo comparison again after dropping states whose pre-1988 prediction error was more than 20, 5, and 2 times California’s. California’s post-1988 error was about 130 times its pre-1988 error, and no other state came close.

          Practice

          What to report

          1. The donor pool and the reason each excluded unit was dropped.
          2. The predictors and pre-period years used to choose the weights.
          3. The weights, and a table comparing the treated unit with its synthetic control on the predictors.
          4. The graph of both lines, and the graph of placebo gaps.
          5. The permutation p-value, with and without poorly fitted placebos.
          6. An in-time placebo.
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. https://doi.org/10.1198/jasa.2009.ap08746The estimator and placebo inference.
          • Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510. https://doi.org/10.1111/ajps.12116In-time placebos and leave-one-out checks, applied to German reunification.
          • Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425. https://doi.org/10.1257/jel.20191450A practical guide to using the method well.
          • Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118. https://doi.org/10.1257/aer.20190159Synthetic difference-in-differences.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 6 · Advanced

          Feasibility, fit, and robustness.

          Synthetic control is transparent, but it is not automatic. It works when the treated unit can be matched by a blend of donors over a long pre-period, when the donors are untouched by the policy and by shocks of their own, and when the specification is fixed before the results are seen. This lesson covers how to check each condition, and what goes wrong when the treated unit lies outside the range of its donors.

          Before thisIntermediate: estimation and placebo inferenceThe estimator and permutation p-values.
          After thisExpert: augmented synthetic controlCorrecting for imperfect fit, and modern inference.
          By the end

          What you will be able to do

          1. Judge whether a setting suits synthetic control before estimating.
          2. Diagnose a poor pre-period fit and explain the bias it signals.
          3. Run leave-one-out checks on the donor pool.
          4. Explain the demeaned estimator and when it helps.
          5. Recognize specification searching and how to guard against it.
          6. Validate the weights on a held-out part of the pre-period without using post-period data.
          Before estimating

          When the method fits the setting

          Abadie (2021) lists conditions to consider before using synthetic control. Most can be judged before any estimate is computed.

          • Effect size and volatility. If the outcome swings widely from year to year, a moderate effect will be hard to distinguish from noise in a single unit.
          • A suitable donor pool. Donors should resemble the treated unit and should not have adopted similar policies or experienced large idiosyncratic shocks during the study period.
          • No anticipation and no interference. If the policy was announced early, the adoption date should be moved back. If it affects neighboring units, those units should leave the donor pool.
          • Convex hull. The treated unit's pre-period characteristics should lie within the range a blend of donors can reach.
          • A long pre-period. A close fit over many periods is more convincing than one over a few, because a short series is easy to match by chance.

          In education, these conditions often favor district- or state-level outcomes measured for many years, such as graduation rates or average test scores, and they warn against settings in which nearby districts adopted similar reforms at the same time.

          Diagnostics

          Pre-period fit and the convex hull

          Abadie et al. (2010) show that when a close pre-period fit is achieved over a long pre-period, the bias of synthetic control is small under a factor model, in which units respond differently to common shocks. The argument depends on the fit. When the treated unit lies outside the convex hull of the donors, for example because its scores rise faster than every donor's, no weighted average can track it. The best fit is then systematically off, and the estimate inherits that error.

          Abadie (2021) recommends against using the method when the fit is poor. Ferman and Pinto (2021) show that when the fit is imperfect, the method can remain biased however many pre-periods are available, if adoption is related to the unobserved common factors that drive outcomes. They suggest a demeaned version, which subtracts each unit's pre-period mean before fitting the weights and so allows a constant gap in levels, as difference-in-differences does.

          Diagnostics

          Validate the weights before the post-period

          A close pre-period fit can be partly an artifact of fitting. With 20 donors and 12 years the weights have room to chase noise, so the in-sample fit overstates how well the synthetic control will track the district afterward. Abadie et al. (2010, 2015) borrowed a remedy from prediction, and Abadie (2021) describes it as a way to select predictors or to evaluate a given set. Divide the pre-period into an initial training period and a later validation period, compute the weights from the training years only, and judge the resulting synthetic control on the validation years. Because no post-period outcome enters, the check cannot be steered by the result.

          The second script in the code panel under the worked example applies it to the same simulated data, fitting on years 1 to 8 and judging on years 9 to 12. For the adopting district the root mean squared prediction error (RMSPE) is 0.45 in the training years and 0.75 in the held-out years, so the in-sample fit was somewhat flattering and the held-out figure is the fairer description of tracking. For the district that rises faster than every donor, the held-out RMSPE is 2.39, more than three times as large, and the estimate is 6.81 against a true effect of 3. The warning arrives before anyone looks at the post-period.

          The check is a screen and not a selector. A pool of five districts has a held-out RMSPE of 0.77, almost the same as the 0.75 of the full pool, yet it gives an estimate of 2.09 instead of 2.99. The level-adjusted version gives 3.00 with a held-out RMSPE of 0.78. Held-out error cannot separate specifications this close, so report the estimates from all of them, as the section on specification searching recommends, and not the one that happened to validate best. Abadie (2021) also cautions that weights chosen by cross-validation are not always unique.

          Comparison

          What regression would have done

          Abadie, Diamond, and Hainmueller (2015) studied the economic cost of German reunification in 1990 for West Germany, using 16 OECD countries as donors. They estimate that reunification lowered West German per-capita GDP by about $1,600 a year on average over 1990 to 2003, about 8% of the 1990 level.

          They also asked what a regression of the outcome on the same predictors would imply. Any linear regression counterfactual can be written as a weighted average of the donors, so its weights can be compared with the synthetic control's.

          CountrySynthetic control weightRegression weight
          Austria0.420.26
          United States0.220.13
          Japan0.160.19
          Switzerland0.110.05
          Netherlands0.090.14
          Greece0−0.09
          Portugal0−0.08
          Italy0−0.05
          Spain0−0.01

          Selected countries from the 16-country donor pool. Both sets of weights sum to one. Source is Abadie, Diamond, and Hainmueller (2015), Table 1.

          The two methods agree that Austria matters most, but they differ elsewhere. Regression gives negative weights to four countries. A negative weight means the counterfactual West Germany lies outside the range of the donors, so regression is extrapolating, and the extrapolation is hidden inside the coefficients. Synthetic control weights are restricted to be nonnegative and sum to one, so the counterfactual is an interpolation that a reader can inspect country by country.

          Their robustness checks follow the pattern in this lesson. A placebo reunification dated to 1975 produced no gap. Dropping each donor in turn left the estimates similar, with the smallest leave-one-out estimate coming when the United States was excluded. Even that version showed an average gap of about $630 a year, about 3% of the 1990 level. Reporting this range tells readers how much the conclusion depends on one comparison country.

          Diagnostics

          Average, nearest neighbor, or synthetic?

          Abadie (2021) uses the reunification example to compare three ways of building a comparison.

          PredictorWest GermanySynthetic West GermanyOECD averageAustria, nearest neighbor
          GDP per capita15,808.915,802.213,669.414,817.0
          Trade openness56.856.959.874.6
          Inflation rate2.63.57.63.5
          Industry share34.534.433.835.5
          Schooling55.555.238.760.9
          Investment rate27.027.025.926.6

          Averages for various periods in the 1980s. The OECD average is a simple mean of the 16 donor countries. Source is Abadie (2021), Table 1.

          The simple average of donors misses West Germany's income, inflation, and schooling, and the single most similar country, Austria, misses trade openness and schooling. The weighted combination matches nearly every predictor. Abadie also stresses that the weights are chosen without looking at outcomes after the intervention, so they can be fixed in the design phase, before results exist.

          He adds a backdating check. Moving the reunification date back to 1980 and refitting with data only up to then, the synthetic control still tracks West German GDP through the 1980s, and a gap still opens shortly after the true date of 1990, with similar shape and somewhat smaller size. A counterfactual that predicts well over a decade it was not fitted to, and that finds the effect without being told when it happened, is more credible. Abadie's data requirements complete the list in this lesson. The outcome must be measured for aggregate units, the pre-period must be long enough to judge the fit, and the post-period must be long enough for effects that arrive with a delay.

          Worked example · simulated data

          Leave-one-out and a district out of range

          The first part of the code repeats the Foundations analysis, which gives 2.99 with a pre-period RMSPE of 0.53, and then drops each donor that carries a weight of at least 0.05. The estimate ranges from 2.77 to 3.69. The largest change comes from dropping district 2, which carried weight 0.49. The fit barely worsens, rising to 0.62, so other donors can substitute for district 2, but the estimate moves by 0.7 points. Reporting this range shows readers how much the conclusion depends on a single donor.

          The second part adds a different adopting district whose scores rise faster than any donor's, with the same true effect of 3. Synthetic control can no longer match it. The pre-period RMSPE nearly triples to 1.45, and the estimate is 6.81. The demeaned version improves the level match but not the trend, with an RMSPE of 1.28 and an estimate of 5.62. Difference-in-differences against the average donor gives 9.13. None is credible here, and the poor fit is the warning sign. The Expert lesson shows a correction.

          The second script holds out the last four pre-period years. It fits the weights on years 1 to 8 and reports the error on years 9 to 12 for three specifications, which the section on validating the weights discusses.

          Run it yourself in R: code and output
          Research practice

          Specification searching

          Synthetic control leaves the researcher many choices, among them the donor pool, the predictors, which pre-period outcomes to match, and how many pre-period years to use. Ferman, Pinto, and Possebom (2020) show that trying several specifications and reporting a favorable one can substantially raise the probability of finding a significant effect when there is none. Their recommendations include reporting results across a range of reasonable specifications.

          Two practices help. First, fix the donor pool and predictors in advance, ideally in a pre-analysis plan, and explain each exclusion. Second, report the main estimate alongside alternatives, such as matching every pre-period outcome, matching only averages, and the demeaned estimator. Abadie (2021) also stresses that the design, not the outcome data after adoption, should guide these choices.

          Case · education

          A district-wide scholarship program

          Say Yes to Education began in Syracuse, New York, in 2008. It guarantees tuition at New York public colleges and many private ones for district graduates and funds extended-day programs, health services, and other supports. Bifulco, Rubenstein, and Sohn (2017) used synthetic control to estimate its effects on district enrollment and graduation rates, and used the case to show how much the method's choices matter.

          They varied two choices. The donor pool was either 22 similar city districts or a comprehensive pool of all city and suburban districts in New York, 275 for enrollment and 236 for graduation. The pre-treatment outcomes used to fit the weights followed six specifications, from only the first and last years to every year. Every synthetic Syracuse also matched the shares of students who were Black, Hispanic, and eligible for free lunch. Say Yes did not reach Buffalo until 2013, after the analysis period, so the donor pool was not contaminated by the program.

          EstimateRange across specificationsShare of 2007–08 enrollment
          Post-period difference672 to 1,166 students3.2% to 5.6%
          Difference-in-differences version885 to 1,171 students4.5% to 5.9%

          Ranges cover six choices of pre-treatment years and two donor pools. Source is Bifulco, Rubenstein, and Sohn (2017), Tables 3 and 4.

          For enrollment the answer was stable. With the restricted pool, Niagara Falls and Rochester, two majority non-White districts with declining enrollment, received at least 70% of the weight in every specification. Enrollment rose relative to synthetic Syracuse from the second year on, reversing decades of decline. Several of the placebo p-values were below 0.10.

          For graduation rates the answer was not stable. Pre-treatment graduation rates were volatile, the weights changed sharply across specifications, and the pre-period fit was poor. Third-year estimates with the comprehensive pool ranged from a decline of 3.9 percentage points to an increase of 6.9. The authors concluded that the method could not provide informative estimates for this outcome. The lesson matches the checks above. A short, noisy pre-period gives the method little to work with, and reporting results across reasonable choices shows readers when a conclusion depends on them.

          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425. https://doi.org/10.1257/jel.20191450Feasibility, data requirements, and robustness checks.
          • Ferman, B., & Pinto, C. (2021). Synthetic controls with imperfect pretreatment fit. Quantitative Economics, 12(4), 1197–1221. https://doi.org/10.3982/qe1596Bias under imperfect fit and the demeaned estimator.
          • Ferman, B., Pinto, C., & Possebom, V. (2020). Cherry picking with synthetic controls. Journal of Policy Analysis and Management, 39(2), 510–532. https://doi.org/10.1002/pam.22206How specification choices can be exploited.
          • Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510. https://doi.org/10.1111/ajps.12116Leave-one-out and in-time placebo checks in practice.
          • Bifulco, R., Rubenstein, R., & Sohn, H. (2017). Using synthetic controls to evaluate the effect of unique interventions: The case of Say Yes to Education. Evaluation Review, 41(6), 593–619. https://doi.org/10.1177/0193841x17742233An education application.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 6 · Expert

          Augmented synthetic control and modern inference.

          The Advanced lesson ended with a problem. When no blend of donors can match the treated unit, synthetic control is biased, and the usual advice is to stop. This lesson covers a correction that uses an outcome model to estimate and remove the bias, and an inference method that tests hypotheses about the effect using the time series of residuals rather than placebo units.

          Before thisAdvanced: feasibility, fit, and robustnessWhy imperfect fit causes bias.
          Next moduleMachine learning for causal inferenceFlexible prediction for adjustment and for finding who benefits. Related: synthetic DiD in Difference-in-differences, Expert.
          By the end

          What you will be able to do

          1. Write the augmented synthetic control estimator and explain its bias correction.
          2. Explain the role of the ridge penalty.
          3. Carry out conformal inference for a hypothesized effect and invert it into a confidence set.
          4. Explain why conformal inference needs a long pre-period.
          5. Choose among synthetic control, its augmented and demeaned versions, and synthetic DiD.
          6. Say why the effect for one treated unit calls for a prediction interval, and name its two sources of uncertainty.
          7. Describe generalized synthetic control and when it applies.
          The problem

          Where the bias comes from

          Let \(X_1\) be the treated unit's pre-period outcomes and \(X_0\) the donors'. Suppose the untreated post-period outcome follows some function of the pre-period outcomes, \(Y_{jt}(0) = m_t(X_j) + \varepsilon_{jt}\). Then the bias of synthetic control is approximately

          \[ m_t(X_1) - \sum_j w_j\, m_t(X_j). \]

          With a perfect fit, \(X_1 = \sum_j w_j X_j\) and, if \(m_t\) is linear, the bias vanishes. With an imperfect fit, the remaining gap in pre-period outcomes passes through \(m_t\) into the estimate. Ben-Michael, Feller, and Rothstein (2021) propose estimating \(m_t\) and subtracting the estimated bias.

          Method

          Augmented synthetic control

          The augmented estimator is

          \[ \hat\tau^{\text{aug}}_t = Y_{1t} - \Big[ \sum_j w_j Y_{jt} + \Big( \hat m_t(X_1) - \sum_j w_j\, \hat m_t(X_j) \Big) \Big], \]

          where \(\hat m_t\) is fitted on the donors. With a ridge regression of post-period outcomes on pre-period outcomes, Ben-Michael et al. (2021) show that the estimator equals a weighted average of donors whose weights can be negative. The negative weights let it extrapolate beyond the donors' range, which plain synthetic control cannot do. The ridge penalty controls how far. A small penalty removes more of the bias but allows larger, more variable weights. A large penalty stays close to plain synthetic control. They recommend reporting how far the augmented weights depart from the original ones, because extrapolation rests on the outcome model being right.

          When the fit is already good, the correction is small and the two estimators agree. The correction matters when the fit is poor, which is when plain synthetic control should not be trusted.

          Worked example · Monte Carlo

          Correcting for a district out of range

          The first script draws 200 samples in which the adopting district's scores rise faster than any donor's, with a true effect of 3. The average pre-period RMSPE of the synthetic control is 1.23, a poor fit. Synthetic control averages 5.65, with a standard deviation of 0.82 and a root mean squared error of 2.77. The demeaned version averages 5.04. Augmenting with a ridge regression and a penalty of 1 averages 3.47, with a larger standard deviation of 0.97 but a root mean squared error of 1.07. A penalty of 10 averages 3.72, with a standard deviation of 0.81.

          The bias falls sharply, though not to zero, and the variance rises as the penalty falls, which is the expected tradeoff. The correction works here because the true outcome model is close to linear in past outcomes. If it were not, the extrapolation could mislead.

          Run it yourself in R: code and output
          Inference

          Conformal inference

          Placebo tests in space treat the choice of treated unit as random, an assumption that rarely describes how policies are adopted. Chernozhukov, Wüthrich, and Zhu (2021) propose permuting over time instead. To test that the effect equals \(\theta\), subtract \(\theta\) from the treated unit's post-period outcomes, fit the synthetic control using all periods, and compute the residuals. If the hypothesis is true, the post-period residuals should look like the pre-period ones. Compare a statistic of the post-period residuals, such as their mean absolute value, with the same statistic after each cyclic shift of the time series. The p-value is the share of shifts with a statistic at least as large.

          Repeating the test over a grid of \(\theta\) values and keeping those not rejected gives a confidence set. The test is exact when the residuals are exchangeable over time and approximately valid under weaker conditions on the errors and the fitted model.

          Worked example · simulated data

          Conformal inference needs time

          The second script applies the test to the reading program with a true effect of 3, first with the module's 12 pre-period years and then with 30. With 12 years, the p-value for no effect is 0.529, and the 90% confidence set contains every value tried from 0 to 8. Fitting all 17 years under the null lets the weights shift toward fast-rising donors and absorb much of the effect, and there are only 17 possible shifts. With 30 pre-period years, the p-value for no effect is 0.029, the smallest possible with 35 periods, and the 90% confidence set runs from 2.1 to 4.6.

          The lesson is that conformal inference trades the random-assignment assumption of placebo tests for assumptions about the time series, and it needs enough periods for both the fit and the permutations to be informative.

          Inference

          An interval for a single unit's effect

          With one treated unit, the effect in a given year is not a fixed parameter of a population. It is the difference between the unit's observed outcome and the outcome it would have had without the program, a quantity that includes a one-time shock in that year. Cattaneo, Feng, and Titiunik (2021) argue that it should be treated as a random quantity to be predicted, and that the right interval is a prediction interval, not a confidence interval.

          Two sources of uncertainty enter. The first is in-sample, because the weights are estimated from a finite pre-period and carry error. The second is out-of-sample, because the treated unit's own outcome in the post-period contains a shock that no amount of pre-period data removes. For the first, they propose a simulation-based bound with finite-sample probability guarantees, which tends to be conservative. For the second, the shock cannot be learned from the data without assumptions, so they recommend a sensitivity analysis over how large and how dispersed it may be.

          This differs from the placebo and conformal approaches above. Placebo tests ask whether the effect is unusual compared with effects for units that were not treated, and conformal inference asks whether the post-period residuals look like the pre-period ones. A prediction interval reports a range for the effect itself and separates the sources of error. The authors provide software for R, Python, and Stata, which is a better starting point than coding the bounds by hand.

          Extension

          Several treated units and different adoption dates

          The synthetic control methods above are built for one treated unit. Xu (2017) extends the idea to several treated units that adopt at different times. The model lets each unit's outcome depend on covariates and on a small number of common time patterns, called latent factors, each with a unit-specific loading. A model with unit and time effects, the one behind difference-in-differences, is the special case in which all units share the same loading on the time pattern, which is why the method relaxes parallel trends.

          The estimation has three steps. First, estimate the factors and the covariate coefficients using only the control units. Second, estimate each treated unit's loadings from its own pre-treatment periods. Third, use those loadings to impute the unit's outcome without treatment in each later period, and take the difference from the observed outcome as the effect. The number of factors is chosen by cross-validation on the treated units' pre-treatment periods, and uncertainty comes from a parametric bootstrap. In the simulations reported by the author, coverage of the 95 percent intervals was close to 0.95.

          The method has costs. It needs enough pre-treatment periods to estimate the loadings, and with too few it can fail. It extrapolates if a treated unit's loadings fall outside the range of the controls', which is the same danger as a poor pre-period fit in plain synthetic control. And the factor structure is an assumption about how outcomes are generated, to be checked with the pre-period fit and placebo tests. The method is available in the gsynth package for R.

          Practice

          Choosing among the methods

          • Good pre-period fit, long pre-periodPlain synthetic control, with placebo inference and robustness checks.
          • Constant level gap but similar trendsThe demeaned estimator, or synthetic DiD (Arkhangelsky et al., 2021), which adds time weights and a level shift.
          • Poor fit that cannot be fixed by changing donorsAugmented synthetic control, reporting the extrapolation it implies, or reconsider whether the design is feasible.
          • Several treated unitsSynthetic DiD, generalized synthetic control (Xu, 2017) when the pre-period is long, or the staggered-adoption estimators in the difference-in-differences module.
          • InferenceReport placebo p-values and, with a long pre-period, a conformal confidence set or a prediction interval that separates in-sample from out-of-sample uncertainty (Cattaneo et al., 2021).
          Check your understanding

          Scenario and diagnosis questions

          Four short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The augmented synthetic control method. Journal of the American Statistical Association, 116(536), 1789–1803. https://doi.org/10.1080/01621459.2021.1929245The augmented synthetic control method.
          • Chernozhukov, V., Wüthrich, K., & Zhu, Y. (2021). An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association, 116(536), 1849–1864. https://doi.org/10.1080/01621459.2021.1920957Conformal inference for synthetic controls.
          • Ferman, B., & Pinto, C. (2021). Synthetic controls with imperfect pretreatment fit. Quantitative Economics, 12(4), 1197–1221. https://doi.org/10.3982/qe1596Bias under imperfect fit.
          • Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118. https://doi.org/10.1257/aer.20190159Synthetic difference-in-differences.
          • Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425. https://doi.org/10.1257/jel.20191450The method's assumptions and limits.
          • Cattaneo, M. D., Feng, Y., & Titiunik, R. (2021). Prediction intervals for synthetic control methods. Journal of the American Statistical Association, 116(536), 1865–1880. https://doi.org/10.1080/01621459.2021.1979561Prediction intervals for the effect on a single treated unit, with software.
          • Xu, Y. (2017). Generalized synthetic control method: Causal inference with interactive fixed effects models. Political Analysis, 25(1), 57–76. https://doi.org/10.1017/pan.2016.2Generalized synthetic control for several treated units with different adoption dates.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 7 · Foundations

          What machine learning can and cannot do for causal questions.

          Machine learning predicts outcomes well. Causal inference asks a different question, what would happen if we changed something. This lesson explains the difference, why a model that predicts well can give the wrong causal answer, and the specific jobs inside a causal analysis where machine learning helps.

          Before thisMatching and weighting, FoundationsAdjusting for observed differences.
          After thisIntermediate: many controlsThe lasso and double machine learning.
          By the end

          What you will be able to do

          1. Distinguish description, prediction, and causal inference as tasks.
          2. Explain why in-sample fit rises with every added control, and what adding many controls does to prediction and to the spread of an effect estimate.
          3. Choose a validation split that matches how a prediction model will be used.
          4. Read a confusion matrix, compute accuracy, recall, specificity, precision, and F1, and say why accuracy can mislead when classes are unequal.
          5. Explain why a model with a high R-squared can give a biased effect estimate.
          6. Name the parts of a causal analysis where machine learning helps.
          7. Explain why the research design still does the identifying work.
          The distinction

          Three kinds of questions

          Hernán, Hsu, and Healy (2019) sort data science tasks into three kinds. Description summarizes data, such as the share of students chronically absent. Prediction maps inputs to an expected output, such as which ninth graders are likely to leave school before graduating. Counterfactual prediction, which is causal inference, asks what would happen to an outcome if we intervened, such as how graduation rates would change if a district added mentoring. The first two can be answered from data alone. The third also requires knowledge of how the data were generated, because the same data are consistent with many causal stories.

          Kleinberg, Ludwig, Mullainathan, and Obermeyer (2015) point out that some policy decisions are pure prediction problems. If a district must decide which students to contact first about attendance, it may only need to know who is most likely to miss school, not why. Many other decisions, such as whether mentoring works or for whom, are causal.

          In the vocabulary of Module 1, a causal question first names its estimand. The later lessons use machine learning to estimate average effects and effects that differ across students.

          Core idea

          Why a better fit can predict worse

          Machine learning methods are judged by how well they predict cases they have not seen. James, Witten, Hastie, and Tibshirani (2013) explain why this differs from how well they fit the data used to build them. A flexible model can follow the training data closely, including its noise. Its predictions then change a lot from one sample to the next, which is high variance. A rigid model varies less but may miss real patterns, which is high bias. Expected prediction error on new data reflects both, so it is usually smallest at a moderate level of flexibility.

          Because training fit always improves with flexibility, it cannot be used to choose the level of flexibility. The standard tool is cross-validation. Split the data into folds, fit the model on all but one fold, measure its error on the held-out fold, rotate, and average. The tuning choice with the lowest held-out error is selected. The same habit of judging a model on data it did not see returns in the causal methods of later lessons, as cross-fitting.

          lowhigherrorrigidflexibleModel flexibilitylowest error on new datatraining datanew data
          Why training fit cannot choose the model. Illustrative curves. Error on the training data falls the more flexible the model becomes, so it points toward ever more flexibility. Error on new data falls at first, then rises as the model starts fitting noise. The choice that predicts best lies at the bottom of the second curve, which is what cross-validation estimates.
          Overfitting

          More controls fit better and can predict worse

          Each variable added to a regression raises its fit to the data at hand, whether or not the variable is related to the outcome. In the extreme case with as many variables as students, the fit is perfect whatever the truth is (Chernozhukov et al., 2024). Two questions follow. How well will the model predict students it has not seen, and how much noise do the extra variables add to the number a study is after?

          The third script in the code panel under the worked example simulates a randomized tutoring study with 120 students and a true effect of 3 points. Of 100 available controls, only two affect the score. A model with those two explains 39 percent of the variance in the sample and 38 percent among 2,000 new students. With 20 controls, the fit in the sample rises to 0.48 and the fit on new students falls to 0.26. With 60 controls the figures are 0.70 and −0.27, which is worse than predicting everyone’s score with the average, and with 100 controls they are 0.91 and −3.38. The adjusted R² stays between 0.37 and 0.39 in every row. It estimates what the best linear predictor could achieve, which is the same in every row because the extra variables have zero coefficients. It does not describe the fitted regression, which must estimate every coefficient from the same 120 students. The adjustment corrects the raw fit when the number of variables is below the sample size (Chernozhukov et al., 2024), but because it describes the best linear predictor, it cannot show how badly a regression with 100 variables and 120 students will predict.

          The study is randomized, so every one of these models estimates the effect without bias, and the average estimate stays between 2.99 and 3.16. What changes is the spread. The standard deviation of the estimate is 1.44 with no controls, 1.15 with the two useful controls, 1.22 with 20, 1.57 with 60, and 2.78 with 100. A control that predicts the outcome removes chance error, as in the discussion of adjustment in Module 1. A control that does not predict it adds noise, and the cost grows quickly as the number of variables approaches the number of students.

          Three habits follow. Look at the number of candidate variables relative to the sample size, not only at the number of variables. Choose the adjustment set from knowledge of the problem before seeing the outcome. When there are truly many candidates, use tools made for that case, namely penalized regression and sample splitting, which the next levels describe.

          Validation

          Validate the way the model will be used

          Cross-validation judges a model on cases it did not use to fit itself, and the judgment is only as honest as the split. A split should mimic how the model will be used. A district that builds an early-warning model for ninth graders will apply it to next year’s students, and possibly to schools that were not in the data.

          Leakage is any route by which information from the held-out cases reaches the fit. Students in one school share features that no data set records, so a random split puts students from every school on both sides of the line. A model given each school’s record can then lean on it. In a simulation of 40 schools with 25 students each, a model that learned each school’s typical score had a cross-validated error of 6.04 points when students were split at random. Splitting by school raised the error to 7.71, the same as a model that ignored schools, and that is the error to expect for a school the model has never seen. Time works the same way. Predictions about next year should be validated by training on earlier years and testing on later ones, since a random split lets the model borrow from the future.

          A second source is feature timing. A variable recorded after the treatment, such as attendance during the program year, can improve prediction and still ruin an effect estimate, because it is a mediator or a collider in the language of Module 1. The worked example below shows the cost. Fix the list of features before the outcome is seen, and record when each one was measured.

          Evaluation

          Accuracy is not the only score

          Many education models classify, for example by predicting whether a student will drop out. A confusion matrix counts four kinds of outcome, namely true positives, false positives, true negatives, and false negatives, and the common scores are built from it (Kelleher et al., 2020). Accuracy is the share of all students classified correctly. Recall, also called sensitivity, is the share of students who drop out that the model flags. Specificity is the share of students who stay that the model leaves unflagged. Precision is the share of flagged students who do drop out. The F1 measure is the harmonic mean of precision and recall, so it is low when either one is low.

          The scores can disagree sharply when the groups differ in size. Take 1,000 students of whom 50 will drop out. A model that flags nobody has an accuracy of 95 percent and finds none of the 50 students. A model that flags 150 students, including 40 of the 50, has a lower accuracy of 88 percent. It is the more useful of the two for a school that wants to offer support.

          ModelAccuracyRecallSpecificityPrecisionF1Average class accuracy
          Flags nobody0.950.001.00Not definedNot defined0.50
          Flags 150 students, 40 of the 50 dropouts0.880.800.880.270.400.84

          Hypothetical counts. The first model has 0 true positives and 950 true negatives. The second has 40 true positives, 10 false negatives, 110 false positives, and 840 true negatives.

          Accuracy rewards the first model because staying students are 19 times as numerous as dropouts. Kelleher et al. (2020) show the same pattern with their churn example and propose the average of the recall for each class, here 0.50 against 0.84, when the classes are unequal and neither is more important. Which score to use is a question about costs. If a missed dropout is much worse than a wasted offer of support, recall deserves the weight, and if support is scarce, precision does.

          Most classifiers produce a probability, and the flagging rule is a threshold on it, so every score above moves with the threshold. The ROC curve plots the share of true positives against the share of false positives over all thresholds, and the area under it summarizes how well the model ranks students across thresholds (James et al., 2021). The ranking says nothing about where to put the threshold, and that choice belongs to the school.

          Two cautions connect this to the rest of the course. First, validate the way the model will be used. Out-of-time sampling fits the model to one period and tests it on a later one, and the periods need to be long enough to include any seasonal pattern, because a model trained on one season and tested on another does not show how it would perform in use (Kelleher et al., 2020). Second, classification scores do not answer causal questions. A student flagged as at risk is not necessarily a student who would benefit from support, as the discussion of prediction and causation above explains. In Module 2, a propensity model that predicts treatment almost perfectly is a warning, not a success, because it means that treated and untreated students barely overlap.

          The core issue

          Predictions versus coefficients

          Mullainathan and Spiess (2017) frame the difference as \(\hat y\) versus \(\hat\beta\). Machine learning methods are built to produce good predictions \(\hat y\) of an outcome. They are not built to produce coefficients \(\hat\beta\) with a causal meaning. Many different models predict almost equally well while assigning very different roles to individual variables, so the role a model gives a variable is not a reliable guide to that variable’s effect.

          A causal effect estimate depends on which variables are adjusted for, and that choice comes from the design, not from how well the model fits. Adjusting for a variable affected by the treatment, a bad control, usually improves the fit and biases the effect (Cinelli, Forney, & Pearl, 2024).

          Case · published study

          Predicting house values

          Mullainathan and Spiess (2017) illustrate both points with a prediction exercise.

          MethodR² in training sampleR² in hold-out sample
          Ordinary least squares47.3 percent41.7 percent
          Regression tree39.6 percent34.5 percent
          LASSO46.0 percent43.3 percent
          Random forest85.1 percent45.5 percent
          Ensemble80.4 percent45.9 percent

          Log house value predicted from 150 characteristics of owner-occupied units in the 2011 American Housing Survey. Models were trained on 10,000 units. Source is Mullainathan and Spiess (2017), Table 1.

          The random forest fits the training data far better than OLS, 85 percent against 47 percent, but on new houses its advantage shrinks to a few points. Training fit overstates what flexible methods deliver, which is why the hold-out column is the one that matters.

          The authors then asked what the models say about individual variables. They split the sample into ten parts of about 5,000 units and fit LASSO, a regression that shrinks coefficients toward zero and drops some variables entirely, to each. Prediction quality was about the same in every part, yet the set of variables LASSO kept changed from part to part, and more than half were dropped in each run. Correlated characteristics, such as the number of rooms and square footage, can substitute for each other in a prediction. A model that predicts well can therefore give no stable answer about which variables matter, the \(\hat y\) versus \(\hat\beta\) point of the previous section.

          Worked example · simulated data

          A better fit, a worse answer

          In the simulation, tutoring goes mostly to students with low prior scores, and it raises scores both directly and by improving attendance. The true total effect of tutoring is 4 points.

          The raw comparison of tutored and other students gives −5.54, because tutored students started lower. Adjusting for prior score and family income, both measured before tutoring, gives 3.79, with an R-squared of 0.850. Adding attendance raises the R-squared to 0.892, a better predictive model, but the tutoring estimate falls to 1.94. Attendance is part of how tutoring works, so adjusting for it removes part of the effect. A model selection procedure that rewards fit would choose the worse causal model.

          The second script concerns the section on validation. It compares random folds with school folds for a model that has learned each school’s typical score.

          The third script concerns the section on many controls and a small sample. It adds up to 100 controls to a randomized study of 120 students and compares fit in the sample, fit on new students, and the spread of the effect estimate.

          Run it yourself in R: code and output
          The useful jobs

          Where machine learning helps

          Athey and Imbens (2019) describe several jobs inside a causal analysis that are prediction problems in disguise, and where machine learning is well suited.

          • Adjusting for many controls. Under selection on observables, an estimate needs good predictions of the outcome and of treatment from the controls. With dozens or hundreds of candidate controls, machine learning can make those predictions. The Intermediate lesson shows how to do this without biasing the effect.
          • Estimating propensity scores and outcome models in weighting estimators and in doubly robust estimators, which combine a propensity score model with an outcome model. The Advanced level of Matching and weighting and the Expert level of Difference-in-differences show both.
          • Finding heterogeneity. Machine learning can search many characteristics for the students who benefit most, as long as the search and the estimate are kept separate. The Advanced lesson covers this.
          • Targeting. Once effects are estimated for different students, a rule for whom to treat can be learned and evaluated. The Expert lesson covers this.

          In each case the design still does the identifying work. Randomization, a cutoff, an instrument, or the assumption that all confounders are measured is what makes the estimate causal. Machine learning only makes the estimation more flexible.

          Case · published study

          When the prediction is the policy

          Some decisions turn directly on a prediction. A judge deciding whether to release a defendant before trial is, by law, predicting what the defendant would do if released. Kleinberg, Lakkaraju, Leskovec, Ludwig, and Mullainathan (2018) trained a machine learning model on New York City cases heard from 2008 to 2013 to predict that risk and asked whether its predictions could improve the judges’ decisions.

          The causal difficulty is that outcomes are observed only for defendants the judges released. A model’s predictions for detained defendants cannot be checked directly, and judges may have used information the data do not contain. The authors call this the selective labels problem. They addressed it with research designs rather than better prediction. Cases were assigned to judges as good as randomly, and judges differ in leniency, so the defendants released by stricter judges can be compared with those released by more lenient ones.

          Under these designs, one policy simulation suggests that release rules based on the predictions could reduce crime by up to 24.7 percent with no change in jailing rates, or reduce jailing by up to 41.9 percent with no increase in crime, while also reducing racial disparities. The lesson for this module is that a prediction became useful for a decision only once a design for evaluating counterfactual decisions was in place.

          Cautions

          Common misreadings

          • “The model is accurate, so its important variables are causes.”Variable importance describes prediction, not effects.
          • “With enough controls, machine learning removes confounding.”It can only adjust for confounders that are measured.
          • “Machine learning replaces the research design.”It improves estimation within a design. It cannot supply identification.
          • “Any subgroup the algorithm finds is real.”Searching many subgroups finds differences by chance. Honest sample splitting is needed, as the Advanced lesson shows.
          Check your understanding

          Scenario and diagnosis questions

          Short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Mullainathan, S., & Spiess, J. (2017). Machine learning: An applied econometric approach. Journal of Economic Perspectives, 31(2), 87–106. https://doi.org/10.1257/jep.31.2.87Prediction versus estimation. Start here.
          • Hernán, M. A., Hsu, J., & Healy, B. (2019). A second chance to get causal inference right: A classification of data science tasks. Chance, 32(1), 42–49. https://doi.org/10.1080/09332480.2019.1579578Description, prediction, and counterfactual prediction.
          • Athey, S., & Imbens, G. W. (2019). Machine learning methods that economists should know about. Annual Review of Economics, 11, 685–725. https://doi.org/10.1146/annurev-economics-080217-053433A survey of machine learning methods for causal questions.
          • Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015). Prediction policy problems. American Economic Review, 105(5), 491–495. https://doi.org/10.1257/aer.p20151023When a policy question is a prediction question.
          • Cinelli, C., Forney, A., & Pearl, J. (2024). A crash course in good and bad controls. Sociological Methods & Research, 53(3), 1071–1104. https://doi.org/10.1177/00491241221099552Good and bad controls.
          • James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning with applications in R. Springer. https://doi.org/10.1007/978-1-4614-7138-7An introductory text on prediction, the bias-variance trade-off, and cross-validation.
          • Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018). Human decisions and machine predictions. Quarterly Journal of Economics, 133(1), 237–293. https://doi.org/10.1093/qje/qjx032Machine learning predictions for bail decisions, and the selective labels problem.
          • Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., & Syrgkanis, V. (2024). Applied causal inference powered by ML and AI (Version 0.1.1) [Online book]. https://causalml-book.org/Overfitting when the number of variables is not small relative to the sample, and adjusted measures of fit.
          • Kelleher, J. D., Mac Namee, B., & D’Arcy, A. (2020). Fundamentals of machine learning for predictive data analytics: Algorithms, worked examples, and case studies (2nd ed.). MIT Press.Chapter 9 on evaluation, including confusion-matrix scores, average class accuracy, and out-of-time sampling.
          • James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning with applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1A textbook introduction to model assessment, shrinkage, trees, and ROC curves, with R labs.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 7 · Intermediate

          Many controls: the lasso and double machine learning.

          Administrative data often offer hundreds of possible controls. Including them all overfits. Choosing a few by hand invites specification searching. Letting an algorithm choose can quietly drop confounders. This lesson explains why, and two methods built to avoid it, post-double-selection and double machine learning.

          Before thisFoundations: what machine learning can and cannot doPrediction versus causation.
          After thisAdvanced: heterogeneous effectsMeta-learners and causal forests.
          By the end

          What you will be able to do

          1. Explain how the lasso selects variables.
          2. Say when ridge or the lasso is expected to predict better, and what the one-standard-error rule trades away.
          3. Explain why selecting controls by their relation to the outcome alone causes bias.
          4. Carry out post-double-selection.
          5. Explain orthogonal scores and cross-fitting in double machine learning.
          6. Choose learners for the outcome and treatment predictions by held-out error, and check the estimate under more than one learner.
          7. Compute the AIPW score for a binary treatment and explain double robustness.
          Method

          The lasso

          The lasso (Tibshirani, 1996) fits a regression while penalizing the sum of the absolute values of the coefficients,

          \[ \min_b \; \frac{1}{2n}\sum_i (y_i - x_i'b)^2 + \lambda \sum_j |b_j|. \]

          The penalty shrinks coefficients toward zero and sets many of them exactly to zero, so the lasso selects a subset of variables. The penalty level \(\lambda\) controls how many survive. Belloni, Chernozhukov, and Hansen (2014b) give a data-driven choice that scales with the residual standard deviation and the number of candidate variables, which the code in this lesson uses.

          Regularization

          Ridge, lasso, and how strongly to penalize

          Ridge regression and the lasso both fit a linear model while penalizing the size of the coefficients, and the tuning value \(\lambda\) sets how strongly. With \(\lambda = 0\) they equal least squares, and a larger \(\lambda\) shrinks the coefficients further toward zero. The two penalties differ in one respect that matters. Ridge shrinks every coefficient and sets none to exactly zero, while the lasso sets some to exactly zero, so it also selects variables. James et al. (2021, chapter 6) trace this to geometry. The lasso constraint region has corners on the coordinate axes, where the best fit often lands.

          \[ \text{ridge}\;\; \sum_i \bigl(y_i - \beta_0 - x_i^{\prime}\beta\bigr)^2 + \lambda \sum_j \beta_j^2, \qquad \text{lasso}\;\; \sum_i \bigl(y_i - \beta_0 - x_i^{\prime}\beta\bigr)^2 + \lambda \sum_j |\beta_j| \]

          Neither penalty is better in general. The lasso is expected to do better when a few variables have substantial coefficients and the rest are small or zero, and ridge when the outcome depends on many variables with coefficients of similar size. The number of related variables is never known in advance, so \(\lambda\) and the choice between the two are made by cross-validation (James et al., 2021).

          The third script in the code panel under the worked example compares them with 100 students per sample, 30 measures, and a score noise of 3 points. The error reported is the extra mean squared prediction error on new students, averaged over 60 samples. In a sparse world, where three measures matter with a coefficient of 2, least squares has an excess error of 3.92, ridge 2.83, and the lasso 0.98 at the penalty with the smallest cross-validated error. In a dense world, where all 30 measures have a coefficient of 0.5, the figures are 3.99 for least squares, 2.61 for ridge, and 3.92 for the lasso, which is no better than least squares. The lasso keeps 10.1 measures on average in the sparse world, where three matter, and 23.6 in the dense world, where all 30 matter.

          The one-standard-error rule picks the most heavily penalized model whose cross-validated error is within one standard error of the minimum. The reasoning is that when several models look about equally good, the simplest is a sensible choice (James et al., 2021). In the simulation it keeps fewer measures, 4.6 against 10.1 in the sparse world, and it costs accuracy, with excess errors of 1.70 against 0.98 in the sparse world and 5.43 against 3.92 in the dense world. Its value is a smaller model, not a better prediction.

          These are tools for prediction. A penalty chosen to minimize prediction error does not make the retained measures the confounders of a program effect, because the retained set can include noise and omit variables that matter for treatment but only weakly for the outcome. Choosing controls for an effect needs the model for treatment as well, which is what post-double-selection and double machine learning, described below, are for.

          The problem

          Why one lasso is not enough

          Suppose we regress test scores on tutoring hours and 100 candidate controls, and let the lasso choose controls while always keeping tutoring. The lasso keeps controls that improve the prediction of scores. A confounder that strongly predicts tutoring but only moderately predicts scores, once tutoring is in the model, can be dropped because tutoring already carries much of its information. Dropping it is exactly what causes omitted variable bias. Belloni, Chernozhukov, and Hansen (2014b) show that this single-selection approach can produce badly biased estimates and confidence intervals that rarely contain the truth.

          Belloni et al. (2014b) illustrate the failure with a simulation. Across 10,000 replications, the studentized single-selection estimator has a bimodal distribution far from the normal, while the double-selection estimator’s distribution is close to normal.

          Method

          Post-double-selection

          The fix is to select twice. Run one lasso of the outcome on the controls and another of the treatment on the controls. Keep every control selected by either, then run an ordinary regression of the outcome on the treatment and the union of selected controls. A confounder that matters for treatment is now kept even if its role in the outcome equation looks small. Belloni et al. (2014b) show that the resulting estimate and its usual standard error are valid across a wide range of settings in which only a modest number of controls truly matter. Belloni, Chernozhukov, and Hansen (2014a) give an accessible overview.

          Case · published study

          What double selection can change

          Belloni, Chernozhukov, and Hansen (2014a, 2014b) revisited a well-known state-panel study that related state abortion rates after legalization around 1970 to later crime rates. The original specification controlled for a handful of state characteristics with state and year effects. The concern is that states may have followed different nonlinear trends related to both variables. They built 284 candidate controls, including initial levels, interactions with time, and quadratics, which together allow cubic trends that depend on state characteristics.

          EstimatorViolent crimeProperty crimeMurder
          Original controls, first differences−0.157 (0.034)−0.106 (0.021)−0.218 (0.068)
          All 284 flexible controls0.071 (0.284)−0.161 (0.106)−1.327 (0.932)
          Post-double-selection−0.171 (0.117)−0.061 (0.057)−0.189 (0.177)

          Standard errors in parentheses. State panel, 50 states over 12 years. Source is Belloni, Chernozhukov, and Hansen (2014a), Table 1.

          Including all 284 controls leaves too little information, and the estimates swing to implausible values. Double selection keeps the controls that predict either crime or the abortion rate, between eight and twelve depending on the outcome, and these differ from the original controls. The point estimates stay close to the originals for violent crime and murder, but the standard errors grow roughly two-and-a-half to three-and-a-half times, so the data no longer distinguish the estimates from zero. The lesson is about precision and robustness rather than about the substantive question. A result that holds under a few hand-picked controls may not survive a principled search over many plausible ones.

          Method

          Double machine learning

          Chernozhukov et al. (2018) generalize the idea to any machine learning method. In the partially linear model \(y = \theta d + g(X) + u\), predict the outcome from the controls, \(\hat\ell(X)\), and the treatment from the controls, \(\hat m(X)\). Then regress the outcome residual on the treatment residual,

          \[ \hat\theta = \frac{\sum_i (d_i - \hat m(X_i))(y_i - \hat\ell(X_i))}{\sum_i (d_i - \hat m(X_i))^2}. \]

          Two features make this work. The estimating equation is Neyman orthogonal, meaning small errors in the two predictions affect \(\hat\theta\) only through their product, so predictions that are moderately accurate are enough. And the predictions are cross-fitted. The sample is split into folds, each fold’s predictions come from models trained on the other folds, and the roles rotate. Cross-fitting prevents the overfitting of flexible learners from leaking into the effect estimate. The same logic underlies the doubly robust DiD estimators in the difference-in-differences module.

          Theory

          Two biases that double machine learning removes

          Chernozhukov et al. (2018) explain why simply plugging a machine learning prediction into a regression gives a poor estimate of \(\theta\). The first problem is regularization bias. Methods such as LASSO and random forests shrink or smooth their predictions to control variance. If only the outcome is predicted from the controls and the residual is regressed on the treatment, the error from shrinkage feeds directly into \(\hat\theta\), and the estimate converges to the truth too slowly for standard confidence intervals to be valid. Residualizing the treatment as well makes the estimating equation Neyman orthogonal, so the two prediction errors enter only through their product.

          The second problem is overfitting bias. If the same observations are used to fit the predictions and to estimate \(\theta\), a flexible learner can fit the noise in those observations, and that noise is correlated with the residuals used in the second step. Sample splitting removes the correlation, and cross-fitting, which rotates the roles of the folds and averages, recovers the efficiency lost by using only part of the data for each step. Their applications include the effect of 401(k) eligibility on financial assets and the effect of unemployment insurance bonuses on the length of unemployment.

          Choosing learners

          Which learner for which prediction

          Double machine learning and the doubly robust estimators above need two predictions for each student, one for the outcome and one for treatment. They do not require a particular learner. They require learners that predict well enough, and the table summarizes the common choices, the quantity each one needs tuned, and what to watch for.

          LearnerFits well whenWhat is tunedNotes
          Linear or logistic regression, with or without a penaltyRelationships are close to additive and smooth, or the penalty can handle many variables.Ridge, lasso, or elastic net penalty, chosen by cross-validation.Curves and interactions are missed unless they are added by hand.
          Single decision treeSimple rules and a few interactions, with results that can be read.Depth, or how far the tree is pruned.Very non-robust, since a small change in the data can change the tree a great deal (James et al., 2021).
          Random forestThresholds and interactions among many variables, with little tuning.Number of variables tried at each split, often about the square root of the number available.Averages many trees built on random subsets of variables, which makes the trees less alike. More trees do not cause overfitting (James et al., 2021).
          Boosted treesComplicated patterns, when care is taken with tuning.Number of trees, shrinkage, and the size of each tree, chosen by cross-validation.Learns slowly by fitting small trees to the residuals. Unlike a forest, it can overfit if there are too many trees (James et al., 2021).
          Stacked ensembleNo single learner is clearly best.Weights on each candidate’s predictions, estimated on held-out data.Weights from least squares can be large and offsetting when learners are alike. Penalized weights were easier to interpret in an example of Chernozhukov et al. (2024).

          Descriptions follow James et al. (2021) for trees and forests and Chernozhukov et al. (2024) for ensembles.

          Chernozhukov et al. (2024) recommend an ensemble or a best-choice rule over several learners, so that the guarantee is no worse than that of the best candidate. Choose among learners by the cross-validated error of each prediction on held-out folds, and not by which learner gives the treatment estimate that looks best. Then report the estimate under more than one learner as a check on how much the conclusion depends on the choice. The list is not exhaustive. Neural networks, for example, suit very large samples with many features such as text and images, and they are judged by the same held-out error as the others.

          Method

          A score for a binary treatment

          The partially linear model above suits a treatment that enters linearly. When the treatment is binary, such as enrolling in tutoring, the same orthogonal idea gives the augmented inverse probability weighted (AIPW) score, which Chernozhukov et al. (2018, Section 5.1) use for the average treatment effect. Estimate an outcome model for each arm, \(\hat\mu_1(X)\) and \(\hat\mu_0(X)\), and a propensity score \(\hat e(X)\). Each student then receives the score

          \[ \hat\varphi_i = \hat\mu_1(X_i) - \hat\mu_0(X_i) + \frac{D_i\,(Y_i - \hat\mu_1(X_i))}{\hat e(X_i)} - \frac{(1 - D_i)\,(Y_i - \hat\mu_0(X_i))}{1 - \hat e(X_i)}. \]

          The average of the scores estimates the average treatment effect, and their standard deviation divided by \(\sqrt n\) gives its standard error. The first two terms are the outcome-model estimate. The remaining terms correct it with weighted residuals. Module 2 introduces the same estimator with ordinary regression models. It is doubly robust, which means it stays consistent when either the outcome models or the propensity score is right (Bang & Robins, 2005). When both are learned flexibly, their errors enter through a product, as in the previous section, so fit them on one fold and score the other. Extreme propensity scores inflate the weights, so inspect them first (Crump et al., 2009) and trim them. Chernozhukov et al. trimmed at 0.01 and 0.99 in their 401(k) application.

          The second script in the code panel under the worked example uses 2,000 students per sample and a true average effect of 3 points, known only because this is a simulation. Tutoring is offered according to a nonlinear function of two baseline measures, and scores depend on them nonlinearly too. Over 200 samples, the difference in means averages 7.15. A linear outcome model alone averages 4.89 and linear weighting alone averages 4.96, because neither captures the squared term. AIPW with two linear models averages 4.91 and its interval never covers the truth. Making only the propensity model flexible, here a cubic polynomial, brings the average to 2.95 with 95 percent coverage even though the outcome model is still wrong, which is double robustness at work. With both flexible the average is 2.98 with a standard deviation of 0.28, against 0.35 when only the propensity model is flexible, and coverage is 92 percent, slightly under the nominal level.

          The polynomial is a stand-in so that the code runs without extra packages. A lasso, forest, or boosted model can take its place, and with many controls that choice matters.

          Worked example · Monte Carlo

          Tutoring hours with 100 candidate controls

          The simulation draws 100 samples of 500 students with 100 candidate controls. Five controls drive both tutoring hours and scores, five more affect only scores, and the rest are noise. The true effect of an hour of tutoring is 1.

          A single lasso averages 1.318, with confidence intervals that contain the truth in 0 percent of samples. Post-double-selection averages 0.994, with 92 percent coverage. Double machine learning, using a lasso to select and least squares to refit within each of two folds, averages 1.036 with 91 percent coverage. Its spread is larger, 0.130 against 0.099, because each prediction is trained on only half the data. With more folds or larger samples this cost shrinks.

          The second script, discussed in the section on a score for a binary treatment, estimates the average effect of a binary tutoring offer under nonlinear confounding with linear and flexible nuisance functions.

          The third script concerns the section on ridge and the lasso. It compares least squares, ridge, and the lasso with two penalty rules in a sparse and a dense world.

          Run it yourself in R: code and output
          Extension

          Covariates in experiments

          In a randomized experiment, controls are not needed for unbiasedness, but they can improve precision. Bloniarz et al. (2016) show that lasso-based adjustment can reduce the variance of the estimated effect when there are many baseline covariates, extending the regression adjustment results discussed in the randomized experiments module.

          Case · published study

          Beyond the average with many controls

          Belloni, Chernozhukov, Fernández-Val, and Hansen (2017) extend these ideas from one coefficient to the full set of program evaluation parameters, including average and quantile treatment effects and, with an instrument, local average and local quantile effects. Their estimators use orthogonal moment conditions with LASSO-type selection of controls, and their confidence bands hold uniformly across quantiles.

          Their application asks how participation in a 401(k) retirement plan affects accumulated financial assets, using eligibility as an instrument and a much richer set of controls than earlier studies. Participation had a moderate effect at low quantiles of assets and a much larger effect at high quantiles. The authors suggest one reading, that participation matters little for households with a weak preference for saving and much more for households with a strong one. Their results were similar to earlier estimates that used fewer controls, which is itself informative. When a design is credible, flexible adjustment for many controls should not move the estimate much.

          Practice

          What to report

          1. The full list of candidate controls, and why each is measured before treatment.
          2. The learners used for each prediction and how their tuning was chosen.
          3. The number of folds and, because splits are random, results across several splits.
          4. Which controls were selected, for post-double-selection.
          5. The estimate under a simpler specification, for comparison.
          Check your understanding

          Scenario and diagnosis questions

          Short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Belloni, A., Chernozhukov, V., & Hansen, C. (2014a). High-dimensional methods and inference on structural and treatment effects. Journal of Economic Perspectives, 28(2), 29–50. https://doi.org/10.1257/jep.28.2.29An accessible overview of post-double-selection.
          • Belloni, A., Chernozhukov, V., & Hansen, C. (2014b). Inference on treatment effects after selection among high-dimensional controls. Review of Economic Studies, 81(2), 608–650. https://doi.org/10.1093/restud/rdt044The formal results for post-double-selection.
          • Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097Double machine learning, orthogonality, and cross-fitting.
          • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.xThe lasso.
          • Bloniarz, A., Liu, H., Zhang, C.-H., Sekhon, J. S., & Yu, B. (2016). Lasso adjustments of treatment effect estimates in randomized experiments. Proceedings of the National Academy of Sciences, 113(27), 7383–7390. https://doi.org/10.1073/pnas.1510506113Lasso adjustment in randomized experiments.
          • Belloni, A., Chernozhukov, V., Fernández-Val, I., & Hansen, C. (2017). Program evaluation and causal inference with high-dimensional data. Econometrica, 85(1), 233–298. https://doi.org/10.3982/ECTA12723Orthogonal estimators for average, quantile, and local effects with many controls, applied to 401(k) plans.
          • James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning with applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1Chapter 6 on ridge regression, the lasso, and the one-standard-error rule, and chapter 8 on trees, forests, and boosting.
          • Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., & Syrgkanis, V. (2024). Applied causal inference powered by ML and AI (Version 0.1.1) [Online book]. https://causalml-book.org/Chapter 9 on predictive inference with modern nonlinear regression and on combining learners into an ensemble.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 7 · Advanced

          Heterogeneous effects: meta-learners and causal forests.

          An average effect can hide students who gain a lot and students who gain nothing or lose. Machine learning can estimate how effects vary with many characteristics at once. This lesson covers the main strategies, the meta-learners that turn any prediction method into an effect estimator, causal trees and forests, and the discipline of honest estimation that keeps the search for subgroups from finding noise.

          Before thisIntermediate: many controlsCross-fitting and orthogonal scores.
          After thisExpert: inference on heterogeneityTesting whether heterogeneity is real, and policy learning.
          By the end

          What you will be able to do

          1. Define the conditional average treatment effect.
          2. Describe the T-, S-, X-, R-, and DR-learners.
          3. Explain honest estimation and why adaptive subgroup estimates are biased.
          4. Describe causal forests and Bayesian causal forests in outline.
          5. Score and compare models of individual effects with a doubly robust proxy outcome on a separate scoring set.
          6. Explain why a trial’s average effect may not apply to a target population, and what weighting and extrapolation assume.
          The estimand

          The conditional average treatment effect

          The conditional average treatment effect (CATE) is the average effect among students with characteristics \(x\),

          \[ \tau(x) = E[\,Y(1) - Y(0) \mid X = x\,]. \]

          It is identified under the same conditions as the average effect, randomization or selection on observables with overlap, now holding at each \(x\). Two cautions apply from the start. CATE describes groups of similar students, not any single student’s effect. And the more characteristics are searched, the easier it is to find apparent differences by chance.

          Method

          Meta-learners

          Künzel, Sekhon, Bickel, and Yu (2019) describe ways to build a CATE estimator from any prediction method.

          • T-learner. Fit separate outcome models for treated and untreated students and subtract their predictions. Simple, but each model’s errors pass straight into the difference, and a small treatment group yields a noisy model.
          • S-learner. Fit one model with treatment as an input. If the method shrinks or ignores the treatment variable, effects are pulled toward zero.
          • X-learner. Impute each treated student’s effect using the untreated model and each untreated student’s effect using the treated model, then fit a model to those imputed effects. It helps when one group is much larger and when the effect is simpler than the outcome.

          Two later learners use the residualizing logic of double machine learning. The R-learner (Nie & Wager, 2021) fits the effect function by minimizing the squared gap between each student’s outcome residual and the treatment residual multiplied by the predicted effect. The DR-learner (Kennedy, 2023) builds a doubly robust pseudo-outcome for each student and regresses it on characteristics. Both have guarantees that the estimated effect function is nearly as accurate as if the outcome and treatment models were known, provided those models are estimated well enough.

          Worked example · Monte Carlo

          Recovering who benefits from mentoring

          In the simulation, 25 percent of 3,000 students are randomly assigned to mentoring. The effect is 3 points for students with low prior scores and 0.5 for the rest, and scores depend on prior score and school climate in a curved way. Outcome models are degree-4 polynomials, and the X- and DR-learners fit the effect with a simpler degree-3 polynomial.

          Across 100 samples, the root mean squared error of each student’s estimated effect is 1.17 for the T-learner, 0.91 for the X-learner, and 1.20 for the DR-learner. The X-learner gains by modeling the simpler effect directly and by leaning on the larger untreated group. The DR-learner pays for dividing by a 25 percent treatment probability, which makes its pseudo-outcomes noisy in a sample this size. No learner dominates in general. Knaus, Lechner, and Strittmatter (2021) compare many estimators on realistic simulated data and find that performance depends on the setting.

          The second and third scripts concern the sections on scoring effect models and on moving from a trial to a target population.

          Run it yourself in R: code and output
          Key idea

          Honest estimation

          Athey and Imbens (2016) introduced causal trees, which split the sample into subgroups with different effects. Their method relies on honesty. One part of the sample chooses the splits, and a separate part estimates the effects within the chosen subgroups. When the same data do both, the search favors splits where noise happens to make effects look different, so the reported differences are too large and their confidence intervals too narrow.

          The second part of the code shows this directly. Every student has the same effect, and the code searches 17 possible cutoffs on prior score for the largest difference in effects. Using the same data, the chosen difference averages 1.18 points and is “significant” at the 5 percent level in 24.0 percent of samples. Using one half to choose the cutoff and the other to estimate, it averages 0.88 and is significant in 5.5 percent of samples, close to the nominal 5 percent.

          Intuition

          Trees and forests as ways of choosing neighbors

          Athey and Imbens (2019) offer a way to see what a tree does. The prediction in a leaf is the average outcome of the observations in that leaf, so a tree is a matching estimator that chooses a target’s neighbors by the splits. Unlike a kernel, which defines neighbors by distance on every covariate equally, a tree’s neighborhoods are rectangles that stretch along covariates that do not matter and stay narrow along those that do.

          A random forest averages many such trees, each grown on a bootstrap sample or subsample and with splits chosen from a random subset of covariates. The average is smoother and predicts better than any one tree. It also copes with many irrelevant covariates, which degrade a kernel estimator quickly but are mostly ignored by the splits. Honesty, using one part of the sample to choose the splits and another to estimate the leaf means, is what allows the forest’s estimates to be centered on the truth with valid confidence intervals. The causal forest applies the same neighborhood logic to effects rather than outcomes, comparing treated and control students who share a leaf.

          Methods

          Causal forests and Bayesian trees

          A single tree is unstable. Wager and Athey (2018) average many honest trees, each grown on a subsample, into a causal forest, and show that its estimates are asymptotically normal with valid confidence intervals. Athey, Tibshirani, and Wager (2019) generalize the approach to generalized random forests, which also handle instrumental variables and other estimating equations, and incorporate the residualizing step of the R-learner.

          A Bayesian line of work uses sums of regression trees. Hill (2011) showed that Bayesian additive regression trees (BART) fit flexible outcome models well for causal estimation. Hahn, Murray, and Carvalho (2020) separate the outcome model from the effect model in Bayesian causal forests, include the estimated propensity score as an input to reduce bias from confounding, and shrink the effect function toward homogeneity, which guards against finding heterogeneity that is not there.

          Wager and Athey (2018) give two ways to make trees honest. A double-sample tree splits its subsample in half, choosing splits with one half and estimating effects in the leaves with the other, and the halves are redrawn for each tree, so no observation is wasted across the forest. A propensity tree chooses splits using only treatment status, never outcomes. They estimate the forest’s variance with the infinitesimal jackknife, which they show is consistent, and in simulations the causal forest outperformed nearest-neighbor matching, with its advantage growing as irrelevant covariates were added.

          Validation

          Scoring an effect model when no effect is observed

          A prediction can be checked against the outcome that was later observed. A student’s own effect is never observed, because only one of the two outcomes occurs, so there is nothing to compare a predicted effect with. Chernozhukov et al. (2024, chapter 15) propose a workable substitute. Use the doubly robust proxy outcome \(\varphi\), the same quantity as in the DR-learner, as the label, and score each candidate model by its squared distance from the proxy on a scoring set that was not used to fit the candidates.

          \[ \varphi_i = \hat\mu_1(X_i) - \hat\mu_0(X_i) + \frac{D_i\,(Y_i - \hat\mu_1(X_i))}{e} - \frac{(1 - D_i)\,(Y_i - \hat\mu_0(X_i))}{1 - e}, \qquad \hat L(\tau) = \frac{1}{n}\sum_i \bigl(\varphi_i - \tau(X_i)\bigr)^2 \]

          The score of one model on its own is dominated by the noise in the proxy. What the method needs is the difference in scores between two models, which approximates the difference in their true mean squared errors. The proxy has the right average, so the comparison is accurate when either the outcome model or the propensity model is accurate, which is the double robustness of the estimator carried over to model comparison. With nuisance functions estimated on separate data, the book shows that the difference in scores is approximately normal, so it comes with a standard error and one can say whether a model is clearly better or only nominally so. The same scoring can weight several models into an ensemble.

          The second script in the code panel under the worked example uses a randomized mentoring program with probability 0.5, a true effect of 0.5 for students with a low value of one measure and 3.5 for the others, and four candidate models fitted by regressing the proxy on polynomials of the measures. They are fitted on 800 students and scored on a separate 1,000. Over 100 samples, the true error of each model against the true effect is 2.35 for a constant effect, 1.27 for a linear model, 4.18 for a cubic model, and 19.16 for a polynomial of degree 6. The score computed on the training data always prefers the most flexible model, and it did so in all 100 samples. The score on the scoring set chose the linear model in 68 percent of the samples, the constant model in 11 percent, the cubic in 15 percent, and degree 6 in 6 percent, and its choice matched the truly best model in 70 percent of the samples. The flexible model had a gap of 15.36 to the constant model on the scoring set, against −13.36 on the training data.

          Scoring has limits. It says which candidate predicts effects best on the scoring data, not that any candidate is accurate, and the calibration and BLP checks in the Expert level address that second question. The scoring set also loses its protection if it is used many times to pick among models, because it then acts as a validation set, and the final evaluation should use data that did not guide any of the choices (Chernozhukov et al., 2024).

          Generalizing

          Effects for whom

          If every student had the same effect, an estimate from one group would apply to any other group. With heterogeneity, the average effect depends on who is in the study, and an estimate that is credible for the students studied may not describe the students a district cares about (Brand et al., 2023). The same limit appears in each design in this course. A randomized trial speaks for the people and setting it enrolled. An instrument speaks for the students it moves, which is a subgroup of the population. A regression discontinuity design speaks for students near the cutoff. Brand et al. (2023) also describe a compositional problem that arises when a program expands. Early participants may be those most likely to enroll, and later ones, who differ, may benefit less.

          Several approaches try to move from the sample to a target population. One models membership, meaning how likely a student was to be in the study rather than the target population, and uses the inverse of that probability to weight the study sample so that it resembles the target. Another estimates how the effect varies with student characteristics, with a flexible method if needed, and then averages the predicted effects over the target population. Brand et al. (2023) note that weighting can work well when the covariates strongly predict both membership in the target population and effect heterogeneity, and report, citing Kern et al. (2016), that Bayesian additive regression trees extrapolate reasonably well when the observed covariates are sufficient. Both conditions are assumptions about measured variables, not results the data can verify.

          The third script in the code panel under the worked example simulates 3,000 students. A tutoring trial enrolls about 914 of them, with higher-achieving students more likely to enroll, and the effect is 1 plus 1.5 times standardized prior achievement, so the population average is 1.00. In 200 samples the trial’s difference in means averages 1.89, which overstates the population effect by 89 percent. Weighting by the inverse of the enrollment probability gives 1.05 with a standard deviation of 0.73, against 0.39 for the unweighted trial, and the effective sample size falls from 914 to 548. A regression of the outcome on tutoring, achievement, and their product, averaged over the population’s achievement, gives 1.02 with a standard deviation of 0.33. It is precise here because the simulated effect is linear in achievement, the form the regression assumes.

          The second part of the script removes the students below −0.5 from eligibility, which is 31 percent of the population, and makes the effect 3 points larger for students below −1. The population average is now 1.47 and the average among students who could enroll is 1.76. The trial’s difference in means averages 2.13. Weighting gives 1.80, which is close to the effect among eligible students, and no weighting can reach the others, because there are no enrolled students like them. The regression gives 1.03 for the population, which is 0.44 too low. It extrapolates the line fitted on eligible students to students who were absent, and the line does not hold there. An estimate for people the study could not enroll rests on a model and not on the data.

          A report should therefore say who could have been in the study, show how participants differ from the target population on measured covariates, as the balance tables of Module 2 do, and state the target population before choosing an estimator.

          Case · education

          Where does a growth mindset intervention work?

          The National Study of Learning Mindsets randomly assigned 12,490 ninth graders in a nationally representative sample of 65 U.S. public high schools to a growth mindset intervention of under an hour or to a control condition. Yeager et al. (2019) pre-registered the main analyses. For the targeted group of 6,320 lower-achieving students, those at or below their school’s median grade point average, the intervention raised core-course grades by 0.10 grade points, an effect size of 0.11.

          The study was designed around heterogeneity. Effects on students’ reported mindsets were similar across schools, but effects on grades varied across schools. They were smaller in the highest-achieving schools, and larger where peer norms supported taking on challenges, reaching 0.14 to 0.18 grade points in low- and medium-achieving schools in the top two quartiles of the norms measure. A team of statisticians, blind at the time to the hypotheses and the identities of the variables, reanalyzed the data with Bayesian causal forests. They found a near-certain positive average effect and the same pattern of moderation, which gave independent support to the pre-registered results.

          Athey and Wager (2019) analyzed a dataset simulated from a model fit to the national study, prepared for a 2018 workshop, so their numbers are not estimates for real students. Their value is the workflow. A causal forest with students clustered by school gave an average effect of 0.247 (±0.04), but omnibus tests found no clear evidence of heterogeneity. A targeted test of a pre-specified school-level moderator, the school’s average mindset, did detect variation. When the forest ignored clustering, its confidence interval was half as wide and it reported strong heterogeneity, which vanished when the results were evaluated in a way that respected the clustering. Much of the apparent difference between schools was noise.

          Check your understanding

          Scenario and diagnosis questions

          Short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Athey, S., & Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27), 7353–7360. https://doi.org/10.1073/pnas.1510489113Causal trees and honest estimation.
          • Wager, S., & Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228–1242. https://doi.org/10.1080/01621459.2017.1319839Causal forests with valid confidence intervals.
          • Künzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10), 4156–4165. https://doi.org/10.1073/pnas.1804597116The T-, S-, and X-learners.
          • Nie, X., & Wager, S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2), 299–319. https://doi.org/10.1093/biomet/asaa076The R-learner.
          • Kennedy, E. H. (2023). Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2), 3008–3049. https://doi.org/10.1214/23-ejs2157The DR-learner and its guarantees.
          • Athey, S., Tibshirani, J., & Wager, S. (2019). Generalized random forests. Annals of Statistics, 47(2), 1148–1178. https://doi.org/10.1214/18-aos1709Generalized random forests.
          • Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1), 217–240. https://doi.org/10.1198/jcgs.2010.08162BART for causal inference.
          • Hahn, P. R., Murray, J. S., & Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects. Bayesian Analysis, 15(3), 965–1056. https://doi.org/10.1214/19-ba1195Bayesian causal forests.
          • Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Yang, S. M., Carvalho, C. M., … Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364–369. https://doi.org/10.1038/s41586-019-1466-yAn education experiment centered on heterogeneity.
          • Athey, S., & Wager, S. (2019). Estimating treatment effects with causal forests: An application. Observational Studies, 5(2), 37–51. https://doi.org/10.1353/obs.2019.0001Causal forests applied to education data.
          • Knaus, M. C., Lechner, M., & Strittmatter, A. (2021). Machine learning estimation of heterogeneous causal effects: Empirical Monte Carlo evidence. Econometrics Journal, 24(1), 134–161. https://doi.org/10.1093/ectj/utaa014How the estimators compare on realistic data.
          • Athey, S., & Imbens, G. W. (2019). Machine learning methods that economists should know about. Annual Review of Economics, 11, 685–725. https://doi.org/10.1146/annurev-economics-080217-053433A survey of machine learning methods for economists, including trees and forests as neighborhood estimators.
          • Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., & Syrgkanis, V. (2024). Applied causal inference powered by ML and AI (Version 0.1.1) [Online book]. https://causalml-book.org/Chapter 15 on scoring and ensembling models of heterogeneous effects with a doubly robust loss.
          • Brand, J. E., Zhou, X., & Xie, Y. (2023). Recent developments in causal inference and machine learning. Annual Review of Sociology, 49, 81–110. https://doi.org/10.1146/annurev-soc-030420-015345Section 3 on heterogeneity and on extrapolating from an experiment or quasi-experiment to a target population.
          • Li, S., & Chu, Z. (Eds.). (2023). Machine learning for causal inference. Springer. https://doi.org/10.1007/978-3-031-35051-1An edited volume with chapters on estimating treatment effects with machine learning and on trustworthy machine learning.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

          Learn · Module 7 · Expert

          Inference on heterogeneity and policy learning.

          Machine learning estimates of individual effects are noisy, and in high dimensions their confidence intervals can be unreliable. A more robust approach asks coarser questions that can be answered with valid inference. Is there any heterogeneity? How large is the effect among the students predicted to benefit most? This lesson covers that approach, then turns to learning and evaluating rules for whom to treat, and closes with targeted learning and automatic debiasing.

          Before thisAdvanced: heterogeneous effectsMeta-learners, honesty, and forests.
          RelatedDifference-in-differences, ExpertDoubly robust and machine learning DiD.
          By the end

          What you will be able to do

          1. Estimate the best linear predictor of the effect and sorted group average effects.
          2. Explain why results are aggregated over repeated sample splits.
          3. Learn a treatment rule and evaluate it on held-out data with doubly robust scores.
          4. Describe targeted maximum likelihood and automatic debiasing in outline.
          5. Bound the bias from a hidden confounder with partial R² values, benchmark the bounds, and report a breakdown value.
          Method

          Generic inference on heterogeneity

          Chernozhukov, Demirer, Duflo, and Fernández-Val (2025) treat any machine learning estimate of the effect as a proxy, \(S(x)\), learned on one half of an experiment, and ask what can be said about the true effect on the other half. They propose three targets.

          • Best linear predictor (BLP). Regress the outcome on a baseline prediction, the centered treatment indicator \(D - p\), and its interaction with the centered proxy, \((D-p)(S - \bar S)\). The first coefficient, \(\beta_1\), is the average effect. The second, \(\beta_2\), measures how well the proxy tracks real heterogeneity, equal to 1 if the proxy is the true effect and 0 if it carries no information. Rejecting \(\beta_2 = 0\) is evidence that effects vary.
          • Sorted group average treatment effects (GATES). Divide the held-out students into groups by their proxy, for example quarters, and estimate the average effect in each.
          • Classification analysis (CLAN). Compare the characteristics of the most and least affected groups.

          Because each split of the sample gives different estimates, they repeat the analysis over many random splits, report the median, and adjust the confidence level to account for the splitting. The approach is valid whatever machine learning method produced the proxy, which is why they call it generic.

          Worked example · simulated data

          Who gains from a summer program?

          In the simulation, half of 4,000 students are randomly offered a summer program. The true effect is \(1 + 2 \times\) prior score, so the program helps most students but lowers scores for those with the lowest prior scores. The proxy is a T-learner with linear and squared terms, and results are medians over 21 splits.

          The average effect is estimated at 0.95, against a true 1.00. The heterogeneity loading is 0.94 with a t-statistic of 13.4, strong evidence that effects vary and that the proxy captures most of the variation. The least affected quarter has an estimated effect of −1.49 and the most affected quarter 3.43.

          The second script concerns the section on hidden confounding. It biases a double machine learning estimate with an unmeasured variable and bounds the bias.

          Run it yourself in R: code and output
          Method

          Learning whom to treat

          Heterogeneity matters most when it changes decisions. Kitagawa and Tetenov (2018) propose empirical welfare maximization, choosing the rule from a class of allowed rules, such as thresholds on a few characteristics, that maximizes estimated average outcomes, and they bound how much worse the chosen rule can be than the best rule in the class. Athey and Wager (2021) extend the approach to observational data by evaluating rules with doubly robust scores, which keep the guarantees when propensity scores and outcome models are estimated by machine learning.

          In the example, the rule "treat if the predicted effect is positive" treats 69% of students. Evaluated on held-out students with doubly robust scores, it improves the average outcome by 0.35 points per student compared with treating everyone. The true gain, known here because the data are simulated, is 0.39. A simple inverse-probability estimate of the same gain is far noisier, because outcomes near 50 are multiplied by weights of 2.

          Restricting rules to a simple class, such as a cutoff on prior score, keeps them transparent and reduces overfitting. It also raises questions beyond statistics. Withholding a program from students predicted to lose from it may be appropriate, but a rule that withholds it from groups defined by protected characteristics may not be acceptable whatever the estimates show.

          Related frameworks

          Targeted learning and automatic debiasing

          Two related frameworks reach similar goals by other routes. Targeted maximum likelihood estimation (van der Laan & Rubin, 2006) starts from a flexible initial estimate of the outcome model and updates it in a direction chosen so that the resulting plug-in estimate of the effect solves the efficient estimating equation. It is widely used in epidemiology and pairs naturally with ensembles of learners.

          Automatic debiased machine learning (Chernozhukov, Newey, & Singh, 2022) addresses a practical obstacle. Orthogonal scores require a correction term, such as inverse propensity weights, whose form depends on the effect being estimated. Their method learns that correction directly from the data, which extends double machine learning to effects whose correction terms are hard to derive by hand.

          Sensitivity

          How much hidden confounding would it take?

          Double machine learning removes the bias that comes from controlling flexibly for measured variables. It does nothing about a confounder that was not measured. Chernozhukov et al. (2024, chapter 12) work out what that costs in a partially linear model. A hidden variable \(A\), scaled so that the variance of its error is 1, affects the outcome with coefficient \(\delta\) and the treatment with coefficient \(\gamma\). After the measured controls \(X\) are partialled out, the regression of the outcome on the treatment recovers the effect plus a bias \(\varphi\).

          \[ Y = \alpha D + \delta A + f(X) + \varepsilon_Y, \qquad D = \gamma A + g(X) + \varepsilon_D, \qquad \beta = \alpha + \varphi, \qquad \varphi = \frac{\delta\,\gamma}{\gamma^2 + E[\varepsilon_D^2]} \]

          The bias is the product of the two effects of \(A\), scaled by the variation in the treatment that remains after the controls. A second form, adapted from Cinelli and Hazlett (2020), expresses the bias through partial \(R^2\) values.

          \[ \varphi^2 = \frac{R^2_{\tilde Y \sim \tilde A \mid \tilde D}\; R^2_{\tilde D \sim \tilde A}}{1 - R^2_{\tilde D \sim \tilde A}} \cdot \frac{E\bigl[(\tilde Y - \beta \tilde D)^2\bigr]}{E\bigl[\tilde D^2\bigr]} \]

          The first two factors describe the hidden variable and cannot be estimated. The last factor can be estimated from the residuals. The practical step is to assume bounds on the two \(R^2\) values, which bounds \(|\varphi|\), and to widen the interval for the effect by that bound on each side. The book proposes benchmarking the bounds on the measured controls. The reasoning is that the controls were chosen to capture confounding, so what remains is likely to be weaker. The authors caution that controls are often chosen for convenience, so the argument should be treated as a way to calibrate, not as a guarantee (Chernozhukov et al., 2024).

          The second script in the code panel under the worked example puts the idea to work with 1,000 students. Tutoring hours raise a score by 1.0 per hour. An unmeasured motivation variable raises hours by 0.5 and the score by 2.0, so the bias is 0.8. The cross-fitted estimate averages 1.79 over 200 samples, with a standard error of 0.10, and the conventional interval contains the truth in none of them. The hidden variable's true \(R^2\) values are 0.20 for hours and 0.26 for the score given hours, which the analyst would not know. For all measured controls together they are 0.33 and 0.34. The hidden variable is thus roughly 60 to 75 percent as strong as the controls, and it produces a bias equal to 80 percent of the true effect.

          The bounds follow from assuming that the hidden variable is \(k\) times as strong as the controls. For \(k = 0.25\) the average bound is 0.29 and the interval contains the truth in 1 percent of the samples. For \(k = 0.5\) it is 0.60 and 56 percent, and for \(k = 1\) it is 1.34 and all of the samples. Coverage is only as good as the benchmark, and here the true strength lies between the second and third. A breakdown value gives another summary. A hidden variable with the same \(R^2\) for hours and for the score would have to reach 0.42, with a range of 0.32 to 0.50 over samples, to bring the estimate to zero.

          The analysis is not a test and it does not show that confounding is absent. A report states the benchmark, the strengths considered, the resulting range for the effect, and the breakdown value, and lets readers judge whether a hidden variable that strong is plausible. The book provides R and Python notebooks for this analysis of a double machine learning estimate.

          Practice

          What to report

          1. The learners, their tuning, and the number of sample splits.
          2. The BLP coefficients and a test of no heterogeneity.
          3. GATES with confidence intervals, and the characteristics of the extreme groups.
          4. For any targeting rule, its class, how it was chosen, and its value on held-out data.
          5. A statement of which subgroup analyses were planned in advance.
          Check your understanding

          Scenario and diagnosis questions

          Short scenarios in the style of a study you might read or run. Pick an answer to see why each option is right or wrong. Questions you miss come back in the review box the next time you open this lesson.

          Exercises with solutions

          Practice

          Annotated reading list

          Where to read more

          • Chernozhukov, V., Demirer, M., Duflo, E., & Fernández-Val, I. (2025). Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in India. Econometrica, 93(4), 1121–1164. https://doi.org/10.3982/ecta19303BLP, GATES, and CLAN with repeated splitting.
          • Kitagawa, T., & Tetenov, A. (2018). Who should be treated? Empirical welfare maximization methods for treatment choice. Econometrica, 86(2), 591–616. https://doi.org/10.3982/ecta13288Empirical welfare maximization.
          • Athey, S., & Wager, S. (2021). Policy learning with observational data. Econometrica, 89(1), 133–161. https://doi.org/10.3982/ecta15732Policy learning with doubly robust scores.
          • van der Laan, M. J., & Rubin, D. (2006). Targeted maximum likelihood learning. International Journal of Biostatistics, 2(1), Article 11. https://doi.org/10.2202/1557-4679.1043Targeted maximum likelihood estimation.
          • Chernozhukov, V., Newey, W. K., & Singh, R. (2022). Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3), 967–1027. https://doi.org/10.3982/ecta18515Automatic debiased machine learning.
          • Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097Double machine learning.
          • Chernozhukov, V., Hansen, C., Kallus, N., Spindler, M., & Syrgkanis, V. (2024). Applied causal inference powered by ML and AI (Version 0.1.1) [Online book]. https://causalml-book.org/Chapter 12 on unobserved confounders, with the bias formula and sensitivity analysis for double machine learning.
          • Cinelli, C., & Hazlett, C. (2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1), 39–67. https://doi.org/10.1111/rssb.12348Omitted variable bias expressed through partial R² values, benchmarking, and robustness values.

          Last revised November 2026. Numbers in the code examples come from code run in R, Python and Stata.

            Start hereQuiz 0/6 · Exercises 0/3