Module 1 Randomized experiments
- When to use it
- Use it when you control who receives the program and can assign it by chance, for example a lottery for tutoring places or a trial that randomizes schools.
- What it estimates
- The effect of being assigned to the program, compared with not being assigned, for the students or schools in the study. When some assigned students do not take part, it also estimates the effect for compliers, the students who take part only if assigned.
- What must be true
- Assignment is random and carried out as planned.
- One student's assignment does not change another student's outcome.
- Outcomes are measured the same way, and as completely, in both groups.
Chance makes the groups comparable in expectation, including on characteristics nobody measured, so the control group shows what would have happened to the treated group. A single trial can still be unbalanced by chance.
- What to check
- Compare response rates between groups, because attrition that differs can reintroduce bias. Check that the analysis follows the design, for example by school when schools were randomized. Baseline balance tables describe chance imbalance but do not test whether randomization worked.
Estimators and technical notes
- Estimators taught
- Difference in means with Neyman standard errors, regression adjustment and randomization tests (Intermediate). Cluster-level analysis and power (Advanced). Intention to treat, complier effects and Lee bounds (Expert).
- Notes
- Formal result Under complete randomization, regression with a centered covariate interacted with treatment is, in large samples, at least as precise as the difference in means (Lin, 2013).
- Design guidance When schools are randomized, inference must be at the school level, and with few schools it needs small-sample corrections.
- Active methods research Whether a result applies to your target population. Moving from the study sample to other schools needs information about who differs and assumptions about how effects vary.