Knowledge Bank

Difference-in-Difference

January 04, 2022, Evaluation Observatory

Consider the case of a road repair program whose objective is to improve the population's access to labour markets. An outcome indicator to measure this would be the employment rate within the population. If we simply observe before and after changes in employment rates, it is unlikely, that changes pre and post the road repair program would give us an idea of the attributable impact, as several other factors - such as education levels, and access to networks - also known as confounders, are likely to influence the employment rate (Gertler Paul et al., 2012, p. 130) . And, although RCT's or experimental methods are considered the gold standard design to infer causal attribution; given the treatment was not administered through random assignment, neither would an RCT be possible.

In this scenario, a difference in difference approach becomes useful.

The difference-in-difference design is quasi-experimental design that compares the changes in outcomes over time between a population enrolled in a program - i.e. the treatment group - and a population that is not i.e. the comparison group1. The comparison group is usually constituted using 'matching methods' i.e. matching the counterfactual with the treatment., on a set of defining key parameters, for instance, gender, education levels etc., to adjust for confounding effects.

First, we calculate the difference in the before-and-after outcomes for the enrolled group, which controls for factors that are constant in the group, over a period of time. Next, we calculate the difference in the before-and-after outcomes for the non-enrolled group, which captures the factors that vary over time. If we take the difference of these differences in outcome indicators, we would be able to accurately measure the impact by eliminating bias.¹

Diff-in-diff approach compares trends between the treatment and comparison group instead of comparing outcomes.¹ This can be understood with the help of the following diagram.

• 

Source: (Eric, 2019)²

An essential assumption of this approach is that differences between the treatment and comparison groups are constant over time, and hence it is crucial to test the validity of the equal trends assumption in Difference-in-Differences.¹ For a comprehensive understanding of the topic, please refer to recommended resources given below.

For a broad overview

• 
• Blog Post: Difference-in-Differences by World Bank

  Published by World Bank, this resource gives an overview of the method and its implementation. It also highlights the key-assumptions of this method.

• 
• Guidance Document: Quasi-experimental methods: Difference in differences

  Presented by Prashant Bharadwaj of the University of California, this resource explains the diff-in-diff method using the Change in Marriage Laws in Missisipi, USA on education, fertility and marriages, as a case study.

• 
• Blog Post: Introduction to Difference-in-Differences Estimation

  Published by APTECH, this resource explains the concept of Difference-in-Differences Estimation and illustrates the use of this method with the help of case examples.

• 
• Working paper: Impact Evaluation using Difference-in-Differences

  Written by Anders Fredriksson, Gustavo Oliveira, this paper presents the Difference-in-Differences method in an accessible language to a broad range of management scholars interested in applying impact evaluation methods. The paper describes the method, assumptions, and regression specifications using numerous examples.

• 
• Guidance Document: Differences?in?Differences (using Stata)

  This document by Princeton University is a step-by-step guide on using Stata for Difference-in-Difference. It guides practitioners on using various commands in Stata for estimating impact. You can view a similar document for the commands using R, here: Differences?in?Differences (using R)

• 
• Video: The Difference in Differences (DD)

  Developed by the Partnership for economic policy (PEP), this video series gives a conceptual overview of the diff-in-diff method and explains the various underlying assumptions required for the technique's validity. It also illustrates the estimation of the diff-in-diff method using an example.

• 
• Video: Difference-in-Differences method for policy evaluation

  Developed by EU Science Hub, this video explains why diff-in-diff is a powerful 'Counterfactual Impact Evaluation' method to decide the most effective projects which give good value for money.

• 
• Video: Differences in Differences Animation (Beginner)

  Published by Empiricist Academy, this animation illustrates how to obtain the differences-in-differences estimate using a regression model.

• 
• Videos: Difference in Differences by Andrew Heiss

  Presented by Andrew Heiss, this video series provides an introduction to the method and its various assumptions. It also consists of a series ofhas five lecture on using R for difference in difference method

For in depth understanding

• 
• Book Chapter: Impact Evaluation in Practice

  Published by World Bank group, Chapter 7 of the book has detailed discussion of the Difference-in-Difference method. It gives the rationale to the use of the method, its utility, process and assumptions. It also highlights the methods limitations and provides practitioners with a checklist for effective impact measurement.

Case Study

• 
• Social Protection: Impact Evaluation Report: Testing a Gender-Transformative Approach from Asia in Africa

  Published by Care Evaluations, this case study evaluates the differences in outcomes and processes of the gender-transformative EKATA approach compared against a standard “Gender Light” approach in the outcome areas of gender equality, and food security and economic well-being.

• 
• Agriculture: Difference in Difference Method to Impact Evaluation: A Case Study of Fruiting Africa Project

  Written by Ken Njogu Mwangi, this paper evaluates whether Fruiting Africa Project (FAP)--an agroforestry project implemented by World Agroforestry Centre (ICRAF), made a difference in the livelihoods of beneficiaries using the difference in difference method.

• 
• Health: The Difference-in-Difference Method: Assessing the Selection Bias in the Effects of Neighbourhood Environment on Health

  Published by National Centre for Biotechnology Information, this paper uses the difference-in-difference estimation approach to explore the self-selection bias in estimating the effect of neighbourhood economic environment on self-assessed health among older adults.

Further Reading

• 
• Working paper: How Much Should We Trust Differences-In-Differences Estimates?

  Published by the National Bureau of Economic Research and written by Marianne Bertrand, Esther Duflo, Sendhil Mullainathan, this paper explains bias in the estimated standard errors introduced by serial correlations and discusses two simple techniques to solve the issue.

• 
• Working paper: Inference with "difference in differences" with a small number of policy changes

  From the National Bureau of Economic Research and written by Timothy Conley, Christopher Taber, this paper provides a new method for inference in Difference-in-Difference models when there are a small number of policy changes. The paper demonstrates the diff-in-diff approach by analysing the effect of college merit aide programs on college attendance and shows that the standard approach can give misleading results in some cases.