PIRU aims to carry out methodological projects, either on its own or as part of a particular evaluation, which will contribute to the development of better methods for policy evaluation. Almost every evaluation of pilots of a new approach has to tackle the potential for selection bias when comparing the new with the previous arrangements. This raises issues at each stage of the evaluation, including how to select pilots and controls, what to measure, for how long, which analytical methods to choose, and how to present the results. Our work aims to illustrate the methodological alternatives faced at each stage, and to consider the potential impact on the results of alternative methodological choices.
We have also looked more widely at the specification, design, execution and use of findings from evaluations of a number of high profile pilot programmes commissioned by the Department of Health in recent years in order to examine how, and for what purpose, recent pilots were initiated and how decisions about their organisation have influenced opportunities for evaluation.
One project looked at approaches for evaluating the effectiveness and cost-effectiveness of continuous treatments (e.g. level of financial incentives) rather than treatments defined as a binary variable. Our project compared regression and propensity score approaches for addressing selection bias when evaluating continuous treatment effects. We also used a machine learning approach to estimate both the propensity score and the parametric dose-response function. We compared this approach to parametric implementations in re-analysing a published evaluation of alternative ways of organising neuroscience services for patients following acute traumatic brain injury. The study compared mortality rates for an intervention group who had "early transfer" to a specialist neuroscience centre with those who had "no or late transfer". We concluded that machine learning is an attractive approach for estimating the effects of continuous treatments in policy evaluation.
A second project examined the use of a synthetic control approach for evaluations with a quasi-experimental design. This method was compared with the difference-in-differences (DiD) method in estimating the effect of a pay-for-performance (P4P) scheme called the Advancing Quality (AQ) programme in North West England. Both methods aim to reduce selection bias due to confounding in the absence of randomisation, but the synthetic control method, unlike DiD, allows variation of potential confounders over time. The effect of the AQ on 30 day in-hospital mortality was re-estimated for conditions incentivised by the AQ, as well as for non-incentivised conditions using 28 quality measures including mortality and readmission rates. In contrast to the DiD analysis, the synthetic control method found that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality, and that there was a statistically significant increase in mortality for non-incentivised conditions.
The outcomes for the project looking at the purpose of DH policy pilots can be found here >>
A working paper on the use of machine learning for estimating the effects of continuous treatments was presented at the University of York in August 2014 and may be accessed here >>
The article "Evaluation of the effect of a continuous treatment: a machine learning approach with an application to treatment for traumatic brain injury" was published in Health Economics in September 2015 and may be accessed here>>
The article "Examination of the synthetic control method for evaluating health policies with multiple treated units" was published in Health Economics in October 2015 and can be accessed here>>