Presenter: Arnaud Vaganay
Abstract: In most welfare states, policy evaluations are sponsored by the very organisations that designed and implemented the intervention in the first place. Research in the area of clinical trials has consistently shown that this type of arrangement creates a moral hazard and may lead to overestimates of the effect of the treatment. Yet, no one so far has investigated whether social interventions were subject to such ‘confirmation bias’.
The objective of this research series was twofold. Firstly, it assessed the scientific credibility of a sample of government-sponsored pilot evaluations. Three common research prescriptions were considered:
- (a) the proportionality of timescales,
- (b) the representativeness of pilot sites; and
- (c) the completeness of outcome reporting. Secondly, it examined whether the known commitment of the government to a reform was associated with less credible evaluations.
These questions were answered using a ‘meta-research’ methodology, which departs from the traditional interviews and surveys of agents that have dominated the literature so far. The PILOT dataset was developed for that specific purpose. PILOT includes data systematically collected from over 230 pilot and experimental evaluations spanning 13 years of government-commissioned research in the UK (1997-2010) and four government departments (Department for Work and Pensions, Department for Education, Home Office and Ministry of Justice).
The results show some evidence of bias but no obvious association between bias and policy commitment. Policy recommendations are provided, as this project was severely hindered by important ‘black box’ issues and by the poor quality of evaluation reports.