Skip to content


Academics can show governments how to evaluate SIBs more rigorously

23 April 2018
By Chris Fox

A wide range of approaches can help identify causality and effectiveness even in complex environments.

We can – and we should – improve our evaluations of SIB and Payment By Results (PBR) programmes. They should focus more on causality, rather than simply contract compliance or implementation.

If we don’t focus on attribution, it will become hard to demonstrate that SIBs are more than a series of interesting pilots. We’ll miss the chance to test an alluring proposition – that SIBs could transform the large scale commissioning and delivery of health and social welfare programmes.

Getting evaluation right in this field is not - as some might suggest – intrinsically challenging. SIBs and PBR projects do not create unusual difficulties for evaluation techniques. We have the knowhow - sophisticated, diverse tools are well-developed that could settle most questions thrown up by SIBs. The real issue is: will those who champion SIBs expose such initiatives to the full rigour of the evaluative tools that exist?

Academic responsibility

The academic community can help ensure that rigour. Management consultants, contracted to perform evaluations, tend to provide what governments specify, which, so far, has been limited and fallen short of what’s required. Academics could set out a wider, more exacting range of evaluation options that are more suitable. We should show policy makers clearly how better evaluations could be achieved, particularly if the case for widespread adoption of SIBs is to be made.

This difficulty in properly assessing the impact of SIBs seems to be a particularly British problem. In the United States, most SIBs have been accompanied by fairly rigorous counterfactual evaluations, including randomised control trials (RCTs). There, the credibility of the SIB model among commissioners and investors has required demonstration of its ability to deliver tangible outcomes. This may be because, in the US, more funding has come from wealthy individuals or private foundations, with an investment ethos. In Britain, funding tends to spring from philanthropic organisations which seem interested in testing concepts over categorical outcomes.

Evaluations are too based on performance management

Whatever the reasons, SIB pay-outs in the UK typically rely more on performance management information to demonstrate the achievement of outputs. Supporters of this approach say that complicated counterfactual evaluations add to the already high transaction costs associated with SIBs. That’s understandable for individual SIBs. However, cumulatively, this approach hinders the quest to find out whether SIBs really work. It undermines the case for wider roll-out.

Evaluations can and should answer two major questions about SIBs. There’s “attribution”: whether SIBs actually achieve the outcomes desired. Second, we need to understand SIBs as a mechanism and establish how effective they are compared with other models of commissioning. This is important because there are less expensive, less complicated methods than SIBs for commissioning services in this field.

The attribution issue has become unnecessarily mired in a polarised debate about whether RCTs are suitable for SIBs projects. Opponents contend that RCTs are not particularly useful in this field because SIBs interventions tend to take place in highly complex environments. While it’s true that these interventions often occur amid complexity, that actually strengthens the case for RCTs. It becomes even more important to understand whether an intervention is indeed responsible for any of the impacts being observed.

Testing theories of change

Good RCTs would strengthen SIBs evaluations because they would be theoretically informed. They would start with a theory of change setting out the potential causal mechanisms that are of interest. In contrast, many SIBs evaluations rely on contractual frameworks and demonstrating whether they have worked, rather than testing hypothesised causality. Most good RCTs today are also accompanied by high quality implementation evaluation. So they have a dual strategy.

Well organised RCTs avoid “one-shot” design. They are actually a sequence of evaluations that build by testing, at a granular level, particular moderators of change, rather than simply focussing on the overall social outcome and trying to come to a one-shot conclusion. This is how, in reality, even medical research works. You don’t do a single RCT. You build from small scale studies through to larger scale studies.

Sequences of evaluations are good

The wider evaluation world is focussing more on sequencing evaluations and ensuring that tools employed are appropriate to the point of a programme’s development. This avoids problems that one shot evaluations can create: that you evaluate too early; that the throughputs you were promised never arrive; that you end up developing an evaluation design which is underpowered to identify the changes that you’re looking for; there are inconclusive findings that have cost a lot of money but don’t provide the hoped for insights.

I advise against the one shot model. Instead, we like to start evaluations early without diving straight in with an RCT. We focus on developing a sequence. That’s the strength of the Education Endowment Foundation evaluation model. It begins with small scale pilot studies that focus on theory of change and early implementation, then efficacy trials that are more like a small RCT, leading up finally to effectiveness trials. Only at that point - when causalities have been established - is control finally handed over to implementers.

Building commissioner confidence

This sequential approach gives commissioners confidence. You’re saying to them that this isn’t a “one shot, put all your money on the table up-front” model. It’s about gradually building knowledge and providing gate-keeping points where a commissioner can ultimately say: “This isn’t working, we need to rethink. We may need to reinvest or, even disinvest.” That’s helpful to commissioners, especially if they are being asked to back innovation that feels risky.

Small ‘n’ designs

In some cases, RCTs are not possible, but there are many alternative models of impact evaluation that could be considered for SIBs. “Small n” designs provide ways to think about causal attribution where a programme does not have sufficient numbers to allow a traditional impact evaluation design. Process tracing is an example of “small n” design, where one uses theory to identify critical points in a change process that need to be tested. Then one selects cases to test these critical points, using interviews and observations of what’s going on. This Popperian approach acknowledges that there is no absolute objective knowledge. However, it can find ‘smoking gun’ evidence that strongly suggests causality, even if that may not amount to absolute proof.

These process approaches that search out causality would be an improvement on current tests of some SIB or PBR programmes which, if they can’t do an RCT, tend to opt for process/implementation evaluations that are less demanding - usually interviewing stakeholders and writing a report, but lacking a more theory-driven approach.

More rigour is needed

I’ve set out ways in which SIB and PBR evaluations could be improved by RCTs or hybrids that avoid the unnecessarily polarised debate between the pro- and anti-RCT lobbies. Beyond RCTs, there are other approaches to evaluating causality, suitable in instances where there are small numbers of cases. We should learn from this wider discussion of evaluation techniques. Academics owe it to those investing and working in SIBs to ensure that policy makers adopt a rigorous approach to evaluation. We need to know what works and what doesn’t if SIBs are ever to be widely adopted.

Chris Fox is Professor of Evaluation and Policy Analysis and Director of the Policy Evaluation and Research Unit at Manchester Metropolitan University. He is co-author of “Payment by results and social impact bonds: Outcome-based payment systems in the UK and US”, published by Policy Press in February 2018.

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

Impact bonds could offer a paradigm shift towards more effective public services

16 April 2018
By Emily Gustafsson-Wright

Social and Development Impact Bonds require enormous effort for the partners involved, but they have a potential to transform the financing and delivery of social services across the globe.

In winter 2015, Courtney arrived at Frontline Services, a not-for-profit US organisation that helps citizens in Cleveland, Ohio. She was 28, living in a shelter for homeless women, struggling with mental health and substance abuse issues and parenting three young children who were in the custody of the county.

Courtney had just about given up hope that she would ever be able to care for her children on her own. Until that point, the county caseworker assigned to her family had little incentive to reunite Courtney and her children because the caseworker’s primary job was to protect the children.

Living with a birth parent is almost always better for a child’s development than foster care, provided the home environment is safe and healthy. Nevertheless, before entering Frontline Services, Courtney had few ways to change the trajectory of her children’s lives. Fortunately for her children, she was walking that day into a social services experiment, one of only seven similar experiments in the US at the time. In this experiment, a social impact bond (SIB) – designed to “pay for success” - the county’s government had pledged to repay private investors for successful reductions in out-of-home placements for children whose primary caregivers were homeless.

This incentive meant that Courtney was assigned a caseworker dedicated to her - someone who would look at her particular circumstances and tailor a plan to help her turn her life around and unite her with her children. Courtney’s caseworker could work across county service providers to identify the right mix of services for her.

The SIB meant that a dedicated group of stakeholders was meeting regularly across government and non-government entities to focus on one thing - reuniting Courtney with her children, and doing the same for other families in similar circumstances.

SIB contracts focus on outcomes, so service providers tailor their services to what works for the target population. They helped Courtney to address her debts with classes in financial management and offered family counselling.

As a result, Courtney was able to reunite with her children, enrol them in supportive school environments and stop the cycle of dependency on the foster care system. The result was not only a better family outcome, but also a reduction in the enormous costs to the county, had Courtney’s children remained in the county’s care.

Impact bonds are changing developing countries

Meanwhile, nearly 12,000 km away, in a village in rural Rajasthan, India, lives a 13-year-old girl, named Punam. She comes from a poor family - her parents are labourers. Although Punam started school at age seven, she became one of India’s three million out-of-school girls, when she was forced to drop out to tend to her family’s goats.

In the same year as Courtney arrived at Frontline Services in Cleveland, Ohio, a field co-ordinator, working for an organisation called “Educate Girls”, arrived in Punam’s home in Rajasthan. He spoke with her parents, explained the benefits of educating Punam and tried to convince them to send her back to school.

Even after multiple attempts, the parents didn’t agree to send Punam back to school. The Educate Girls caseworker returned some weeks later with a volunteer from the community but again failed to persuade the parents. “What benefit will it give her or us?” they asked. “She will eventually marry and her responsibilities will revolve around doing household chores, assisting in farming life, raising children and taking care of her family.”

Nevertheless, Educate Girls made a further attempt to encourage her parents to let Punam attend school. This time, they asked the school’s principal to join them in a final visit to her home. With this added influence, Punam’s parents agreed finally to sending their daughter to school.

Why was an impact bond so important in this case? Because the contract was based on the achievement of outputs and outcomes, Educate Girls field-staff were empowered to innovate at the field level, trying to find solutions for getting Punam into school. Now, two years later, Punam, and many girls like her, are enrolled in and enjoying school thanks to Educate Girls and this Development Impact Bond (DIB), based on the same principle as a SIB, but with a third-party outcome funder, instead of the government.

These two stories capture the real human benefit that can emerge from outcome-based contracts such as SIBs or DIBs.

How impact bonds work

Let’s just re-cap for a moment how impact bonds actually work. In an impact bond, private investors supply upfront capital to service providers to deliver an intervention or program to a population in need. Upon the achievement of a set of agreed-upon results, the investors are then repaid by an outcome funder. With a SIB, this outcome funder is the government. With a DIB, outcomes are financed by a third-party organisation, such as a foundation or donor.

Since the launch of the first SIB in the UK in 2010, the impact bond market has grown exponentially. Last year, some 32 new contracts were signed. As of January, 2018, there were 108 contracted impact bonds (103 of them SIBs, 5 of them DIBs) across 25 countries, along with many more in design. All but one of the 103 SIBs were in high income countries: last year marked the contracting of the first SIB in a low- or middle-income country, the Workforce Development SIB in Colombia.

Most (42) SIBs are in the UK, the country that pioneered the impact bond model with the Peterborough SIB in 2010. The results of that SIB – aimed at rehabilitating ex-offenders - were released last year: reoffending of short-sentenced offenders dropped by 9 percent and the investors were repaid in full. The US has also established itself as a player in the field, coming in second with 19 impact bonds.

The five DIBs include Educate Girls in India, one for coffee and cocoa production in Peru, as well as one for physical rehabilitation across three countries in West Africa, a poverty graduation program in Kenya and Uganda, and the recently launched Utkrisht impact bond for Maternal and Newborn Health in Rajasthan, India.

Characteristics of SIBs

Most SIBs contracted globally are in the employment field, followed by the social welfare sector, which includes programmes to reduce homeless – “rough” – sleeping, or reduce out-of-home placements as in the case of Courtney and her children. Other areas for SIBs are health, criminal justice, education, the environment and agriculture.

Probably about 30 or 40 impact bonds are in design in high-income countries while more than 20 are being designed in low and middle-income countries. We see some difference comparing high income and low or middle-income countries. The majority of impact bonds in the latter are in the health sector, followed by employment and, then, agriculture.

What do these impact bonds look like in terms of size? The smallest one, in terms of beneficiaries, reaches 22 individuals – that’s in Canada. The largest one reaches 650,000 individuals in Washington DC, which is an environmental impact bond focusing on developing infrastructure. It is perhaps a little bit unfair to compare that one in terms of size with the rest of them because it’s a city-wide programme. The next largest in size is the Maternal and Newborn Health DIB in Rajasthan, India, with 600,000 potential beneficiaries.
However, the median impact bonds are reaching about 565 individuals, so they are quite small. Capital commitment of bonds ranges from $80,000 to $25 million. Again, that $25 million is the one in DC. The average is about $4 million and the total upfront capital invested across the impact bonds is over $300 million.

Who is benefiting?

The bonds mostly target marginalised populations, including women affected by violence, young migrants, single mothers, with a few for ex-convicts, vulnerable and young people, people diagnosed with mental health conditions, refugees and individuals with physical disabilities.
What do we know, eight years in about their performance? There have been some outcomes achieved and payments made, such as in the case of the Peterborough SIB, mentioned earlier. In an Australian SIB, 203 children were reunited with their families and the return to investors was nearly 16.5 per cent over the four years of the scheme.

Shifts in public programme behaviour

However, perhaps the more interesting observation is a real shift seen in government and service providers to thinking about outcomes as opposed to paying for inputs. Impact bonds are also driving performance management so service providers are introducing or improving systems of performance management in their programmes.

Impact bonds are incentivising collaboration, between the public and private sectors, but also across government, vertically and horizontally. They are building a culture of evaluation because outcomes must be measured and monitored. Most impact bonds so far have been focused on investment in preventive interventions. There has also been some reduction in risk for governments, which have not paid for outcomes that weren’t achieved.

What are we not seeing so far? It had been hoped that impact bonds would lead to an influx of additional private funding. However, given that government or outcome funders ultimately repay the investors, then that’s not really more money for a particular social service. Impact bonds have also yet to achieve change at scale: the majority are reaching very few individuals and are fairly small in terms of investment.

Many thought that impact bonds would focus on experimental interventions. So far, we haven’t seen that: investors have been unwilling to take that risk. We’re seeing SIBs used in the middle phase of development of interventions, rather than at the “seed” or “at scale” ends of the process. However, the flexibility that service providers are allowed in terms of their service delivery has the potential to encourage innovation. It’s also probably too soon to say whether or not impact bonds can achieve sustainable outcomes in the long run through the systematic change that’s happening but it does appear that the partners currently involved have indeed shifted their thinking.

Challenges of impact bonds

What are the challenges? This is a new form of government contracting, a new way to do business. Co-ordination of all of the stakeholders is difficult: sometimes they don’t understand each other well; just getting all those people around the table can be really difficult. There can be some political constraints and legal barriers.

Key questions remain. Can impact bonds be used at scale? Are they more effective than input-based based financing or traditional payment-by-results? Do the actors in social service provision have the capacity to adapt to the demands of financing tied to results? Can they manage the rigorous focus on performance management that this is likely to entail?

Next steps

It is worth considering what would be needed to expand the use of impact bonds or, more generally, payment by results. The evidence base needs to grow and there is a need to collect more information on services that work and on their costs. Also, potential outcome funders and investors need to be educated in not only the potential of an impact bond approach but also the challenges. There needs to be supporting legislation and regulations to facilitate paying for outcomes both at a country and local level but also within organisations.

To achieve scale, countries could establish outcome funds for particular sectors. The UK government has launched seven outcomes funds and efforts are underway in the US to develop outcomes rate cards. There are several outcome funds being developed to tackle tough social issues in the developing world as well. These would allow the outcome funder to set prices for desired social outcomes and then to contract with service providers to deliver interventions to achieve those results. Global investment funds would also benefit from contributing to this new financing mode.

In the US, $800 billion are spent annually on social services. Only one per cent of that spending is evaluated for effectiveness. In the UK, £220bn was spent on social and health services (2015/2016), yet we know very little about the effectiveness of that expenditure. Thus we need more empirical research which asks: “What do impact bonds achieve, compared with input based financing?” It is also important to know how well impact bonds perform compared with traditional results-based financing. These are both hard questions to answer rigorously and will take some time.

Impact bonds and global problems

The social and environmental problems that we face at a global level are enormous. It’s estimated that $1.4 trillion will be needed annually to achieve the Sustainable Development Goals by 2030. There is little evidence that such complex problems can be solved by continuing the same old failed approaches.

Investing in preventive measures can avoid higher costs down the road and make the public and civil society sectors more efficient. By paying for outputs and outcomes rather than paying for inputs that have unknown outcomes, spending should be more effective.

Impact bonds may not be the right solution to every problem. However, they do represent a long-overdue, paradigm shift. They’re a means to an end, an opportunity to think about, and hopefully produce, systematic change. At the very least, they may be the stepping stone to establishing the monitoring and evaluation performance standards and output planning that can ensure every individual receives the services that they need to live safe, healthy and productive lives.

Dr Emily Gustafsson-Wright is a Fellow in the Global Economy and Development Program at the Center for Universal Education, Brookings Institution, Washington DC.

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

“Rethink questions to patients in general practice and focus more on improving primary care”

20 July 2017
By Tommaso Manacorda

The “Friends and Family Test”, seeking patients’ views, has created anxiety among practices but shed little light on patients’ concerns. It’s time for a rethink, suggests our study.

The many ways in which patients can now feedback to GPs should offer a rich source of information for those keen to improve their services. However, when we evaluated the latest addition to patient consultation in England – the “Friends and Family Test” of patient satisfaction – we found that “more” did not necessarily equate with “better”.

We found that general mistrust of the FFT process, combined with inappropriate framing of its core question is undermining the initiative. But we also identified ways both to bolster trust and make patient feedback more useful for improving primary care.

Two important pieces of learning emerge from examining this initiative. First, the issues raised by the “Friends and Family Test” could usefully prompt a rethink about how views are gathered from patients about their primary care experiences. Our main recommendation is that the FFT should be revised, although discontinuing it should also be considered, as its reception within general practice has been largely negative.

Second, GPs need more encouragement and guidance to tackle inadequacies revealed by patients’ experience. We found that commitment to quality improvement is uneven across primary care and remains a low priority in some, often in those practices where patient participation groups (PPGs) meet rarely and patient surveys are infrequently conducted. Nurturing a rigorous improvement culture is at least as important as getting right the questions posed to patients.

Wording of the FFT

The “Friends and Family Test” puts a single question to patients about their general practice: “We would like you to think about your recent experience of service. How likely are you to recommend our GP practice to friends and family if they needed similar care and treatment?” Answers are recorded on a five point scale from “extremely likely” to “extremely unlikely”. Additionally, patients may be asked to comment on their reasons for the score they have given.

General practices across the NHS are required to make the FFT available to patients after every contact, collecting the data with the method that suits them best. The majority of them use handwritten cards, but tablet kiosks and online apps are also used.

Our evaluation, involving 42 practices and 118 interviews with clinicians, practice staff and patients’ representatives, found two sets of problems with the FFT. The first concerned the usefulness of the information it produced. The second problem lay with how general practices understood the FFT’s purpose and how they engaged with it.

The experiences of hospitals, where the FFT was first introduced, showed that the quantitative scores were statistically unreliable. So the metrics could not be used to compare providers for quality. That’s because the FFT does not involve a representative sample and is vulnerable to selection bias. One can’t tell whether the scores are representative of all patients. So, when the FFT question was subsequently rolled out in 2014 into primary care, the widespread awareness of these statistical limitations contributed to unease in general practices about the FFT approach.

In our study, the FFT question was deemed inappropriate by most interviewees. Many, particularly in rural areas, lacked choice of general practice, so the question of where they might send friends or family lacked reality. Also, patients found it difficult to compare their personal care with what someone else might receive, because that depends on individual factors such as age, sex and existing health conditions. Most interviewees suggested the use of a more straightforward question.

The additional space provided for further comment did potentially offer some useful feedback. However, staff in general practices felt that the anonymity of patients made it difficult to act on these comments. That’s because patients were often unclear which service they had received, from whom and the precise nature of issue that concerned them. For example, a patient complained about a “terrible phone service”. But staff said it was difficult to respond to this statement because they didn’t know who the complainant was, whom the patient had spoken to or what they needed.

Most patient feedback collected by practices was positive, but anonymity and vagueness made it difficult to identify and reward good practice. Positive feedback would often lift staff morale. However, the inability to act on complaints was often reported to be very frustrating for staff.

When additional comments gathered by the FFT pointed at some specific issues, their contribution was still considered to be of little value because the issues were already known to staff from other sources, such as practice surveys and patient participation groups.

Professionals mistrust the process

The second issue with the FFT concerned the negative reception that it received from primary care professionals. Some feared that it left them vulnerable to hostile patients. Someone might, for example, be correctly denied an antibiotic for sound clinical reasons but then score down the practice unfairly. The practice would have no way to question the scoring.

Monthly reporting of scores to the DH and NHSE added to professional concerns that the FFT process was, as some claimed, “a stick with which to beat General Practice”. Even though the Government had stated that it would not use FFT scores to rank practices, that fear of unfairness remained unassuaged, which disheartened hard-working professionals.

As a result, although all the practices in our evaluation had made the FFT available to patients, few practices felt committed to, or “owned”, the process. Even though practices had been assured that the test was intended for local quality improvement, not regulation (or criticism) by national bodies, they remained doubtful, particularly because of the monthly reporting. One GP, not realising that the scores were for meant for practice consumption, even asked their practice manager during a joint interview: “Do we have to open the box?”

Shift to reporting quality improvements

We recommend that monthly reporting of FFT scores should be stopped, and maybe replaced by a qualitative report on local quality improvements, perhaps submitted annually. That would help restore trust among practices that central bodies are not “spying” on them.

Particularly valuable aspects of the FFT are the chance for patients to comment briefly and quickly about their experiences, and the opportunity for practices to collect such feedback rapidly and easily. This comment facility could, for example, be kept within an FFT that had a less confusing core question, and patients could be encouraged to be more specific about particular services (e.g. phone consultations, clinics for chronic diseases, immunisation, etc.), making it easier to identify and address problems on aspects of care that they want to be improved.

The key issue in the long-term will be whether, and to what extent, patients’ views will contribute to making services better. Commitment to quality improvement was found to be uneven across practices. A revised FFT might play an useful role in addressing this problem, being easy to implement and thus a feasible option particularly for smaller practices with less capacity for data collection. But extra detailed guidance is needed on how to ensure that patient feedback leads to service improvement. It will be important to make clear that the Government’s priority is aligned with that of GPs in being focussed on securing higher quality primary care.

Tommaso Manacorda is a Research Fellow at the London School of Hygiene and Tropical Medicine. His report “Implementation and use of the Friends and Family as a tool for local service improvement in NHS general practice in England” is published by PIRU and co-authored by Dr Bob Erens, Professor Sir Nick Black and Professor Nicholas Mays.

This commentary summarizes an independent report commissioned and funded by the Policy Research Programme of the Department of Health for England, via its core support for the Policy Innovation Research Unit, with additional funding provided for data collection from the main sample of general practices. The views expressed are those of the authors and not necessarily those of the Department.

Social Impact Bonds offer challenges and opportunities in health and social care

21 April 2017
By Alec Fraser, Stefanie Tan & Nicholas Mays

SIBs bring promises of extra cash and reduced fragmentation but also concerns about private sector interests, practicality and governance.

What does research tell us about Social Impact Bonds (SIBs) and their applicability to health and social care? That’s our big question as we review the issues raised by our experts in their thoughtful commentaries over the past few weeks.

It’s clear how the mood music around SIBs has appeal, notably concerning two issues that bedevil health and social care – funding shortfalls and fragmentation. Ring-fencing funds and staff to achieve specific outcomes is a key SIB feature, particularly highlighted in the contribution from Ben Jupp of Social Finance. Those in health and social care who feel frustrated by juggling cash for competing tasks – having to “rob Peter to pay Paul” – would like to get on with their jobs, unencumbered by unexpected resource cuts.

Likewise, part of SIBs’ appeal is the feeling that they address chronic fragmentation of service delivery, particularly between health and social care. In principle, SIBs incentivise everyone to work in harmony on behalf of the patient/user.

Three questions

Some of our commentators have cast doubt over these possible benefits. Ring-fencing of spending and staffing can, for example, be achieved by other means, such as minimum nurse staffing ratios. Our contributors also highlight that SIBs can’t cure the structural problem of health and social care fragmentation, though they can be helpful on a case by case basis.

We would like to draw attention to three further issues raised by our commentators: the public’s general discomfort with the idea of private interests in NHS provision; the practicality of outcome-based, debt-funding in health and social care compared with traditional forms of payment; and the governance challenges created by SIBs as they reintegrate the purchasing and providing roles that have been separated in the English NHS since the 1990s.

On the other hand, we suggest that SIBs can contribute to mitigating some longstanding issues facing health and social care. These include the need for a better focus on prevention and outcome delivery; personalising services around the user/patient; and better partnership in health and social care planning and delivery between purchasers and providers.

1. Discomfort with private financing of healthcare

The early UK SIB-funded programmes have typically dealt with issues outside the mainstream of health and social care provision, such as youth unemployment and probation services. SIBs in health and social care have mainly focused on preventive interventions to improve health (e.g. social prescribing). We’ve yet to see proposed applications that address core NHS treatment activities such as funding hospitals and primary care. Such proposals might be highly controversial to the public.

The NHS retains a unique status for voters, unlike, for example, tertiary education, where private fees are widely considered to be acceptable, or social care, which is means-tested. The NHS is a universal service. There is a general belief that it should be paid for publicly. There is also less pressure to find alternative sources than there is, for example, in the case of some local authority social services because the NHS has been better protected from financial austerity than other public services.

2. Important practical and effectiveness questions

Our second theme concerns practical questions about the effectiveness of SIBs. Alex Nicholls and Mildred Warner have drawn attention to the weakness of the evidence in this area. There is a lack of evidence to demonstrate that SIBs provide better services that are more cost-effective or more likely to save money than those delivered using conventional funding methods. A further issue is how to make cashable any notional savings generated by improving user outcomes: for example, being able to close wards and sack nurses as a result of a successful SIB would not be a popular approach in such a highly politicised public service. Unlike many social services, such as problematic families, the public generally regards spending more on the NHS as a positive thing, so there may be less political support to pursue “savings” in health.

Healthcare commissioners in the UK are well versed in scrutinising cost-effectiveness evidence but healthcare lacks the interoperability of information systems that aids oversight of the outcomes and costs arising from interventions. So, for example, it’s currently easier at the Department for Work and Pensions to assess a SIB’s impact on employment rates among NEETs (Not in Employment, Education or Training) than for the NHS to track whether a preventive health SIB has reduced use of primary and hospital care.

Healthcare commissioners may also be concerned that SIBs typically have high set-up/transaction costs and quite opaque management costs. They may be reluctant to pay for outcomes when the sources of money needed to pay intermediaries and other actors lack transparency. This caution about SIBs is well founded given the continuing costs of previous PFI projects, which created new facilities quickly but left the NHS with debt and concern among some that providers may have secured poor value for money.

3. SIBs and governance issues in healthcare

Thirdly, there is a governance challenge that may accompany the adoption of SIB-funding for health and social care services. SIBs are attractive because they are designed to bring stakeholders into partnership in pursuit of a set of goals around a particular client group. This may imply, for healthcare, a degree of reintegration of purchasing and providing. This may be problematic as it appears that a genuine market in intermediaries is currently lacking.

We also do not yet have a recognised governance framework setting out how intermediaries and the other parties should operate at particular stages of the SIB development process. This is of crucial importance for policymakers as SIBs currently develop in a space that lacks agreed rules and processes. It will be important to resolve these governance issues and create an explicit framework for managing potential conflicts of interest.

Digging a little deeper, the SIBs debate should remind us of the need to maintain a healthy scepticism about all providers, even if they are from the Third/not-for-profit sector. They should be subject to due scrutiny, like any other providers of contracted-out services.


There are important questions and doubts about the wisdom of widespread application of SIBs to health and social care. Nevertheless, we should be careful and avoid prematurely rejecting a still largely untested innovation that could have niche applications for entrenched social issues. It is also worth remembering that traditional health and social care procurement models are far from perfect.

SIBs are rightly taken seriously because they help to question services that have, through bureaucracy, size and provider-domination, sometimes failed to meet the needs of patients or the requirement for cost-effectiveness. But there are important questions to be asked about the transparency, costs and accountability of SIBs. Our commentators have shown that SIBs are not a panacea and that they bring unanswered questions with them. More robust and more plentiful evidence would be needed to justify widespread roll-out of SIBs in UK health and social care.

Stefanie Tan and Alec Fraser are Research Fellows at the London School of Hygiene & Tropical Medicine and part of the Policy Innovation Research Unit (PIRU) at LSHTM.
Nicholas Mays is Professor of Health Policy at LSHTM and Director of PIRU.

Acknowledgement and disclaimer:
This is an independent commentary commissioned and funded by the Policy Research Programme in the Department of Health. The views expressed are not necessarily those of the Department.

Just how bad is the NHS at innovation? The answer – we don’t know

16 March 2017
By James Barlow

“There is strong encouragement for the NHS to make better use of innovation, but we’ve not known how our system performs internationally. Now we are developing a potential measure.”

The Accelerated Access Review (AAR) – which aims to help speed the introduction of health innovations into the NHS – was published last October. Readers with long memories may recall a string of previous reports all tackling the need to improve the UK’s health innovation system: the House of Commons Health Committee’s inquiry into the use of new medical technologies (2005) [1], Best Research for Best Health (2006)[2], the Cooksey review (2006)[3], Innovation for Health (2007)[4], Our NHS Our Future (2012)[5], Innovation Health and Wealth (2012)[6].

The AAR calls for a broader remit for NICE, to include more medtech and digital health innovations. Other recommendations are for closer alignment between regulatory requirements and processes, and for once-only assessment by NHS England of innovations not referred to NICE. Simpler and swifter procurement processes are part of the future agenda. And in an implicit nod towards the work of Clayton Christensen, the AAR suggests there should be some funding to support the commercialisation of disruptive innovations that have the potential to significantly improve care pathways.

These are all sound recommendations, but they are only the beginning. Recommendations now need to be turned into specific actions with budgets, owners, milestones and deliverables.

The good news from the AAR is that consensus seems to have broken out, with a genuine belief by government and NHS leaders in the potential of innovative technologies to improve patient care. The consensus extends to a call for more flexible approaches to pricing and reimbursement which can support the adoption of innovations.

The bad news is that the NHS is in deep financial trouble. For all the Churchillian rhetoric of healthcare leaders – “we have run out of money, now we must think” – and exhortations that a crisis is a great opportunity for innovation, the distinct tendency in the NHS is to hunker down, deal with the immediate concerns and shuffle innovation into the “too difficult” pile.

Goals, targets or crisis can of course concentrate the mind and, when coupled with the right kind of support and incentives, generate useful new ideas – a good example was Scotland’s Unscheduled Care Collaborative Programme in the mid-2000s[7]. However, the magnitude of the challenge of modernising the NHS, whilst continuing to run services with shrinking budgets, seems overwhelming. We know we need innovation, we know what kinds of innovation we need and we have a pretty good idea what is the potential impact of many innovations on improving care – the problem is introducing and establishing them in the NHS.

The AAR, if its recommendations are implemented, might help to speed up the flow of some new technologies into everyday practice, but it will not tackle the other layers of innovation that are needed. These were nicely described by the Health Foundation in 2015[8] as five interlinked components of a framework for change, all of which require innovative thinking – population health management, ways of delivering care, process improvement, active cost management, and scientific discovery, technology and skills. The report argued that all these are capable of delivering improvement over different timescales. Interestingly, the last of these – science and technology – is felt to have the most limited potential impact.

But just how bad is the NHS at innovation? There is much anecdote and received wisdom – “the UK is great at generating innovations but poor at adopting them”, “the NHS is always a late adopter”, “developing countries health systems are where the really interesting new ideas are”, “the USA over-adopts healthcare technology”.

Much seems to depend on perception. The 2016 Global Diffusion of Healthcare Innovation study surveyed 1,356 frontline health workers and healthcare leaders in six countries and found that the USA and UK were ranked as the two most important sources of innovation, but there were variations partly according to location – India was perceived as by far the most important by interviewees in some countries.

The volume of research on healthcare innovation processes has grown over the last fifteen years or so. We know what constrains the adoption of innovation and we know what innovators need to support them from research to commercialisation.

What we lack, though, is any kind of assessment of where different countries stand in relation to each other in their healthcare innovation capacity – how good they are at adopting innovative solutions developed elsewhere and originating innovations themselves. None of the reports on the state of healthcare innovation in the UK described above provide any real insight into our performance compared to other countries. Apart from limited work by Deloitte[9], PWC[10] and Accenture[11], there has been no attempt to create an international comparative healthcare innovation index (the Global Diffusion of Healthcare Innovation is not an index because it teases out perceptions of innovativeness).

There are many indices or composite indicators measuring aspects of national performance in innovation in general. Indicators include measures of entrepreneurship, technology development and scientific research, innovation in general, and innovation in public sector organisations. However, these remain underdeveloped in relation to health.

A carefully constructed global health innovation index would be valuable in two ways:

  • It would focus attention on the relative performance of the UK’s (or any other country’s) health system, pinpointing its strengths and weaknesses, identifying bottlenecks and issues for attention. This would enable policy makers and the healthcare sector to identify and prioritise the levers that can have the greatest impact on innovation.
  • It would help medical technology and life sciences companies by providing an understanding of the attractiveness of each country’s health system as a market for products, as a source of innovations, and as a location for R&D.

A project funded by PIRU has carried out the groundwork needed to develop a global health innovation index[12]. We reviewed existing global innovation indices and explored the conceptual, methodological and practical issues that must be addressed. We draw three conclusions from our review.

First, one should proceed with caution when deriving policy, research or other implications from composite indicators. Before we can be confident of their implications for the comparative performance of countries or regions and establish benchmarks to underpin policy or other decisions, it is essential to understand how innovation indicators impact on innovation processes. This in turn requires indices to be underpinned by a clear and strong theoretical framework.

Second, there are a number of data issues which have to be addressed. To explore the dynamics and evolution of health innovation across countries through an index requires panel data (collected over time). While the availability of cross-sectional data (between countries) is fraught with difficulties, the use of panel data is even more so. Another issue is variation at the regional or local level. Policy makers are interested not only in their national scores, but in whether different regions in their country perform differently.

Finally, policy makers should be careful not to draw conclusions about the relationship between composite innovation indicators and other indicators of policy interest such as health outcomes. While it may be of interest to correlate innovation index scores with those of an index of health outcomes, we must always remember that correlation does not mean causality – introducing a new drug or device may well lead to better health outcomes, but an index will not tell us this.

We are now hoping to create a prototype index for a few countries. This involves investigating data sources and more work on what the index is measuring, for whom and for what purpose.

Anyone interested in finding out more should contact James Barlow at Imperial College Business School,

James Barlow is a Professor of Technology and Innovation Management (Healthcare) at Imperial College Business School and a member of PIRU. His latest book, Managing Innovation in Healthcare, was published by World Scientific in January 2017.


  1. House of Commons Health Committee (2005) The Use of New Medical Technologies Within the NHS: Fifth Report of Session 2004–2005.
  2. Department of Health (2006) Best Research for Best Health: A New National Health Research Strategy: The NHS Contribution to Health Research in England.
  3. HM Treasury (2006) A Review of UK Health Research Funding.
  4. Strategic Implementation Group & Healthcare Industries Task Force (2007) Innovation for Health: Making a Difference.
  5. Department of Health (2007) Our NHS Our Future: NHS Next Stage Review – Interim Report.
  6. Department of Health (2011) Innovation Health and Wealth, accelerating adoption and diffusion in the NHS.
  7. Dattée B, Barlow J (in press) Multilevel organizational adaptation: Scale invariance in the Scottish healthcare system. Organization Science; Dattée B, Barlow J (2010) Complexity and whole-system change programmes. Journal of Health Services Research & Policy 15, S2, 12-18.
  8. The Health Foundation (2015) Shaping the Future. A Strategic Framework for a Successful NHS.
  9. Deloitte (2012) Innovation Indicators for Healthcare in Emerging Countries. Understanding and Promoting Innovation in Emerging Markets.
  10. PWC (2011) Medical Technology Innovation Scorecard. The race for global leadership.
  11. Accenture (2013) Measuring the Unmeasurable. A New Framework for Assessing Healthcare Innovation.
  12. Cravo Oliveira T, Barrenho E, Vernet A, Autio E, Barlow J (2017) Developing a Global Healthcare Innovation Index. PIRU Reports 2017-20.

Next step: develop Social Investment Partnerships

15 March 2017
By Ben Jupp

The experience of Social Impact Bonds can inform a wider set of relationships to help address social needs, argues a key innovator in the field.

The health and social care system I work in has rarely been in greater need of transformative investment. As demographic and financial pressures threaten to pull it apart, better responses to people’s needs are urgently required.

In my experience, developing more community focused and preventative health and care services will rest on approaches which combine both innovation and rigour. It will rely on strengthening the capacity of services to understand needs, learn from others, adapt and implement change. It will take an approach that looks beyond the annual budgeting of the NHS.

Many Social Impact Bonds seek to change systems

To me, and my colleagues at Social Finance, Social Impact Bonds (SIBs) have always been one tool to support such transformative responses. Take, for example, the Reconnections Service in Worcestershire. It aims to address chronic loneliness, responding to a need that was highlighted by older people in the county themselves.

Establishing Reconnections has involved designing a new service, pioneering long-term cost-benefit analysis on the impact of chronic loneliness, and helping draw together a partnership with a network of half a dozen voluntary and community sector organisations. For the range of commissioners involved, an outcome-based contract was attractive: making payments only if and when the new approach is effective. For the service itself, an outcome-based contract has also enabled flexibility and innovation.

The first year of Reconnections has been hard. Service providers, volunteers and investors have all had to work in ways they had not expected. We have had to learn and adapt. But, slowly, the six charities and community organisations are developing better responses to needs that have often been neglected for far too long, to the detriment of individuals and the health and care system as a whole.

The value of Social Investment Partnerships

Our experience of developing the Reconnections Social Impact Bond and other transformative programmes has also highlighted the value of creating broader Social Investment Partnerships (SIPs) in health and care.

Such Social Investment Partnerships encompass approaches to address social needs which include Social Impact Bonds but also wider forms of collaboration where commissioners, providers and investors share risk and pool knowledge and experience in the design and delivery of services. Highlighting partnerships reflects the need for commissioners, investors and providers to work collaboratively and flexibility, rather than the more adversarial separation between ”purchasers” and ”providers” of services that has characterised the NHS in England for the last 30 years.

For example, in a Social Investment Partnership to improve community-based end of life care, we have been working collaboratively with commissioners and providers for six months to scope out the need jointly. The investors and commissioners have then jointly considered which organisation would be best placed to develop the service; sometimes through a competitive process, sometimes by just building on an existing service.

Alternatives to outcome-based contracts

In such partnerships, we have also found that an outcomes-based contract is only one way to transform services. A focus on the underlying outcomes enabled by a service is important: an outcome-based contract can have significant advantages in terms of allowing service flexibility and providing focus. But for other areas, it’s impossible to capture the full range of people’s needs with a focus on a small number of objectives.

That is why, for example, in our partnerships to develop better Shared Lives social care (the Shared Lives Incubator), we have invested directly in the provider without an outcomes-based contract. Creating the conditions in which someone wants to share their home with another person in need of care is difficult to stipulate in a contract which prioritises one or two impacts. So the investors and commissioners jointly identify the provider, based on their overall qualities and experience. The investor then provides the upfront funding for developing the local service and repayment is made simply as proportion of the service revenue, if and when it grows. In other partnerships, risks are likely to be shared on the basis of a capitated budget – a set amount of funding for the population.

Drawing in investment from a range of sources

Finally, I have found that the concept of a “Social Investment Partnership” draws attention to the need for new resources and capacity and a long-term approach when tackling social challenges, rather than focusing on a particular type of financing. In some of the partnerships which we are building, such as those that develop better employment support for people with health conditions, the source of the funding is a mixture of public sector, charitable and external socially motivated investment. In nearly all such partnerships, we are combining financial investment with access to clinical and operational support. For example, because Health and Employment Partnerships operates across multiple areas, it has the scale to able to employ clinicians and service managers with deep experience of managing similar services before and is helping to manage NHS innovation funding as well as social investment.

I’m proud of the pioneering approaches that my team and colleagues across the sector are engaged in through Social Impact Bonds such as Reconnections. By broadening the approach to developing Social Investment Partnerships, we now have the opportunity to enable a wider range of contractual forms, further blur traditional adversarial relationships, and draw in resources from a number of sources.

These partnerships, whether traditional Social Impact Bonds or a not, are united by a passion for supporting communities and the most vulnerable in particular. They also recognise the importance of both innovation and rigour to achieve social change and ensure that organisations and systems have the capacity to adapt.

Ben Jupp is a Director at Social Finance. He was Director of Public Services Strategy at the Cabinet Office. He has also been Director of the Office of the Third Sector in the Cabinet Office.

Council information brokers could help technology to play vital role in dementia care

2 March 2017
By Martin Knapp and James Barlow

There’s no shortage of gadgets. But users and carers require help to make the most of what is available. Industry also should focus more on understanding what people really need from assisted living technologies, argues a new PIRU report.

In 25 years’ time, when many of today’s middle-aged population can expect to be living with dementia, technology will play a big part in their care. It will help them to stay in their homes longer and to avoid being institutionalized. It will also support their carers.

But right now, this isn’t happening enough, even though there is no shortage of gadgets developed by industry. “Why is this?” we ask in our report entitled “The Case for Investment in Technology to Manage the Global Costs of Dementia”. What can be done to accelerate the adoption of assisted living technologies, given the high levels of need among those who have dementia, the stress and isolation of their carers, and the urgency with which the cash-strapped health and social care systems need to control demand for costly treatment and support?

Slow adoption is partly explained by the particular characteristics of those whom we expect eventually to adopt these technologies – people with dementia themselves, their carers – and the local authorities and NHS bodies that might facilitate change. More on these issues in a moment.

Supply-side industry issues

However, responsibility also sits with those who develop the technologies. Research programmes which have looked into the use of assisted living devices continue to find that industry fails to optimise the user interfaces of technology, whether for health and care professionals or the general public.

That said, industry ought not to take all of the blame. It does, given the right circumstances, create and sell low-cost, user-friendly health-related technologies such as Fitbits, which many older people feel comfortable adopting. And most older people have mobile phones. So why doesn’t the large but niche market for people living with dementia show a similar level of growth and development?

Partly, it is because industry has not been as sensitive to this market as it has been to more able-bodied and -minded consumers. It can produce some really helpful, easy-to-use innovations, such as weight-sensitive mats that tell carers remotely whether someone has got out of bed and is walking about. However, as far as more sophisticated devices are concerned, research shows that suppliers often still tend to blame users when technology doesn’t work well. They should instead ask themselves more searching questions that would identify often simple design difficulties in their products.

Suppliers are not necessarily fully cognisant of the need to match technology to the sensory and tactile skills that older people have. Major companies are sometimes naïve about both the basics of what ageing means for individual abilities and the markets for a product. Developers in the ICT industry are still seduced by bigger, more obvious markets in which less trouble has to be taken in show people how to use the technology.

Despite all the effort put into user-led design, there is also still a big gap between research and development and engaging users to ensure that products are right from the outset. There are also plenty of examples where researchers and companies appear to develop a solution that’s in search of a problem.

Lack of support for users and carers

Nevertheless, despite these supply-side concerns, perhaps the biggest issues with regard to dementia care and assisted living technologies are on the consumer or demand side. We can’t be scientific, but perhaps about 80 per cent of the challenge is human, organisational and funding, compared with 20 per cent of the problem being about developing the technologies in the first place.

Funding issues are acute in the UK, where, like in most countries, the health and social care systems are fragmented. That means the costs and benefits of assisted living technologies fall into different silos, which can block take-up of innovations.

From previous research we reckon that an eighth of the costs of dementia care fall on the NHS, a quarter on social care services and the rest on informal carers. As a result, it is sometimes difficult to persuade one system to fund technology when the cost savings might accrue elsewhere. This sharing of cost is particularly difficult in the UK because social care is means-tested while healthcare is not.

An underlying issue, which our report also highlights, is that these technologies can be challenging for everyone, but particularly for older people, many of whom are not competent or confident with what for them are new technologies. That is particularly true for people with cognitive difficulties arising through dementia.

Carers will generally not have the cognition issues – although many carers are quite old themselves, and may have their own health issues - but they may not be familiar enough with the technologies to take on new challenges and tasks within the care setting. Older carers in particular might not have spent much of their working lives using ICT. They may perceive broadband to be expensive, even though the actual cost is not particularly high.

People need reliable advice and support

Often advice and support that’s available from the local authority is neither confident nor competent. The Care Act 2014 requires support for carers, but this duty has coincided with a sharp reduction in social care funding and availability, so that implementation has not matched the ambition.

Our report finds that local authorities are not good at sign-posting technologies to carers or at providing other associated supports. This concerns not just the monetary costs of purchasing technology but the costs of learning new skills. Such help would require fresh investment by councils.

However, this support could be vital to carers who need sound advice around technologies that they themselves might buy. They are not alone in being in the dark: a whole range of people in the system - including commissioners and care homes - lack knowledge about what is available, and what it might achieve.

That is why we argue that the public sector should take on an information broker role, offering a source of unbiased advice on assisted living technologies. A council might even use its purchasing power to negotiate attractive deals with suppliers, so that people feel assured that they are buying the right product at the best possible price.

There are examples of healthcare professionals – including dentists, GPs and pharmacists - providing this kind of advice to vulnerable patients and carers, even if their own service cannot foot the actual bill. The Third Sector has also long provided important consumer support, going right back to the founding of the Consumers’ Association. Some of the charities that support older people are well-positioned to take on such a role.

It may well be that industry – keen to expand its own markets – has a part to play in easing this problem. It should consider how it can support delivery of unbiased information and support. Carers and patients need all the assisted living help that they can get. They can’t wait for another 25 years.

Martin Knapp is Professor of Social Policy, Director of the Personal Social Services Research Unit at the LSE, and Director of the NIHR School for Social Care Research.

James Barlow is Professor of Technology and Innovation Management (Healthcare) at Imperial College Business School.

Both authors are members of the Department of Health’s Policy Innovation Research Unit (PIRU). Their report, written with several other colleagues, “The Case for Investment in Technology to Manage the Global Costs of Dementia”, is published by PIRU.

SIBs don’t work for complex problems because they’re unaccountable to service users

1 March 2017
By Stephen Sinclair, Neil McHugh and Michael J.Roy

Supporting change through social policy is an iterative process which develops participatory relationships with users. Social Impact Bonds fail on this front, rendering them unfit for purpose.

SIBs are intrinsically unsuitable for programmes that seek to transform people’s lives. That’s because such transformative interventions typically succeed only when they involve partnership with the people who are in need of help – and that’s not one of SIBs’ strengths.

SIBs are generally unaccountable to service users; they usually answer to funders. That’s their core failure and limitation. An inadequate relationship with service users means that SIBs typically lack sufficient understanding of – and sensitivity to – the causes of social problems that is required to achieve transformative change.

We should face up to these issues because, although there are only 32 SIBs operating in the UK, there is a considerable amount of enthusiasm for them. They have, and continue to receive, a lot of vocal and financial support from government.

Difference between technical and transformative interventions

In social policy, it’s helpful to distinguish between technical and transformative interventions. The former tend to be simple responses that encourage relatively uncomplicated changes and behaviour. Often they are about increasing service use, such as, for example, raising the uptake of vaccinations. SIBs could be suitable for such programmes and randomised controlled trials could be one way to evaluate them.

In contrast, transformative interventions try to generate profound changes in the dynamic between conditions and actions. So they must engage the agency of service users or the recipients of those interventions. These interventions address complex problems, often with multiple stakeholders, and try to ameliorate poor conditions. They’re very much contextual. Crucially, they’re about doing things with people rather than doing things to them. So, they might involve trying to reduce or tackle obesity and the problems around that very complex issue. Or they might seek to reduce health inequalities which are a major problem within the UK.

Recipe for social transformation

For social interventions to be considered transformational, they must engage with the means and the identities of those they seek to help. They should cultivate capability for independent living. There shouldn’t simply be a focus on producing outcomes. A good way of thinking about this is to consider education policy: it should be about cultivating independent and critical thinking rather than focusing simply on outcomes and achieving certain rates for test passes.

This approach should encourage us to rethink best practice. Where transformation is concerned, it shouldn’t be about uniform standardisation. Policy should respond to varying conditions and individual circumstances. What works in one area may not work elsewhere. We should resist the urge to scale up interventions indiscriminately.

Different approaches to evaluation

Evaluations should, in turn, provide insights that help us understand these very complicated human interactions and social conditions. Good policy development and evaluation should be about understanding interactions between agents and the subjective experiences of service users. It should be about getting to the bottom of why things work, which is a difficult and challenging issue.

There tend to be two approaches to evaluation. One tradition - let’s call it “positivist” - is exemplified by much of the evidence-based policy movement. It tends to favour randomised control trials in trying to attribute causality.

An alternative tradition – let’s call it “phenomenological” or “interpretivist” - regards policy in relation to complex social problems as less linear and more of an iterative process. It’s based more on reasoning from incomplete or imperfect data. It highlights, to some extent, the limitations of the evidence-based policy movement.

A great deal can be learned from the field of public health. It has a long history of engaging in, developing and evaluating complex social interventions. Indeed, the Medical Research Council (MRC) has issued guidance on how to develop and evaluate complex interventions. The MRC says it’s an iterative process and that the first step should be to understand subjects’ views and how they understand their own condition and treatment.

The political nature of social policy

So social policy, when trying to do anything complex or transformational, shouldn’t try to reduce issues to mechanistic terms such as providing inputs and then producing outputs. We should also recognise that the design of social policies is a political process. Indeed, simplifying complex conditions and problems into depoliticised inputs and outputs, far from being apolitical, is actually a form of ideology itself.

Nevertheless, SIBs tend to be represented as a technocratic and depoliticised approach to policy. They involve shifting responsibility for welfare policy away from government. In this vision, the role of government is diminished, becoming about purchasing or brokering public services.

SIBs are highly political
We would, instead, argue that SIBs are highly political in two key respects. First, they turn public services into commodities, rather than being services to citizens that require a relationship with those citizens. So SIBs “commodify” public services. Second, they give financial value to those fictitious commodities. So SIBs change what were non-monetary social relationships into financialised commodities.

This process diminishes and dehumanises service users who become reclassified as customers. This diminution is not immediately obvious because it is masked by language. Indeed the language of consumer culture – in which the key agent is the customer who is always said to be “right” – might seem to raise rather than diminish the status of the service user. But this is actually a fraudulent representation.

A real customer has control of the cash, providing him or her with consumer sovereignty. In reality, with SIBs, there is very limited consumer sovereignty because citizens’ welfare entitlements become transformed into contracts between organisations and stakeholders.

In this change, the quality of these services is altered. The previous public sector ethos is lost, replaced by a profit incentive. As a result, the service user, instead of being at the heart of everything, becomes an externality, a troublesome afterthought to the main activity which is a series of transactions between principal stakeholders.

What next for SIBs?

Where does all this leave us? SIBs encourage a focus on what is measurable. They also prioritise outcomes rather than trying to understand how these effects are actually achieved. We would argue that complex multi-dimensional social problems require a different approach that involves collaborative policymaking. This policy should involve trying to alter conditions for service users’ rather than imposing strict theories of change upon them.

Moving forward from SIBs

Recognising these deficiencies, we propose a different approach which prioritises service users’ perspectives and which is a first step towards co-determination of policy. A good way forward is to use Q methodology, a research tool used in psychology and in social sciences to study "subjectivity" (namely views, opinions, values and beliefs). It was developed in the last century by the psychologist and physicist, William Stephenson.

This approach is a well-established way to identify and describe viewpoints. First, a card sort (of statements) is used to generate data. Then, there is a form of factor analysis which identifies patterns and similarities between individual card sorts. These methods reveal the structure and form of subjectivity on a particular topic, allowing subjective perspectives to be described.

Q methodology has potential uses in policy development and evaluation because it could help to ascertain the relative importance of problems, the causes of problems, as well as potential solutions and policies. However, it’s just one practical, albeit important, way to actualise service users’ perspectives and so improve policy design and evaluation.

Crucially, this approach is aligned with genuine social innovation whose starting point is that it should empower service users. It should enhance their capacities. That’s missing in SIBs. If social policy is really going to make a difference, then we need to replace SIBs with more participatory approaches to policy development.

Dr Stephen Sinclair is a Reader in Social Policy in the Yunus Centre for Social Business and Health, Glasgow Caledonian University
Dr Neil McHugh is a Researcher at the Yunus Centre for Social Business and Health, Glasgow Caledonian University.
Dr Michael J. Roy is a Senior Lecturer in the Yunus Centre for Social Business and Health, Glasgow Caledonian University

Dream of Social Impact Bonds should not blind us to their dangers

13 February 2017
By Mildred E. Warner

In the US, SIBs are the height of social policy fashion, but the risks are clear and numerous.

The dream of Social Impact Bonds (SIBs) sounds pretty exciting. It’s that you can invest in something good for society and at the same time get a market return so everybody wins. The investor wins, the client wins and, because it’s a better programme, society wins.

SIBs are growing around the world and mainly focus on prevention, which is good to see, because cure is often more costly than prevention. There is considerable enthusiasm in the US where 35 states are building programmes to encourage SIBs. Only ten SIB projects are active in the US, but there are probably hundreds in the pipeline.

Political support

In Congress, support is coming from both sides of the aisle. The US Government Accountability Office told me: “You know, Mildred, the Republicans like this because it lets the private sector get access to public social welfare dollars and the Democrats like it because it might increase public investment in social welfare”. So it’s a win-win politically.

Large organisations are lining up to act as intermediaries in establishing SIBs and evaluators are honing their skills to get the contracts to assess the outcomes. I’ve even talked to people on the street who have heard about SIBs. “I’d like to invest in that,” they’ll say. “I’d love to put my money in something that’s going to yield a good return for everyone.” But they’re not quite sure what this thing is.

Considerable concern

And there’s lots of concern as well. Providers are responding to this new landscape - some are nervous about payment. Academics are probably the most critical. Government managers feel cautious, worried about how much work is involved in putting these things together. Sometimes, officials feel forced reluctantly down the SIB pathway.

Take, for example, the SIB at Riker’s Island prison, designed to reduce the rate at which juvenile offenders return to jail. The Vice Mayor of New York City told me she would not have done the Riker’s Island project as a SIB if she could have funded the program directly. That would have been quicker and easier. But budget limits prevented that and she didn’t want to wait as, each year, more teenagers get caught in the prison system.

So we should take a long hard look at what’s going on. We should make sure that we’re not swept along by a tide of wishful thinking that could leave disappointment in its wake.

SIBs transform social services

This aspect of SIBs is particularly important because they represent a major upheaval in the design and delivery of social services. Typically, SIBs require intermediary management, private investment and some kind of outside evaluation which allows you to link performance to financial returns.

SIBs take public management to a new place because government is no longer at the centre, as in the past, but an intermediary organisation is organising and running things. It’s true that the Government is at the top, calling the shots, defining the structure, defining the goals, but the intermediary is at the centre of the process making everything else happen, linking to the service provider, linking to the outside investors, commissioning the evaluator. The intermediary is becoming a very important actor.

Clients lack a voice

There are some real concerns about the way SIBs work. They tend to focus on areas where the client is pretty weak or voiceless or maybe despised by society. These are areas where we haven’t been able to motivate sufficient public investment in prevention because who cares, for example, about a prisoner and their re-entry into the community?

Yet, ironically, SIBs seem to leave the clients voiceless. These are homeless people, little kids, people who are vulnerable. There seems to be very little thought that we’re not giving the consumer or the client a voice in SIBs. That’s deeply troubling.

I would sum up other concerns as relating to: the suitability of SIBs for complex social problems and solutions; the difficulties of contracting; the transparency of SIBs; whether private investment in SIBs is, or can ever genuinely be, a reality; and the potential impact of private investment on the core values of public services.

Do SIBs suit complex problems?

You’ve got to be careful because social problems are by definition very complex. People are complicated and we live in a very complex world. There’s a lot going on. So focussing on a simple, short-term intervention to deliver a single outcome may not be the best approach.

The Riker’s Island SIB, for example, funded a behaviour modification for young people in prison. Those teenagers were then sent home with very limited skills often to distressed family and neighbourhood situations and to try to engage in an economy that had pretty much left them behind. But they had been taught how to be polite. That’s cheap and it certainly doesn’t hurt. But don’t we need a more comprehensive approach? Aren’t we simply treating the symptom and not the cause? Not surprisingly, some critics have said: “We really need some longer term structural change and this is just a Band Aid”.

Some SIBs are being developed in the US to fund pre-school provision. Indeed the roots for SIBs in the US lie in work done around developing private investment funding models for preschool provision. I can see the appeal. Preschool is inexpensive compared with the costs of early childcare. It’s relatively short term and you can get a good measure of the investment’s efficacy when you see how kids perform in kindergarten. It’s linked with some wonderful long-term improvements in health, education and employment.

When all these savings are calculated, one model found that preschool offers a 17 per cent internal rate of return, which is better than anything the stock market could give. And three quarters of that return is savings to government which offers the chance to get government to pay for SIBs out of savings in future programme costs. Nevertheless, it’s pretty heroic to assume that preschool provision can be credited with producing all the benefits you hope for when a child reaches adulthood.

At this point, it’s also worth remembering that SIBs – like many social policy innovations – are created around the assumption that you can take model programmes that have been proven to work in one place and then scale them up. But have you ever tried to make brownies for 40 people instead of ten? You actually have to change the recipe. You don’t just quadruple the recipe, because it won’t work.

Contract complexity

I worry about the complexity of contracts that typically accompany SIBs. I’ve been studying contracting for 20 years in local government in areas like water and garbage collection. Those contracts can be quite complex, but they are simple compared with contracts for social services.

We also know that cost savings from contracting out water and garbage collection are at best ephemeral. In the longer term, markets require management, which can be expensive – especially when, at a local government level, there are not really markets for your public services. I measured how many alternative providers there are for any one of 72 different services that local governments provide in the US. On average, they have one alternative provider. This does not a market make. That’s why local governments do a lot of reverse privatisation or contracting back in – to ensure that there is some element of periodic competition.

High transaction costs

Transaction costs are also high to set up the contracts. Most local officials would say: “If I’m not going to get a ten per cent savings, I’m not going to go out because I’ll spend more than that just designing the contract.” Additionally, because public markets aren’t competitive, there needs to be monitoring, but monitoring is also expensive. That’s especially true for SIBs: they require high quality evaluation because the financial planning is based on delivery against metrics and, ideally, there should be a counter-factual, some sort of comparison group, not subject to the SIB.

Poor transparency

SIBs lack transparency – there is a lot secrecy about discussions until the deal is done. People may not hear about a project until it has already been designed and, even then, documents are often unavailable to the public for scrutiny. It makes them hard to study and is a problem for open governance.

Inflexible innovation?

SIBs could actually promote inflexible innovation, because they typically involve a model programme for a process which has been proven and which is then locked into a contract. Fortunately, it’s usually only for three to five years, whereas public private partnerships for infrastructure might be 25 or 30 years. But that’s still a constraint on innovation.

Will the private sector invest?

The promise is that SIBs will engage the private sector as an investor in public services and so increase funding for effective programmes and build the political will for policy change. So the long-term outcome could actually be more public spending on these projects once the private investment has shown the value of investment.

That is problematic in the US context because, in seven of the ten projects that currently exist in the US, more than 50% of the finance is being guaranteed by private philanthropy. One of the key investors enjoying this underwritten status has been Goldman Sachs.

I had a student recently from the finance sector who did his Master’s thesis on SIBs. He concluded that the risk of SIBs is too high to attract private capital - a secondary financial market will be required to provide private sector funding. That’s something we should be watching for.

Public values at risk?

Some of the enthusiasm around SIBs is that we’re going to insert efficiency and investment logics to make the social sector better. That sounds interesting. But there is the risk that we will lose some of the values that have underpinned social service policies, such as social justice and citizen empowerment. If financial logics prevail over social values, that could undermine other social goals. And if there are savings from SIBs, why does government have to mortgage those savings to private investor profit, rather than spend it on future investment?

Then there are also those who worry that everything is priced in a SIB. I recently saw the play, “A Curious Incident”, about a child living with Asperger’s Syndrome. In the past, we would have justified spending on such a child as a matter of their right to an education and society’s obligation to accommodate all children in our world. But if we shift towards thinking about the “investible child”, justifying expenditure now by its returns in the future, will that child still get the investment s/he needs? I worry about social Darwinism creeping into social policy, especially as the clients’ voice is largely silent in SIBs.

Meanwhile, what role may intermediaries take in redefining the values of public services? The US already has a largely devolved social welfare state. SIBs may hand intermediaries even more power than they already have to determine what social policy looks like.

New political allies

So there are many challenges. But it’s also important to recognise that SIBs are making a difference, sometimes in unexpected ways. For a start, it’s clear to me that many city managers are trying to ride this wave of marketisation in social policy and get something positive out of it for their communities.

So for example, all the discussion about investment returns on preschool provision is bringing in new political partners to the cause of developing better services for children. Important CEOs in the local communities are saying: “This is important. We want to see investment here”. I remember the head of one of the public social welfare agencies saying: “This is the first time I’ve ever been able to go to city council and tell them that their expenditures are investments, positives rather than negatives.” This was because she was able to talk about the return on investment. SIBs often focus on quite short term returns. This style of language gains political attention, and motivates leaders more than long term returns because, they think: “In the long run, I’m not going to be in office”.

New policy tools

We are also seeing the development of new policy tools. People in the welfare arena are accustomed to rules and rigid regulations governing social services. In contrast, the language of economic development encourages entrepreneurship. In the social welfare world, people find it liberating to be in a culture in which they feel incentivised to deliver a policy rather than simply being governed by regulations and rules.

SIBs aren’t going away. They are growing. In the US they are likely to mushroom, but without the careful study and scrutiny that such a policy reform requires. We should not be blinded by the dream of greater investment and greater program effectiveness. Neither has been proven in the early SIBs. And there are clearly many issues about management and public values. We need to be honest about what really is happening. We should give special attention to how to maintain the important values that underpin our public services.

Dr Mildred E. Warner is a Professor in the Department of City and Regional Planning at Cornell University. Her published articles on SIBs and government services can be found on her website

'We should model complex public health interventions before piloting them'

3 February 2017
By Zaid Chalabi

Mathematical models can test multiple variables cheaply and quickly, giving early indications of what really matters. That helps in designing pilots and understanding how context can affect a policy’s success, argues Dr Zaid Chalabi.

I was part of a team that evaluated the implementation and cost-effectiveness of the government’s Cold Weather Plan (CWP) for England. The CWP is a guidance document which aims to reduce the thousands of additional deaths that typically occur in England when temperatures plummet. The fall in temperature can increase risks for elderly people as well as those with heart or breathing problems and other chronic conditions.

The plan’s principle is that, when cold weather is expected, the authorities are alerted and they can enact various measures, suggested in the CWP. They might, for example, contact or visit vulnerable patients, check that they have medication, that they are warm and have enough food to last a cold snap. The CWP guidance is quite general and it is up to each local authority to implement the plan in its own way.

Assessing the Cold Weather Plan

There is a lot that we do not know in assessing the CWP’s cost-effectiveness. How fully will each health authority implement the plan? Which aspects will they focus on and what impacts are made by each particular action?

Somehow, we need to know how the CWP, implemented in its various ways, might impact on the health of the population and also avoid hospital admissions to save on costs to the NHS. The costs of elective admissions - postponed if there are weather-induced surges in emergency admissions - also must be assessed. We must evaluate the quality of life, and the years of life extended by the CWP, as part of the final cost-effectiveness calculation.

Clearly all of this is hard to evaluate because there are so many variables. Also the circumstances may not occur often: a decade of mild winters might pass before there is a harsh year. Because the CWP has been running for only a few years, we do not have much data, yet it is also a life and death scenario, so policy makers need advice on how best to implement the CWP in order to deploy limited resources well.

Options for evaluators

What should evaluators do? A meaningful analysis of the outcomes of the CWP would require the ability to compare vigilant health authorities with those less engaged, and to evaluate the CWP over several harsh winters. We cannot afford to wait that long, and it may be unwise to stop some authorities making preparations to protect their populations.

Modelling helps when data is missing

We adopted mathematical modelling as the best approach in such circumstances, where lots of important data are not yet available. It is possible to search the literature or seek expert opinion to make estimates for almost every scenario and action. We can estimate, for example, the benefit to patients’ health of being contacted during cold weather. Theoretical costs can also be factored in for the different options – be it a phone call or a more expensive actual visit. These figures can never be absolutely accurate. But, by building in as much data as possible, the model begins to reveal which variables are significant and which ones do not matter much.

Modelling also provides indications of where the extra costs of the CWP might occur (probably in social services) – and which areas are likely to enjoy savings (probably acute hospitals). In health and social care, implementation of cost-effective innovations is often held up because silo working means there are both losers and gainers. Because the losers may not be compensated, the policy may not be implemented, thus depriving the system of a net gain overall. Modelling can identify who wins and who loses, giving policy makers the chance to equalise the outcome between the silos.

This approach to evaluation, in identifying the many variables and assumptions, also helps us to see gaps in knowledge so we can focus research on gathering any important missing data.

Modelling and other public health interventions

The value of modelling goes beyond policies such as the Cold Weather Plan that can take years to bear fruit. Most public health interventions occur in complex environments and involve multiple variables. It is important to understand the role that context plays in their success.

Models provide an opportunity to vary contexts and see what matters and what does not. They are the obvious precursor to setting up a pilot that can then be designed around key factors that seem to be important, regardless of context. Inserting modelling into the process of early evaluation, before piloting, could be vital if we are to escape the scourge of successful pilots that are not rolled out, because they seem to work only in one place.

Dr Zaid Chalabi is Associate Professor in Mathematical Modelling at the London School of Hygiene and Tropical Medicine.