Combating the risks of increasingly hot summers in the UK: Findings from the PIRU evaluation of the Heatwave Plan for England
27 May 2020
Lorraine Williams, Bob Erens, Stefanie Ettelt, Shakoor Hajat, Tommaso Manacorda and Nicholas Mays
In the midst of a coronavirus pandemic, the awareness of planning and preparedness for crises has never been more important. Images of local communities working together to support care home residents trapped by the floods of Storm Dennis contrast with more recent images of care home workers struggling to keep residents and themselves safe from Covid-19.The year 2020 has become a testing time for public health services and communities. The risk of heatwaves adds another layer.

Heatwaves are often referred to as ‘silent killers’. They may not have the same visibility as other emergencies such as storms or floods, nor be as deadly to a population as a virus pandemic.They are, nevertheless, a cause of much premature death and ill health; a severe heatwave occurring in the summer of 2003 accounted for over two thousand excess deaths in the UK. National heatwave plans go some way to safeguard people who are most at risk, especially those in health and care facilities, but there remain a number of groups who are still unaware of their heat-health risks and who do not follow public health advice.

As we enter the summer period in England, the risk of increasingly severe hot weather as a result of global warming presents an additional threat to people’s health, and provides a rationale for having clearly defined national and local strategies for managing heatwaves. This is especially important as heatwaves have been shown to have a significant impact on the health of older and more socially isolated people, and those with chronic respiratory and cardiovascular diseases.

Severe heat places an extra strain on the body to regulate temperature, and this risk increases with age, leading to heat-related conditions such as dehydration, heat cramps or, more seriously, heatstroke. It also accelerates other chronic conditions and can result in hospitalisation or death, even within the first day or two of a heatwave. It is agreed that many heat-related deaths and illnesses could be prevented with good preparation, responsive care, and individual behavioural change. However, despite England having a national plan to reduce the adverse health impacts from heatwaves, significant excess deaths and illness still occur, placing an extra burden on health and social care systems. For example, in 2019, the deaths of almost 900 people were attributed to high summer temperatures.

A heatwave plan (HWP) has been in place in England since 2004. We identified gaps in the system in how some potentially vulnerable groups were identified and protected during the heat-health alert period, particularly older people with health problems who were not in regular contact with health or social care services, many of whom may be living on their own. Many health and care managers relied on general public health information, through national or local media, to reach these groups, and assumed that recipients would act on that advice.

Most people had positive views of hot weather, and for many it was something they looked forward to as it invoked feelings of good health and well-being. In addition, most adults did not feel that hot weather posed a risk to themselves. As a result, protective measures were often not taken, even those that people felt were effective, such as staying out of the heat and in the shade. Younger adults were most likely to underestimate their risks and ignore public health advice on sunbathing and drinking alcohol in hot weather.

Similar results were found among people who were most vulnerable to the effects of heat, particularly in their perceptions of risk. For example, only 40% of participants aged 75 or older saw themselves personally at risk of hot weather. According to many of those interviewed, risk was associated with physical and mental frailty, not age itself, and there was some resistance, even resentment, to any ‘vulnerable’ label that may be applied to them in this context, as they did not consider themselves ‘frail’. Participants felt that hot weather increased their risk of sunburn and skin cancer, however there was much less awareness of the thermoregulatory risk of hot weather, and how this risk increased with age. Security and cost, in addition to self-perception of risk, were key considerations for this group. For example, many did not open windows at night for fear of intruders, or use an electric fan due to the cost of running one. The expectation that any period of hot weather would likely be over quickly was often weighed against any discomfort experienced.

As we move towards more frequent hotter summers, developing systems of identifying, reaching, and supporting our most at-risk communities will increase in importance. As a first step we recommend that Public Health England revise their advice and publicity to improve public awareness of the risks of hot weather. Messages should be tailored to the information needs and media usages of different population groups, and take account for the finding that many people do not self-identify as vulnerable, and therefore should focus more on the risks of hot weather, and help people self-assess their own risk more realistically without the label of ‘vulnerability’.

The published report can be accessed here>>

The two meanings of personalisation in residential care and why they matter for the future of care homes
12 May 2020
BY STEFANIE ETTELT, LORRAINE WILLIAMS, JACQUELINE DAMANT & RAPHAEL WITTENBERG

The situation of older people in care homes and the difficulty of protecting them during the current coronavirus pandemic attracts much attention these days, and rightly so. The devastating effect of the pandemic on care home populations highlights a number of shortcomings in residential care in England, which for years has neither received adequate levels of funding nor much policy attention. Yet over 350,000 older people currently live in care homes, because they require high levels of care during most of the day and night. By ‘care home’ we mean organisations that provide long-term accommodation and services to people who need 24-hour care and attention, including help with personal care (in a residential care home) and medical care (in a nursing home). A care home is usually a person’s last home, but it is nonetheless their ‘home’. So it is important that they are able to live their lives as fully and as independently as possible.

In England, consecutive governments have emphasised the importance of personalisation for people receiving long-term care services. Personalisation refers to the process of tailoring care services to the specific needs and preferences of service users. In the past, this has mostly involved community care for individuals who receive care in their own homes, with policy-makers giving less attention to personalisation in care homes. This is regrettable, as it reflects the way many people associate a care home with an ‘institution’ or ‘place of last resort’, rather than a destination of choice in later life. Most initiatives to make care more personalised have been driven by the charitable sector (e.g. through initiatives such as ‘My Home Life’) or by care homes themselves.

Our study has sought to investigate how personalisation in residential care is conceptualised, which approaches are likely to succeed in promoting personalised care, and which barriers need to be overcome to provide an effective service to people living in a care home.

There is a difference of view of what personalisation means in the context of a care home. Essentially there are two sets of ideas, pulling in different directions. The first emphasises residents’ choice and control over the services they receive. This idea was influenced by attempts to promote ‘voice and choice’ among patients receiving health services in the NHS. Choice and control have been given prominence in domiciliary services for service users eligible for local authority support, who are given a choice between accepting care services organised by the local authority, and accepting a budget to organise their own care in the form of a direct payment. In this context, choice and control are often used interchangeably with ‘self-directed support’, which has been a key aim first set out by the disability movement. This movement has sought to increase the inclusion of people with disabilities in society, and to improve the services they receive. Its success has led to a major shift in awareness of what people with disabilities are able to achieve and the quality of the care and support received. It has been argued that this is due to the effects of individuals holding the purse strings for their care on the balance of power between them and the service provider. However, research has shown that many older people in residential care are less likely to benefit from choice and control in this way, given the extent of their care needs and the nature of how care homes are financed.

The second set of ideas developed out of concern for the quality of care for people with dementia. Typically referred to as ‘person-centred care’, it emphasises the importance of care and caring, and the role of the professional carer in promoting the wellbeing of the person receiving care. This concept gives the carer and the care home the responsibility of ensuring that people with dementia are supported in maintaining their identity and to articulate their wishes and preferences. A central aim is here to help people with dementia to maintain a sense of continuity between their lives before and after the onset of dementia, and their experience before and after moving into a care home.

Clearly, these two sets of ideas are not mutually exclusive. Proponents of person-centred care would always emphasise the importance of giving residents choices and control over their daily activities and to support them to lead as fulfilled lives as possible. Likewise, the idea of choice and control does not preclude people from enjoying caring relationships: indeed, most direct payments in domiciliary care are used by service users to select and employ their own carers.

However, these two sets of ideas differ in how they conceptualise the approach to personalisation and make different assumptions about the person receiving care. While person-centred care is rooted in a philosophy that accepts that the need for care and support is part of the human condition and a universal experience, choice and control casts the individual as a consumer in a market for care. They therefore differ in their assumptions about the relationship between the individual and the provider of care, the ability of the person to exercise choice and make their own decisions, and by extension, the person’s cognitive capacity. They also make assumptions about the level of support required for individuals to derive benefit from such choices and decisions and how this support is organised and funded. Both versions of personalisation have cost implications. Both necessitate adequate levels of staffing and training. Yet while the investment in staff is associated with the provision of customer care in one model, the other model requires an adequate number of skilled care staff to be able to build the relationships that underpin the provision of person-centred care.

In our study, we found both sets of ideas reflected in the care homes. When interviewing managers in care homes, almost all recognised the concept of person-centred care, noting the importance of trust between staff, residents and their family members; a cornerstone of personalised care. Some evoked images of the ‘family’ of care home residents and staff to describe these close, trusting relationships. In contrast, others used the image of the care home as a ‘hotel’ to express their aspiration to provide comfort and convenience to their customers, as if they were ‘on an expensive holiday’. Such customer care was reflected in practices such as providing menus on the tables of their ‘restaurant’ (rather than dining rooms); and it resonates with practices in certain types of more upmarket homes.

What are the implications of different concepts of personalisation in care homes? In principle, the availability of different models of personalised care enhances choice. If someone wishes to live in a ‘hotel-style’ care home they should be able to do so, even if the type of care such homes provide may be less focused on personal relationships. In practice, however, people with fewer resources have considerably less choice of care home than those with greater resources. People with low incomes, few savings and no relatives able to contribute to the costs of their care face a choice limited to those care homes which accept residents at local authority rates. Those with relatives able to top up local authority rates have more choice of care home. People with high incomes or substantial savings, while not eligible for local authority support, have the widest choice of care homes. This means that there is socioeconomic inequality in choice of care home. In addition, there is also the question as to whether the model of personalisation they offer, perhaps especially for ‘hotel-style’ homes, will be suitable for people whose care needs are substantial and who may be better served by a different care model such as one modelled on small ‘family’ units.

To date, the diversity of approaches to personalisation in the care home market and their implications for service users are not well understood.When the care home sector emerges from the pandemic, time should be taken to reflect on the funding and policy support needed to develop a sector that is not only more resilient to emergencies, but is also more equitable. It is time to reimagine the future of residential care to create homes that people would like to live in as a form of supported, communal living, rather than a place of ‘last resort’.

Ettelt S, Damant J, Perkins M, Williams L, Wittenberg R (2020): Personalisation in care homes for older people. London, Policy Innovation and Evaluation Research Unit.

A summary of the report can be found here >> and the full report can be found here >>

Shining a light on the dark side of coproduction
28 November 2019

BY KATHRYN OLIVER, ANITA KOTHARI & NICHOLAS MAYS

Advocacy of co-produced research

Many, if not most applied research projects, are undertaken with some degree of collaboration between researchers and ‘stakeholders’, who include research funders, policymakers or practitioners, members of the public and civil society, or other actors such as patients in health studies. This collaboration takes various forms (e.g. consulting on research topics, or working iteratively throughout the research process), and in general, people feel positively about it. Funders often require it, or at least support it, and researchers have argued that it is the most effective way to ensure that evidence is used or translated into practice. Today, there are many forms of collaborative research practice, including co-production, co-design, co-creation, stakeholder and public engagement, participation/involvement and integrated knowledge translation which sit under this umbrella. They reflect a very diverse set of motivations, activities, identities and discourses about how research interacts with the rest of society. Here we use ‘co-production’ as shorthand to discuss these practices, but acknowledge the diversity and differences in the above.

People argue that collaborative and co-productive research

Facing up to the challenges and costs of co-production

Undoubtedly, making research more relevant and used is the aim for many applied researchers, and their attempts to do so and document this are laudable. There is a need to help uncover if, when, and how collaboration is the best way to achieve this (normative) aim.

Firstly, there is very limited empirical evidence about whether collaborative research does lead to improved uptake of findings – even if we could agree what that looks like in reality.

Secondly, collaborative research brings significant challenges. Collaborative research may be more uncertain, slower, or less innovative than non-collaborative research. These challenges and related costs may be experienced throughout the research process, as the lists below indicates.

Challenges of coproduction

Developing mixed research teams

  • Stakeholders not homogeneous, and can disagree
  • ‘Usual suspects’ can take over, where coproductive discussions are dominated by certain individuals

Framing research questions

  • Stakeholders and researchers may have different priorities and values
  • Useful research can lack originality
  • Research can be co-opted by partners, for example, to justify status quo or historical decisions

Collecting data

  • Researchers may pressure stakeholders to allow their organisational resources to be used to facilitate data collection – e.g. using staff time or applying pressure for site access

Analysing and interpreting data

  • Stakeholders may want to know which participant agreed to participate or what they contributed to the dataset
  • Stakeholders may want to help analyse the data

Formulating recommendations

  • May be little agreement about the importance of research
  • Researchers may be pressed to frame findings in particular ways

Disseminating research

  • Researchers or stakeholders may be prevented from sharing unwanted findings
  • Stakeholders may want to share findings before researchers are ready

Implementing change

  • Tension between advocating for research, or advocating for policy/practice changes
  • Researchers show little interest in providing assistance with implementation efforts

Costs of coproduction

Developing mixed research teams

  • The research process may take more time compared to a traditional research process
  • Shared decision-making is threatened when process dominated by certain voices or interests

Framing research questions

  • Damage to interpersonal or organisational relationships
  • Damage to research careers
  • Damage to researcher independence and credibility

Collecting data

  • Damage to interpersonal or organisational relationships, particularly with more powerful stakeholders

Analysing and interpreting data

  • Violation of research ethics obligations
  • Researcher needs to train stakeholders and format data in an appropriate way to conform with research ethics obligations

Formulating recommendations

  • Findings are misrepresented
  • Damage to researcher independence and credibility

Disseminating research

  • Damage to researcher independence and credibility
  • Damage to the credibility of the research process

Implementing change

  • Can damage relationship with practice or policy colleagues
  • Implementation of research findings fail

Finally, there are other very significant potential costs of co-production, which, in our experience are often unequally borne by junior, untenured, female members of staff.

  • Practical and administrative resources such as the time required to arrange meetings with busy people for whom research is not their primary activity. Managing group dynamics in such meetings can require interpersonal skills with which researchers may not be endowed.
  • Stressful interpersonal interactions: these can be dismissed as mere relational difficulties. We must take the potential for disagreement, conflict, reputational risk and power imbalances seriously as part of the research process. The consequences of mismanaging these are severe.
  • Individual researchers already have to balance their teaching, research and administrative responsibilities. Developing another set of professional skills and networks to create collaborative research projects with real world impact is simply too great a leap for some. The implication of this is that some or all of the co-production activities related to a research project could be led by a specialist in knowledge transfer and exchange rather than by members of the research team. This approach has been advocated and implemented by some.
  • Some feel that taking part in applied, highly collaborative research can lead to researchers becoming, or seeming partisan and biased, or as academic “lightweights” producing little of substance
  • The research outputs themselves may be co-opted to serve the political agendas of others. Co-produced research findings (possibly no different from any research) can be appropriated and used to serve the self-interest of more powerful groups. Some groups lack the skills to engage in the use and promotion of research findings so lose out to more skilled and better connected groups. There is also the risk that co-produced research is more likely than other forms of research to produce findings biased in favour of prevailing norms of what is ‘correct’. This last type of research leads to repetitious, ‘safe’ research

What do we need to do differently?

We may be able to avoid some of these costs, or they may be an intrinsic part of collaborative processes, in which case we need to work out the best balance between costs and benefits.

Armed with a better understanding of the costs and benefits of co-production, those planning new projects should be much better placed consistently to ask themselves:

  1. What is everyone bringing to the table? For example, policy-makers and funders bring money, knowledge of the political context, pressure for answers; researchers bring topic and methodological expertise; the public and patients bring their experiences.
  2. Under which circumstances are these resources needed, for what purpose, and at which stages of the research process? For example, when is it better to have patient representatives articulate the user perspective rather than derive understanding from a systematic review of patient experiences?
  3. What are the costs, and how will they be borne and defrayed by those involved?
  4. How will decisions about the direction of the research be taken, and how will responsibility and accountability for decisions be shared? Will group dynamics, market forces, formal authority or some other basis be used? In turn, how will this be governed?

In parallel, research organisations and funders also need to consider:

  1. How to create (co-create) and support the infrastructure and leadership for co-production
  2. How to provide training in co-production, and help interested researchers and funders take this seriously as a necessary skill
  3. How to reward good practice, and to recognise the work co-production may take even if it does not lead to research impact
  4. How to evaluate the potential impact(s) of co-production
  5. How to ensure that co-production supports diversity and quality in research and policy

Conclusion

Co-production is an exciting approach to research that can, with care, generate truly novel, unexpected findings and impacts. Yet it takes time and investment, and there is still little evidence about how co-production changes research, policy or practice, or how it compares to alternatives. We think more reflection about how, why and when we do co-production would be helpful, as would more discussion about how co-production influences the process of research, and the roles and responsibilities of everyone involved in collaborations. Ultimately, we need to be clear about how and why exactly we work with each other, to transform evidence for society.

This blog grows from a paper by the authors published in Health Research Policy and Systems 2019: 17: 33 https://doi.org/10.1186/s12961-019-0432-3

Kathryn Oliver is Associate Professor of Sociology and Public Health, Department of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, UK

Anita Kothari is Associate Professor, School of Health Studies, University of Western Ontario, Canada

Nicholas Mays is Director of the Policy Innovation and Evaluation Research Unit (PIRU) and Professor of Health Policy, Department of Health Services Research & Policy, London School of Hygiene and Tropical Medicine, UK

SIBs may provide components for outcome-based contracting, but they are not a panacea
6 September 2018

BY STEFANIE TAN, ALEC FRASER & NICHOLAS MAYS

Public service reform can benefit from outcome-focussed approaches to commissioning. However, we should avoid assuming that Social Impact Bonds are likely to have wide application, explain PIRU’s research team.

Social Impact Bonds (SIBs) are likely to become just one part of the search among policy makers, practitioners and researchers for viable approaches to outcomes-based commissioning of public services. That’s what we conclude in our assessment of the SIBs Trailblazers in health and social care in England, set out in our recent report. Here we reflect on a number of issues raised in our own research and by previous blogs in the PIRU series on SIBs.

Testing, as yet unproven, outcome-based models of commissioning is important for public services, which are often clear about their goals but hamstrung by inadequate implementation including unsupportive funding mechanisms. In health policy, for example, there is a goal to develop healthier citizens – not just to treat more sick people. Reaching that goal could reconcile the ambition to provide better health care with the need to control budgets. However, how can health and social care – and the other parts of the public sector – work together to achieve such goals? We lack a tried and trusted mechanism.

Focus on outcomes-based funding
In this light, the focus that SIBs provide on developing outcomes-based approaches to commissioning services is welcome. SIBs in health and social care have the potential to break through institutional inertia and generate innovations in public service delivery. At best, they can free and motivate front-line service provider staff to focus strongly on the needs and desired outcomes of their clients.

One of the issues holding back public service innovation, particularly in the last decade, has been a lack of funding for prevention as the demands of those currently in need of support have become ever more pressing. Amid austerity and lack of time for contract management, commissioners have found it difficult to think strategically. In this context, SIBs have been welcomed by policy-makers, commissioners and providers, drawn by the promise of upfront funds to develop services designed to improve outcomes in the longer term.

Issues with SIB upfront funds
What’s not to like about that promise? We have two caveats. Some Third Sector organisations lack the capacity to monitor user outcomes well and thus to demonstrate accountability for the upfront funding that they receive. Thus, upfront funding needs to be used not just for service delivery, but also for developing capacity at the provider level in areas such as outcomes measurement and data management.

A second - and perhaps more fundamental - concern about SIBs’ provision of upfront funds is whether this is an efficient way to generate extra cash. The injection of more funds can be seen as a good thing per se, but, it brings with it two further problems.

First, investors are often risk-averse, which can get in the way of innovation, a raison d’être for SIBs and for outcomes-based funding in general. There may also be clashes between the values of private investors and those of the ultimate payer for results – the public commissioner on behalf of the public/tax payer. Additionally, there may be adverse impacts on Third Sector staff’s motivation if the pursuit of specific outcomes is prioritised ahead of meeting the specific needs of individual clients.

Second, many SIBs are set up with the help of intermediary organisations which can add to their cost. As a result, these costs have to be set against the value of any client outcome improvements. The activities of such intermediaries may also inhibit capacity building at the provider level if they assume long-term project management and monitoring roles.

Need for experiments to stretch the SIB model
Ultimately, the public purse foots the bill for SIBs, even if this only takes place after the successful delivery of agreed outcome improvements. Yet, there may well be other more efficient approaches to outcomes funding. We may need to have experiments that stretch the already flexible SIB model and compare it with other approaches. So, for example, Big Society Capital, the Big Lottery Fund, or other socially minded investors could experiment with conditional grants, rooted in outcomes-based contracts, to spur innovation in service delivery, and compare these with SIBs. In her contribution to this blog series, Katy Pillai, Investment Director at Big Issue Invest, signalled that investors are keen to support research into what works best in this regard.

In our experience, most up-front SIB finance does not come from business investors. It comes from philanthropic investors, alongside central government sources such as the Life Chances Fund. For example, the eventual outcomes payments made to investors in the Trailblazer SIBs which we studied were mostly paid for by tax-funded public commissioners upon delivery of the service. Upfront philanthropic funds (investors) were useful and the Big Lottery (which provides a quasi-governmental subsidy for outcome payments) helped to fund the additional effort needed for commissioners to consider and experiment with outcomes payments.

Given how SIBs are largely working this way in practice, would it be wise to experiment with a different SIB model that removes the commercial imperative altogether? This SIB model is, in ways, a variant on the experiments suggested above for socially-minded investors such as Big Society Capital and the Big Lottery Fund. It could amount essentially to organising forgivable loans to Third Sector providers from philanthropic agents, alongside dedicated contract and performance management support. Such an approach would remove the financial architecture of ‘investment’ that surrounds SIBs at the moment but could reduce their transaction costs and accelerate the innovation process.

Clarify values of outcome-based contracting
A big issue when discussing SIBs – or any form of outcome-based funding – is to define their purpose. Are they simply about driving efficiency and making savings to the public purse or are they about transforming the social value of services? Are they a tool to create better monitoring and evaluation cultures in non-profit organisations, so improving accountability and value for money? What are the values being prioritised in such contracts? Thinking and practice in the Asia Pacific region, discussed in the blog by Chih Hoong Sin and Ichiro Tsukamoto, focussed, for example, on a further role for SIBs in transforming the relationship between civic society and the State.

What PIRU and other research has found
So far at least, it has not generally been possible to show that SIBs have produced cashable savings to the public purse. Globally, the main demonstrable success of SIB projects has been in helping marginalised groups that had previously been badly neglected by public services. Here, there was often a baseline of little or no provision, so it was highly likely that anything would have helped in such circumstances. In areas with some previous provision, it is less clear that SIB-funded services, for example, for people with chronic health conditions, have led to superior improvements in health.

Next Steps
The general argument, in principle, for developing outcome-based commissioning and contracting is strong, given the difficulties that policy makers have in matching their goals with the limited funding mechanisms currently available to them. However, we are some distance from resolving all the difficulties in making an outcomes-focus work in public service commissioning and in establishing the case for applying these approaches at scale. So, it is wise to experiment widely and to keep learning.

In this light, policy makers should focus on the components within SIBs that show promise in strengthening outcome-based contracting, while avoiding the notion that the whole package offered by a SIB is necessarily the way forward. SIBs, as currently conceived, may have some dedicated roles, but these are likely to be in particular circumstances (for instance, when outcomes are relatively easy to measure and relatively uncontroversial). The greater value of SIBs and related thinking, in the long run, may be in informing the wider development of other models of outcome-based contracting for mainstream service delivery rather than in replacing these approaches.

Stefanie Tan and Alec Fraser are Research Fellows at the London School of Hygiene & Tropical Medicine and part of the Policy Innovation Research Unit (PIRU) at LSHTM.

Nicholas Mays is Professor of Health Policy at LSHTM and Director of PIRU.

Japan highlights innovative Asia Pacific model for Social Impact Bonds
14 May 2018

BY CHIH HOONG SIN & ICHIRO TSUKAMOTO

SIBs are emerging as a way to reconfigure relationships between the state and civil society in ways that their Western inventors may not have anticipated but may wish to replicate.

Japan’s experience of Social Impact Bonds (SIBs) offers some important insights. First, it challenges the notion - held by some - that SIBs should be seen chiefly as a way to cut costs: in Japan we see a typically Asia-Pacific framing of SIBs that’s more focussed on developing civic society. It’s an approach that may catch on more widely around the world.

Secondly, the Japanese experience demonstrates the importance of institutional infrastructure and culture – and not only availability of finance - to make SIBs work well. SIBs require a complicated culture in which to thrive: Japan shows how it can take time for the various elements to be established.

Japan expressed an early interest in SIBs, becoming part of the OECD Social Impact Investment Taskforce. A five year nationally-funded research project, based at Meiji University in Tokyo, looking into developing SIBs for Japan is almost complete.

On the face of it, Japan might look to be in the same boat as, for example, the UK. The country’s national debt at the end of the last fiscal year was 254% of GDP, compared with the UK’s 88%. So, one might think, Japan would use SIBs to drive down spending and increase efficiency.

In fact, although Japanese rhetoric initially focussed on fiscal savings, interest swiftly shifted to SIBs steering improvements in societal wellbeing that might require more, rather than less, public spending. Japan has also been much more like other Asia-Pacific countries where SIBs have been seen as a means to reconfigure relationships between the state and civil society and focus on long-term issues of public health.

Past and existing projects have looked at, for example, how to create learning supports in deprived areas for NEET young people – those not in education, employment or training. A nationwide health SIB has just been launched focussing on better managing or preventing long-term conditions such as dementia, diabetes, cancer, cardio vascular disease.

Japan’s government has shown a high level of interest in SIBs. It’s seen its role largely as being to signal interest and legislating to make money available. In December 2016 Japan harnessed dormant bank accounts, like the UK has done, to set up the equivalent of Big Society capital, which is envisaged to go live from 2019. In addition, the Ministry of Economy, Trade and Industry (METI) and the Ministry of Health, Labour and Welfare have provided pilots around the country with funding for operations.

So pilot programmes have tended to suffer not from shortages of capital. Their problems have, rather, concerned lack of measurement or been because service providers are short on thinking or capacity to deliver outcomes-based contracts.

Institutional weaknesses in Japan
Japan has been institutionally handicapped in developing SIBs in a number of ways. The country has a weak public management culture: transparency, a focus on efficiency and on prioritising clients’ views of outcomes are less developed than in the UK.

The country is only slowly creating the building blocks for outcomes-based markets. The UK government has supported the creation of outcome payers: SIB investment funds such as the Innovation Fund, the Fair Chance Fund, Commissioning Better Outcomes Fund, and the Life Chances Fund. Britain has invested in developing an evidence base that can support outcomes-based contracting. There is also legislation that directly or indirectly supports SIBs, including the Public Services Social Value Act and provision of social investment tax relief. Japan has had a much less developed basic infrastructure.

In comparison, the UK’s voluntary – not-for-profit – sector is also more robust, diverse and heterogeneous, including organisations of different sizes. There are strong umbrella bodies such as the National Council for Voluntary Organisations. Japan has a long history of social sector organisations but the flowering of the not-for-profit sector is more recent, encouraged in the past 20 years by legislation liberalising regulation of the sector.

Many Japanese organisations remain small and hyper-local, which raises questions about their capacity to engage with SIBs. The country’s not-for-profit umbrella bodies are also relatively weak in terms of their lobbying capacities and the extent of their networks across the sector.

There are fewer charitable foundations than in the UK. So, social sector organisations in Japan largely rely largely on government funding, whereas UK social sector organisations tend to have more diverse funding. This financial dependence makes it harder for social sector organisations to question government.

Problems measuring impact
These days, the UK social sector is familiar with measuring and demonstrating impact. Japan’s social sector organisations are less experienced in measurement and, where they are, their focus tends largely to be on activities and inputs.

Lack of diversity in markets - in terms of service providers, investors and intermediaries – also hampers the growth of SIBs in Japan. Until 2017, when the Yokohama SIB was developed, all pilots were driven by the Nippon Foundation or its subsidiaries. So a single, large player was trying to nudge the market along.

There are significant differences between the UK and Japan in the maturity of intermediary markets. The UK has moved from a virtual monopoly among a few intermediaries in the early days of SIBs to a growing, diverse market, bringing in many different skills. Intermediaries no longer offer only financial backgrounds. There are people with commissioning support experience as well as good provider engagement experience. The same cannot be said quite of Japan, where the emergent intermediary market remains in its infancy.

From the commissioner side in Japan there remains a poor understanding of SIBs so it is difficult to collaborate with providers about constructing SIB models. As a result, commissioners may not know what to specify. They tend to identify general outcomes which may lack meaning and be unachievable within the timescales.

Building underdeveloped civil society
Nonetheless, Japan’s vision for SIBs is innovative as it tries to use them to place individuals at the heart of public services that have tended to be dominated by centralised bureaucracies. As in China, SIBs offer Japan a model that facilitates an underdeveloped civil society sector to grow and play a more visible role. SIBs allow responsibility to be devolved downwards while retaining central control through outcome-based funding. SIBs help square the circle for centralised governmental structures that wish to devolve responsibility while retaining policy direction.

Western countries such as the UK, where SIBs originated, would be wise to take stock of this evolving emphasis. Just as football played in Brazil and Spain has turned out to be more exciting than the game played by its inventers, SIBs, in their cultural adaptation, may prove to be a more transformative model than originally envisaged in Britain.

Attractions of Asia Pacific model
This adapted model could turn out to be an attractive export even to its inventors. Austerity – the ground into which SIBs were originally planted – will not last forever. SIBs may, in the longer run, offer opportunities for a more ambitious and hopeful agenda of improved well-being that may be politically realistic in more prosperous times. Such reinvention and rebranding of SIBs might also make them more attractive to their Western critics. The Left in Britain and US has tended to dismiss SIBs, suspicious of them as vehicles for commodisation and privatisation of public services. A revisioning of SIBs might suit a more left-wing agenda.

This SIBs model with a different emphasis might also offer an approach to reform in Britain where big providers such as Serco and Capita - focussed on driving down costs rather than on quality delivery to the person – are edging out smaller local players. Since 2010, many small civil society organisations have collapsed in the UK and others have had to merge as size has become the key to survival. Might SIBs eventually offer a way to address this trend and recognise financially the value of local deliverers with smaller caseloads but higher quality?

As the SIBs story unfolds, the purposes and future of this approach to service delivery remain in flux. Analysing the Japanese – and wider Asia Pacific - approach is a reminder that we may not have scratched the surface of SIB’s potential for achieving a wide range of agenda.

Dr Chih Hoong Sin is Director of Innovation at the Office of Public Management, London, UK

Ichiro Tsukamoto is Professor of Public Management at the School of Business Administration, Meiji University, in Tokyo, Japan.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

We should ask three big questions about SIBs
8 May 2018

BY ELEANOR CARTER & CLARE FITZGERALD

Evaluation should test key SIB promises to government: greater collaboration, prevention and innovation, argue two researchers from Oxford University’s Government Outcomes Lab.

To discover whether Social Impact Bonds work, we would be wise, first of all, to ask: “Work for whom?” Our response is: “For public service commissioners.” This answer acknowledges a reality: if SIBs don’t make sense to government, they are unlikely to be more than a passing fashion.

This standpoint cuts through a myriad of promises that typically surround SIBs and other payment by results programmes. It also helps focus how to evaluate them. We should measure SIBs by looking at what, if anything, they can deliver that public service commissioners seek.

That’s what we’re doing in the Government Outcomes Lab – GO Lab – at the Blavatnik School of Government, where our work is currently co-sponsored by the School and the UK government.

The dominant narrative around SIBs, born in an age of austerity, is that they can deliver cashable savings to government. Of course, every government seeks such dividends. However, this claim, far from proven - at least at scale - may or may not be true. Either way, we believe SIBs go well beyond promising the efficiency and entrepreneurialism claimed by conventional outsourcing. SIBs make a more fundamental offer that should be rigorously tested: that they provide an opportunity to remake the public sector itself.

Public sector reform has sat at the heart of every government’s agenda – Conservative and Labour – for nearly 40 years, since the election of Margaret Thatcher in 1979. Its goals – aside from making savings – have included three principles that SIBs directly claim to address.

SIBs address 3 principles of public sector reform
The first principle is to improve collaboration in the commissioning and provision of services so that the collective effort is better focused on the needs of the service user.

The second principle is to shift activity to a more preventive footing, so that service provision is moved upstream of social challenges and towards earlier identification of risk.

The third principle is to make innovation less risky for commissioners so they move to evidence-based procurement, decommissioning underperforming services and becoming better at reaching those, often experiencing the most complex mix of social issues, who may be poorest served by existing provision.

We should, therefore, question, in our evaluations, whether SIB models are improving collaboration, shifting services towards prevention and creating spaces in which public service commissioners and delivery organizations can innovate.

Principles largely untested by evaluations
Our review of evaluations suggests that they rarely explicitly or rigorously test any these three principles by comparing a SIB commissioning approach with a grant, fee-for-service or even in-house delivery for a given population. Evaluations also face the “doughnut challenge”. That’s the difficulty, particularly with impact evaluations, in teasing apart the effects of the commissioning architecture (the dough) from the effects of frontline services operating in different ways (the jam). Indeed, most evaluation material is quite descriptive and exploratory. It doesn’t typically offer or test a mechanism by which a particular SIB is expected to work in practice.

Collaboration is seen
However, where we have tried to read these three principles into the evaluations that have been produced, we see glimmers of each logic being present in SIBs. So Peterborough’s SIB – focused on reducing ex-offender recidivism – saw front-line organisations pull together in cohesive ways, wrapping around the service user.

We’ve also seen multiple commissioners joining forces as, for example, where the Department for Education, Cabinet Office and the Department for Work and Pensions came together for a fund-based approach to support young people vulnerable to disengaging from school and work. We see top-up funding approaches from the Cabinet Office to acknowledge that there might be wider public service benefits springing from a SIB that are not captured within the outcome payer’s domain.

“Step-down services”, more than prevention
As far as prevention is concerned, very early intervention programmes are rare in the SIB field, but there is a focus on “step-down” services. The Birmingham Foster Care SIB is a good example that is already demonstrating cashable savings.

A growing number of SIBs try to de-escalate situations of family conflict or address unstable conditions for children and young people. Typically, for example, the Department for Work and Pensions has worked with people only once they are employed. Now, through SIB programmes, the Department, through the Young Engagement Fund, is becoming involved in programmes that try to understand and anticipate future needs. It is working with school children to prepare them for a smooth transition into work.

Space for innovation
There are also examples of SIBs supporting innovation and helping to dissipate cultural inertia. Typically commissioning has a cycle – there’s the planning stage, the doing phase and then there’s the evaluation. A SIB can help reduce risk at any or all of these stages, allowing SIB commissioners flexibility to think differently about their work. So, for example, with an innovative programme such as the GLA Rough Sleeping project, a SIB allowed commissioners to ask of themselves: ‘Do I know how to be the best contract holder, or would I benefit from bringing in a third party that might have a better appreciation or knowledge base about that side of things?’

SIBs can also enable a certain amount of experimentation by providing quicker feedback through performance management. It’s this feedback that allows frontline organisations to change the way they are doing things to better meet the needs of their clients. A well-designed SIB also creates impetus for high quality impact evaluation within a payment mechanism. That inclusion can, in the long run, only help to improve the quality of future commissioning.

Time to test public sector reform potential
In conclusion, we cannot demonstrate, on the basis of existing evaluation evidence, that SIBs fulfil the many and varied promises that their champions make for them. But observing SIBs through the lens of public sector reform – and their possible role in it – we offer an approach to evaluating whether they might be a tool for remaking the public sector.

Eleanor Carter and Clare FitzGerald are research fellows at the Government Outcomes Lab (GO Lab)at Oxford University’s Blavatnik School of Government.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Investors need rigorous assessments of Social Impact Bonds
30 April 2018

BY KATY PILLAI

A major investor highlights the vital role that research and evaluation should play in developing this form of outcomes funding.

Evaluation and research into Social Impact Bonds (SIBs) is a hot topic for Big Issue Invest. We are one of the UK’s leading social impact investment firms, having invested in approximately 350 charities and social enterprises since 2005, in our quest to dismantle poverty and create opportunity.

Our investments support areas such as access to housing, financial and social inclusion, mental and physical health and well-being, and employment, education and training.

We have made several SIB investments since 2012 and have watched the market develop. Our first-hand experience is that there is more work to be done to refine the model, but there have been impressive outcomes for programme participants and the charities delivering the contracts in which we have invested.

How we use research and evaluation in SIBs
We aim to address the structural challenges of SIBs and maximise their individual and collective social impact. Research and evaluation can help in this goal. Many commissioners and service delivery providers are unfamiliar with SIBs and I often direct them to impartial, well-informed research to build awareness and understanding.

Discussing Big Issue Invest’s learnings and experiences from individual programmes with evaluators helps us to contextualise the situation and identify emerging trends in this rapidly-developing field. These partners can develop the tools and models to test and critique our theories and insights ‘from the field’ and evaluate the wider market, whereas our frame of reference is often limited to our specific investments.

Research in social investment and SIBs is at a nascent stage. It is crucial that the right foundations are laid today to enable good-quality ‘market’ level analysis in the future. Consistency and rigour of approach will pave the way for systematic reviews and meta-analyses, essential if outcomes-based approaches are to become commonplace in the commissioner and policymaker toolkit.

We also welcome more quantitative approaches and ambitious evaluations that compare SIBs to traditional fee-for-service mechanisms or payment by results more broadly. SIBs are often conflated with outcomes-based approaches in general, which makes it very hard for investors to assess – and improve – their social impact.

Funding is, of course, needed to allow this work to take place. At the moment, evaluations are too often the balancing item in a very limited budget, constraining their ambition. We wholeheartedly support the calls for a ring-fenced fund for SIB evaluations, recognising the value of the output to commissioners, central government and potentially also philanthropic funders who might seed the fund.

Understanding what, why, how
We are an impact investor: outcomes are our reason for being, not just a by-product of our investment activity. Social impact due diligence underpins every investment decision we make.

We look closely at the theory of change for each SIB. There needs to be a coherent and credible hypothesis for how outcomes will be improved for the programme participants and - beyond that - how the programme could help to tackle the underlying issue through, for example, earlier intervention. We interrogate potential perverse incentives in order to mitigate them. Research and evaluation from previous programmes helps us to do this: we rely on it to validate the causal link between inputs, outputs and outcomes and complete our due diligence of the intervention.

It’s important that everyone involved reviews the theory of change periodically after the contract launches. One of the strengths of SIBs is that they shine a light on what works and what doesn’t, enabling real-time improvements and sharing of learnings for future contracts. If we scrimp on monitoring and evaluation, we undermine the programme and indeed the SIB model.

Evaluations therefore need to be robust, relevant and timely. We want to understand not only the results achieved but also why they are (or are not) achieved and how we can replicate and improve on them. That might be a programme evaluation or an impact evaluation, or qualitative or quantitative approach, depending on the context but we certainly need more than outcomes verification.

We seek insights into the drivers of success so they can be reflected in future projects. The GLA Rough Sleeping SIB in 2012 was divided into two lots awarded to separate providers, one funded by Big Issue Invest. We know the absolute outcomes achieved by the programme but would like to dig deeper into whether different operational or investment approaches had a bearing on success.

We are keen to work with the evaluator community to design the evaluations and contribute to them. It’s important to be confident that Big Issue Invest’s loan has achieved its social objectives – and those of our investors in turn – so we are a consumer of evaluations as well as contributors to them.

We are one of very few organisations that has worked across several SIBs in different regions and policy areas. We can contribute data, insight and practical experience and welcome the opportunity to do so. At a practical level, we can coordinate with the evaluator to minimise the data collection burden on the service provider’s staff and the programme participants. If we can bring the evaluator in to the design phase early, we can also incorporate their evaluation into the delivery model early to avoid duplication or complication down the line.

Using data and analysis to target interventions
Reliable data and analysis is essential to high-quality SIB design. For example, we are involved in an ‘edge of care’ SIB for young children where it is unfeasible to roll out an intensive (in other words, expensive) intervention to all children on social services’ radar. Rather than only work with children on the very cusp of care – when it is often too late to reverse their trajectory or the trauma they have suffered - a researcher is working with commissioner data to identify early risk factors that increase the child’s propensity to enter care. The programme will be targeted towards these high risk children as well as those on the very cusp of care. This allows the commissioner to fund an early intervention service that is also cost-effective, often challenging in outcomes-based commissioning. There is huge potential to harness data and analysis in this way to design preventative services.

The value of timely feedback
Speed of evaluations is a challenge. Evaluations are valuable to SIB stakeholders when developing follow-up programmes and carrying out due diligence. If investors have a good level of confidence in the achievable outcomes, the cost of their risk capital should be lower. That is in everyone’s interest. It doesn’t encourage evidence-based commissioning if the evidence is released after the next programme is launched!

Midline and end-line reviews as part of a formal evaluation are, of course, extremely important but they are not enough. Outcomes-based programmes also need shorter, informal feedback loops, preferably involving the evaluator. Early results and findings can be used to improve programme delivery - but not if they are shared in an end-of-programme evaluation that takes a year to publish. Ideally, we’d like a quarterly or a six monthly check-in with the evaluators that can identify and unpick performance and its drivers.

We recognise the tension between this approach and concerns that the evaluation will influence the programme outcomes. A balance needs to be struck. SIBs support people with complex needs who deserve the best possible chance of better life outcomes, so although evaluation rigour is crucial, we owe it to them to make the intervention as effective as it can be.

Importance of counterfactuals
Three inputs are usually needed to assign a value to a SIB outcome: (1) the projected costs to deliver a programme (preferably validated through a competitive procurement process); (2) the costs per outcome achieved under comparable programmes, if known; and (3) the savings case (the projected benefits for the commissioner). If you don’t have an understanding of what would have happened anyway, at least one of these calculations will be flawed. That’s why we can’t afford to disregard the counterfactual.

That doesn’t mean that every SIB needs to link payments to performance compared to a counterfactual, measured by an RCT. There are lots of factors to consider when designing the payment mechanism and there is no single ‘right’ approach. However the counterfactual can always be taken into account. Under a rate card approach, the rates should be set after considering deadweight – even if the assessment is imperfect, it is better than ignoring it completely. The counterfactual can then be assessed in the programme evaluation and used to inform the pricing of future contracts.

I am not saying SIBs should be commissioned only if there is perfect data to value the counterfactual. Rather, I am emphasising the need for new approaches that measure outcomes and cost-per-outcome to allow commissioners to make evidence-based decisions in future. Big Issue Invest is trialling approaches that allow an outcomes-based contract to be launched with imperfect information, while ensuring checks and balances limit windfall gains and losses and include mechanisms to tackle the information gap.

One option is to run an initial ‘discovery phase’ of the contract for one to two years. The discovery phase outcomes pricing estimates the counterfactual, but sets parameters to ensure that no party makes excess gains or losses. In this way, the partners have the opportunity to implement the SIB delivery model. During this time, outcomes and the counterfactual are measured rigorously. The data and analysis is then incorporated into a revised payment mechanism for the rest of the contract, after which point it operates like a ‘standard’ SIB. This approach bridges the knowledge gap without delaying a potentially high-impact programme or risking inequitable risk and return.
Where next?

SIBs bring together different worlds. The success of SIBs is dependent on partnerships where the whole is greater than the sum of the parts. They require new ways of working for everyone involved – for investors, providers and commissioners. I expect they can seem strange to evaluators as well. Forging new links and understanding the perspectives of others is crucial.

We are starting to see these worlds come together and collaborate for better outcomes. There is more interaction and understanding between researchers and evaluators, policymakers and budget holders, delivery organisations, and investors. It is early days but the outlook is promising.

Katy Pillai is Investment Director, Big Issue Invest: www.bigissueinvest.com @katyjones | @bigissueinvest

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Academics can show governments how to evaluate SIBs more rigorously
23 April 2018

BY CHRIS FOX

A wide range of approaches can help identify causality and effectiveness even in complex environments.

We can – and we should – improve our evaluations of SIB and Payment By Results (PBR) programmes. They should focus more on causality, rather than simply contract compliance or implementation.

If we don’t focus on attribution, it will become hard to demonstrate that SIBs are more than a series of interesting pilots. We’ll miss the chance to test an alluring proposition – that SIBs could transform the large scale commissioning and delivery of health and social welfare programmes.

Getting evaluation right in this field is not - as some might suggest – intrinsically challenging. SIBs and PBR projects do not create unusual difficulties for evaluation techniques. We have the knowhow - sophisticated, diverse tools are well-developed that could settle most questions thrown up by SIBs. The real issue is: will those who champion SIBs expose such initiatives to the full rigour of the evaluative tools that exist?

Academic responsibility
The academic community can help ensure that rigour. Management consultants, contracted to perform evaluations, tend to provide what governments specify, which, so far, has been limited and fallen short of what’s required. Academics could set out a wider, more exacting range of evaluation options that are more suitable. We should show policy makers clearly how better evaluations could be achieved, particularly if the case for widespread adoption of SIBs is to be made.

This difficulty in properly assessing the impact of SIBs seems to be a particularly British problem. In the United States, most SIBs have been accompanied by fairly rigorous counterfactual evaluations, including randomised control trials (RCTs). There, the credibility of the SIB model among commissioners and investors has required demonstration of its ability to deliver tangible outcomes. This may be because, in the US, more funding has come from wealthy individuals or private foundations, with an investment ethos. In Britain, funding tends to spring from philanthropic organisations which seem interested in testing concepts over categorical outcomes.

Evaluations are too based on performance management
Whatever the reasons, SIB pay-outs in the UK typically rely more on performance management information to demonstrate the achievement of outputs. Supporters of this approach say that complicated counterfactual evaluations add to the already high transaction costs associated with SIBs. That’s understandable for individual SIBs. However, cumulatively, this approach hinders the quest to find out whether SIBs really work. It undermines the case for wider roll-out.

Evaluations can and should answer two major questions about SIBs. There’s “attribution”: whether SIBs actually achieve the outcomes desired. Second, we need to understand SIBs as a mechanism and establish how effective they are compared with other models of commissioning. This is important because there are less expensive, less complicated methods than SIBs for commissioning services in this field.

The attribution issue has become unnecessarily mired in a polarised debate about whether RCTs are suitable for SIBs projects. Opponents contend that RCTs are not particularly useful in this field because SIBs interventions tend to take place in highly complex environments. While it’s true that these interventions often occur amid complexity, that actually strengthens the case for RCTs. It becomes even more important to understand whether an intervention is indeed responsible for any of the impacts being observed.

Testing theories of change
Good RCTs would strengthen SIBs evaluations because they would be theoretically informed. They would start with a theory of change setting out the potential causal mechanisms that are of interest. In contrast, many SIBs evaluations rely on contractual frameworks and demonstrating whether they have worked, rather than testing hypothesised causality. Most good RCTs today are also accompanied by high quality implementation evaluation. So they have a dual strategy.

Well organised RCTs avoid “one-shot” design. They are actually a sequence of evaluations that build by testing, at a granular level, particular moderators of change, rather than simply focussing on the overall social outcome and trying to come to a one-shot conclusion. This is how, in reality, even medical research works. You don’t do a single RCT. You build from small scale studies through to larger scale studies.

Sequences of evaluations are good
The wider evaluation world is focussing more on sequencing evaluations and ensuring that tools employed are appropriate to the point of a programme’s development. This avoids problems that one shot evaluations can create: that you evaluate too early; that the throughputs you were promised never arrive; that you end up developing an evaluation design which is underpowered to identify the changes that you’re looking for; there are inconclusive findings that have cost a lot of money but don’t provide the hoped for insights.

I advise against the one shot model. Instead, we like to start evaluations early without diving straight in with an RCT. We focus on developing a sequence. That’s the strength of the Education Endowment Foundation evaluation model. It begins with small scale pilot studies that focus on theory of change and early implementation, then efficacy trials that are more like a small RCT, leading up finally to effectiveness trials. Only at that point - when causalities have been established - is control finally handed over to implementers.

Building commissioner confidence
This sequential approach gives commissioners confidence. You’re saying to them that this isn’t a “one shot, put all your money on the table up-front” model. It’s about gradually building knowledge and providing gate-keeping points where a commissioner can ultimately say: “This isn’t working, we need to rethink. We may need to reinvest or, even disinvest.” That’s helpful to commissioners, especially if they are being asked to back innovation that feels risky.

Small ‘n’ designs
In some cases, RCTs are not possible, but there are many alternative models of impact evaluation that could be considered for SIBs. “Small n” designs provide ways to think about causal attribution where a programme does not have sufficient numbers to allow a traditional impact evaluation design. Process tracing is an example of “small n” design, where one uses theory to identify critical points in a change process that need to be tested. Then one selects cases to test these critical points, using interviews and observations of what’s going on. This Popperian approach acknowledges that there is no absolute objective knowledge. However, it can find ‘smoking gun’ evidence that strongly suggests causality, even if that may not amount to absolute proof.

These process approaches that search out causality would be an improvement on current tests of some SIB or PBR programmes which, if they can’t do an RCT, tend to opt for process/implementation evaluations that are less demanding - usually interviewing stakeholders and writing a report, but lacking a more theory-driven approach.

More rigour is needed
I’ve set out ways in which SIB and PBR evaluations could be improved by RCTs or hybrids that avoid the unnecessarily polarised debate between the pro- and anti-RCT lobbies. Beyond RCTs, there are other approaches to evaluating causality, suitable in instances where there are small numbers of cases. We should learn from this wider discussion of evaluation techniques. Academics owe it to those investing and working in SIBs to ensure that policy makers adopt a rigorous approach to evaluation. We need to know what works and what doesn’t if SIBs are ever to be widely adopted.

Chris Fox is Professor of Evaluation and Policy Analysis and Director of the Policy Evaluation and Research Unit at Manchester Metropolitan University. He is co-author of “Payment by results and social impact bonds: Outcome-based payment systems in the UK and US”, published by Policy Press in February 2018.
https://policypress.co.uk/payment-by-results-and-social-impact-bonds

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Impact bonds could offer a paradigm shift towards more effective public services
16 April 2018

BY EMILY GUSTAFSSON-WRIGHT

Social and Development Impact Bonds require enormous effort for the partners involved, but they have a potential to transform the financing and delivery of social services across the globe.

In winter 2015, Courtney arrived at Frontline Services, a not-for-profit US organisation that helps citizens in Cleveland, Ohio. She was 28, living in a shelter for homeless women, struggling with mental health and substance abuse issues and parenting three young children who were in the custody of the county.

Courtney had just about given up hope that she would ever be able to care for her children on her own. Until that point, the county caseworker assigned to her family had little incentive to reunite Courtney and her children because the caseworker’s primary job was to protect the children.

Living with a birth parent is almost always better for a child’s development than foster care, provided the home environment is safe and healthy. Nevertheless, before entering Frontline Services, Courtney had few ways to change the trajectory of her children’s lives. Fortunately for her children, she was walking that day into a social services experiment, one of only seven similar experiments in the US at the time. In this experiment, a social impact bond (SIB) – designed to “pay for success” - the county’s government had pledged to repay private investors for successful reductions in out-of-home placements for children whose primary caregivers were homeless.

This incentive meant that Courtney was assigned a caseworker dedicated to her - someone who would look at her particular circumstances and tailor a plan to help her turn her life around and unite her with her children. Courtney’s caseworker could work across county service providers to identify the right mix of services for her.

The SIB meant that a dedicated group of stakeholders was meeting regularly across government and non-government entities to focus on one thing - reuniting Courtney with her children, and doing the same for other families in similar circumstances.

SIB contracts focus on outcomes, so service providers tailor their services to what works for the target population. They helped Courtney to address her debts with classes in financial management and offered family counselling.

As a result, Courtney was able to reunite with her children, enrol them in supportive school environments and stop the cycle of dependency on the foster care system. The result was not only a better family outcome, but also a reduction in the enormous costs to the county, had Courtney’s children remained in the county’s care.

Impact bonds are changing developing countries
Meanwhile, nearly 12,000 km away, in a village in rural Rajasthan, India, lives a 13-year-old girl, named Punam. She comes from a poor family - her parents are labourers. Although Punam started school at age seven, she became one of India’s three million out-of-school girls, when she was forced to drop out to tend to her family’s goats.

In the same year as Courtney arrived at Frontline Services in Cleveland, Ohio, a field co-ordinator, working for an organisation called “Educate Girls”, arrived in Punam’s home in Rajasthan. He spoke with her parents, explained the benefits of educating Punam and tried to convince them to send her back to school.

Even after multiple attempts, the parents didn’t agree to send Punam back to school. The Educate Girls caseworker returned some weeks later with a volunteer from the community but again failed to persuade the parents. “What benefit will it give her or us?” they asked. “She will eventually marry and her responsibilities will revolve around doing household chores, assisting in farming life, raising children and taking care of her family.”

Nevertheless, Educate Girls made a further attempt to encourage her parents to let Punam attend school. This time, they asked the school’s principal to join them in a final visit to her home. With this added influence, Punam’s parents agreed finally to sending their daughter to school.

Why was an impact bond so important in this case? Because the contract was based on the achievement of outputs and outcomes, Educate Girls field-staff were empowered to innovate at the field level, trying to find solutions for getting Punam into school. Now, two years later, Punam, and many girls like her, are enrolled in and enjoying school thanks to Educate Girls and this Development Impact Bond (DIB), based on the same principle as a SIB, but with a third-party outcome funder, instead of the government.

These two stories capture the real human benefit that can emerge from outcome-based contracts such as SIBs or DIBs.

How impact bonds work
Let’s just re-cap for a moment how impact bonds actually work. In an impact bond, private investors supply upfront capital to service providers to deliver an intervention or program to a population in need. Upon the achievement of a set of agreed-upon results, the investors are then repaid by an outcome funder. With a SIB, this outcome funder is the government. With a DIB, outcomes are financed by a third-party organisation, such as a foundation or donor.

Since the launch of the first SIB in the UK in 2010, the impact bond market has grown exponentially. Last year, some 32 new contracts were signed. As of January, 2018, there were 108 contracted impact bonds (103 of them SIBs, 5 of them DIBs) across 25 countries, along with many more in design. All but one of the 103 SIBs were in high income countries: last year marked the contracting of the first SIB in a low- or middle-income country, the Workforce Development SIB in Colombia.

Most (42) SIBs are in the UK, the country that pioneered the impact bond model with the Peterborough SIB in 2010. The results of that SIB – aimed at rehabilitating ex-offenders - were released last year: reoffending of short-sentenced offenders dropped by 9 percent and the investors were repaid in full. The US has also established itself as a player in the field, coming in second with 19 impact bonds.

The five DIBs include Educate Girls in India, one for coffee and cocoa production in Peru, as well as one for physical rehabilitation across three countries in West Africa, a poverty graduation program in Kenya and Uganda, and the recently launched Utkrisht impact bond for Maternal and Newborn Health in Rajasthan, India.

Characteristics of SIBs
Most SIBs contracted globally are in the employment field, followed by the social welfare sector, which includes programmes to reduce homeless – “rough” – sleeping, or reduce out-of-home placements as in the case of Courtney and her children. Other areas for SIBs are health, criminal justice, education, the environment and agriculture.

Probably about 30 or 40 impact bonds are in design in high-income countries while more than 20 are being designed in low and middle-income countries. We see some difference comparing high income and low or middle-income countries. The majority of impact bonds in the latter are in the health sector, followed by employment and, then, agriculture.

What do these impact bonds look like in terms of size? The smallest one, in terms of beneficiaries, reaches 22 individuals – that’s in Canada. The largest one reaches 650,000 individuals in Washington DC, which is an environmental impact bond focusing on developing infrastructure. It is perhaps a little bit unfair to compare that one in terms of size with the rest of them because it’s a city-wide programme. The next largest in size is the Maternal and Newborn Health DIB in Rajasthan, India, with 600,000 potential beneficiaries.
However, the median impact bonds are reaching about 565 individuals, so they are quite small. Capital commitment of bonds ranges from $80,000 to $25 million. Again, that $25 million is the one in DC. The average is about $4 million and the total upfront capital invested across the impact bonds is over $300 million.

Who is benefiting?
The bonds mostly target marginalised populations, including women affected by violence, young migrants, single mothers, with a few for ex-convicts, vulnerable and young people, people diagnosed with mental health conditions, refugees and individuals with physical disabilities.
What do we know, eight years in about their performance? There have been some outcomes achieved and payments made, such as in the case of the Peterborough SIB, mentioned earlier. In an Australian SIB, 203 children were reunited with their families and the return to investors was nearly 16.5 per cent over the four years of the scheme.

Shifts in public programme behaviour
However, perhaps the more interesting observation is a real shift seen in government and service providers to thinking about outcomes as opposed to paying for inputs. Impact bonds are also driving performance management so service providers are introducing or improving systems of performance management in their programmes.

Impact bonds are incentivising collaboration, between the public and private sectors, but also across government, vertically and horizontally. They are building a culture of evaluation because outcomes must be measured and monitored. Most impact bonds so far have been focused on investment in preventive interventions. There has also been some reduction in risk for governments, which have not paid for outcomes that weren’t achieved.

What are we not seeing so far? It had been hoped that impact bonds would lead to an influx of additional private funding. However, given that government or outcome funders ultimately repay the investors, then that’s not really more money for a particular social service. Impact bonds have also yet to achieve change at scale: the majority are reaching very few individuals and are fairly small in terms of investment.

Many thought that impact bonds would focus on experimental interventions. So far, we haven’t seen that: investors have been unwilling to take that risk. We’re seeing SIBs used in the middle phase of development of interventions, rather than at the “seed” or “at scale” ends of the process. However, the flexibility that service providers are allowed in terms of their service delivery has the potential to encourage innovation. It’s also probably too soon to say whether or not impact bonds can achieve sustainable outcomes in the long run through the systematic change that’s happening but it does appear that the partners currently involved have indeed shifted their thinking.

Challenges of impact bonds
What are the challenges? This is a new form of government contracting, a new way to do business. Co-ordination of all of the stakeholders is difficult: sometimes they don’t understand each other well; just getting all those people around the table can be really difficult. There can be some political constraints and legal barriers.

Key questions remain. Can impact bonds be used at scale? Are they more effective than input-based based financing or traditional payment-by-results? Do the actors in social service provision have the capacity to adapt to the demands of financing tied to results? Can they manage the rigorous focus on performance management that this is likely to entail?

Next steps
It is worth considering what would be needed to expand the use of impact bonds or, more generally, payment by results. The evidence base needs to grow and there is a need to collect more information on services that work and on their costs. Also, potential outcome funders and investors need to be educated in not only the potential of an impact bond approach but also the challenges. There needs to be supporting legislation and regulations to facilitate paying for outcomes both at a country and local level but also within organisations.

To achieve scale, countries could establish outcome funds for particular sectors. The UK government has launched seven outcomes funds and efforts are underway in the US to develop outcomes rate cards. There are several outcome funds being developed to tackle tough social issues in the developing world as well. These would allow the outcome funder to set prices for desired social outcomes and then to contract with service providers to deliver interventions to achieve those results. Global investment funds would also benefit from contributing to this new financing mode.

In the US, $800 billion are spent annually on social services. Only one per cent of that spending is evaluated for effectiveness. In the UK, £220bn was spent on social and health services (2015/2016), yet we know very little about the effectiveness of that expenditure. Thus we need more empirical research which asks: “What do impact bonds achieve, compared with input based financing?” It is also important to know how well impact bonds perform compared with traditional results-based financing. These are both hard questions to answer rigorously and will take some time.

Impact bonds and global problems
The social and environmental problems that we face at a global level are enormous. It’s estimated that $1.4 trillion will be needed annually to achieve the Sustainable Development Goals by 2030. There is little evidence that such complex problems can be solved by continuing the same old failed approaches.

Investing in preventive measures can avoid higher costs down the road and make the public and civil society sectors more efficient. By paying for outputs and outcomes rather than paying for inputs that have unknown outcomes, spending should be more effective.

Impact bonds may not be the right solution to every problem. However, they do represent a long-overdue, paradigm shift. They’re a means to an end, an opportunity to think about, and hopefully produce, systematic change. At the very least, they may be the stepping stone to establishing the monitoring and evaluation performance standards and output planning that can ensure every individual receives the services that they need to live safe, healthy and productive lives.

Dr Emily Gustafsson-Wright is a Fellow in the Global Economy and Development Program at the Center for Universal Education, Brookings Institution, Washington DC.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

“Rethink questions to patients in general practice and focus more on improving primary care”
20 July 2017

By Tommaso Manacorda

The “Friends and Family Test”, seeking patients’ views, has created anxiety among practices but shed little light on patients’ concerns. It’s time for a rethink, suggests our study.

The many ways in which patients can now feedback to GPs should offer a rich source of information for those keen to improve their services. However, when we evaluated the latest addition to patient consultation in England – the “Friends and Family Test” of patient satisfaction – we found that “more” did not necessarily equate with “better”.

We found that general mistrust of the FFT process, combined with inappropriate framing of its core question is undermining the initiative. But we also identified ways both to bolster trust and make patient feedback more useful for improving primary care.

Two important pieces of learning emerge from examining this initiative. First, the issues raised by the “Friends and Family Test” could usefully prompt a rethink about how views are gathered from patients about their primary care experiences. Our main recommendation is that the FFT should be revised, although discontinuing it should also be considered, as its reception within general practice has been largely negative.

Second, GPs need more encouragement and guidance to tackle inadequacies revealed by patients’ experience. We found that commitment to quality improvement is uneven across primary care and remains a low priority in some, often in those practices where patient participation groups (PPGs) meet rarely and patient surveys are infrequently conducted. Nurturing a rigorous improvement culture is at least as important as getting right the questions posed to patients.

Wording of the FFT
The “Friends and Family Test” puts a single question to patients about their general practice: “We would like you to think about your recent experience of service. How likely are you to recommend our GP practice to friends and family if they needed similar care and treatment?” Answers are recorded on a five point scale from “extremely likely” to “extremely unlikely”. Additionally, patients may be asked to comment on their reasons for the score they have given.

General practices across the NHS are required to make the FFT available to patients after every contact, collecting the data with the method that suits them best. The majority of them use handwritten cards, but tablet kiosks and online apps are also used.

Our evaluation, involving 42 practices and 118 interviews with clinicians, practice staff and patients’ representatives, found two sets of problems with the FFT. The first concerned the usefulness of the information it produced. The second problem lay with how general practices understood the FFT’s purpose and how they engaged with it.

The experiences of hospitals, where the FFT was first introduced, showed that the quantitative scores were statistically unreliable. So the metrics could not be used to compare providers for quality. That’s because the FFT does not involve a representative sample and is vulnerable to selection bias. One can’t tell whether the scores are representative of all patients. So, when the FFT question was subsequently rolled out in 2014 into primary care, the widespread awareness of these statistical limitations contributed to unease in general practices about the FFT approach.

In our study, the FFT question was deemed inappropriate by most interviewees. Many, particularly in rural areas, lacked choice of general practice, so the question of where they might send friends or family lacked reality. Also, patients found it difficult to compare their personal care with what someone else might receive, because that depends on individual factors such as age, sex and existing health conditions. Most interviewees suggested the use of a more straightforward question.

The additional space provided for further comment did potentially offer some useful feedback. However, staff in general practices felt that the anonymity of patients made it difficult to act on these comments. That’s because patients were often unclear which service they had received, from whom and the precise nature of issue that concerned them. For example, a patient complained about a “terrible phone service”. But staff said it was difficult to respond to this statement because they didn’t know who the complainant was, whom the patient had spoken to or what they needed.

Most patient feedback collected by practices was positive, but anonymity and vagueness made it difficult to identify and reward good practice. Positive feedback would often lift staff morale. However, the inability to act on complaints was often reported to be very frustrating for staff.

When additional comments gathered by the FFT pointed at some specific issues, their contribution was still considered to be of little value because the issues were already known to staff from other sources, such as practice surveys and patient participation groups.

Professionals mistrust the process
The second issue with the FFT concerned the negative reception that it received from primary care professionals. Some feared that it left them vulnerable to hostile patients. Someone might, for example, be correctly denied an antibiotic for sound clinical reasons but then score down the practice unfairly. The practice would have no way to question the scoring.

Monthly reporting of scores to the DH and NHSE added to professional concerns that the FFT process was, as some claimed, “a stick with which to beat General Practice”. Even though the Government had stated that it would not use FFT scores to rank practices, that fear of unfairness remained unassuaged, which disheartened hard-working professionals.

As a result, although all the practices in our evaluation had made the FFT available to patients, few practices felt committed to, or “owned”, the process. Even though practices had been assured that the test was intended for local quality improvement, not regulation (or criticism) by national bodies, they remained doubtful, particularly because of the monthly reporting. One GP, not realising that the scores were for meant for practice consumption, even asked their practice manager during a joint interview: “Do we have to open the box?”

Shift to reporting quality improvements
We recommend that monthly reporting of FFT scores should be stopped, and maybe replaced by a qualitative report on local quality improvements, perhaps submitted annually. That would help restore trust among practices that central bodies are not “spying” on them.

Particularly valuable aspects of the FFT are the chance for patients to comment briefly and quickly about their experiences, and the opportunity for practices to collect such feedback rapidly and easily. This comment facility could, for example, be kept within an FFT that had a less confusing core question, and patients could be encouraged to be more specific about particular services (e.g. phone consultations, clinics for chronic diseases, immunisation, etc.), making it easier to identify and address problems on aspects of care that they want to be improved.

The key issue in the long-term will be whether, and to what extent, patients’ views will contribute to making services better. Commitment to quality improvement was found to be uneven across practices. A revised FFT might play an useful role in addressing this problem, being easy to implement and thus a feasible option particularly for smaller practices with less capacity for data collection. But extra detailed guidance is needed on how to ensure that patient feedback leads to service improvement. It will be important to make clear that the Government’s priority is aligned with that of GPs in being focussed on securing higher quality primary care.

Tommaso Manacorda is a Research Fellow at the London School of Hygiene and Tropical Medicine. His report “Implementation and use of the Friends and Family as a tool for local service improvement in NHS general practice in England” is published by PIRU and co-authored by Dr Bob Erens, Professor Sir Nick Black and Professor Nicholas Mays.

This commentary summarizes an independent report commissioned and funded by the Policy Research Programme of the Department of Health for England, via its core support for the Policy Innovation Research Unit, with additional funding provided for data collection from the main sample of general practices. The views expressed are those of the authors and not necessarily those of the Department.