Current projects
Integrated Care Pioneers evaluation
Integrated Care Pioneers evaluation

Better coordination within the health care sector, and between the NHS and local government, in the commissioning and provision of services, has long been a policy aspiration in England. It is frequently stated that integrated care should lead to more person-centred, better coordinated care, deliver more efficient services that improve outcomes for individuals, and provide better value for money. The national programme of Integrated Care and Support Pioneers aspires to encourage the ‘most ambitious and visionary’ local areas to become integration Pioneers which will drive change ‘at scale and pace, from which the rest of the country can benefit’.

PIRU has been commissioned by the Department of Health to carry out an evaluation of the Pioneers for up to 5 years (starting in July 2015). This follows two projects PIRU carried out from autumn 2013 through June 2015.

  • The first project was to convene an expert group to provide advice to the Pioneers on a set of candidate indicators of integration performance based on routinely collected data that they can use individually and collectively to monitor their progress.
  • The second project was to carry out a largely qualitative early evaluation focusing on progress of the 14 first wave Pioneers in the first 15-18 months in relation to their initial integration objectives.

Longer-term evaluation

The overall aim of the longer-term evaluation is to assess the extent to which the Pioneers are successful in providing ‘person-centred coordinated care’, including improved outcomes and quality of care, in a cost-effective way. The evaluation is expected to help build the evidence on what works best in delivering quality integrated care in different contexts. All 14 first wave and 11 second wave Pioneers are included. The evaluation consists of three interdependent work packages (WPs).

WP1 aims to:

  • Understand how Pioneers are pursuing their aims, and their experiences of making integration-related service changes. This involves surveys and interviews with a panel of managerial and delivery staff from all 25 Pioneers and other local stakeholders, such as patients/service users. The surveys take place on a regular basis throughout the life of the evaluation.
  • Develop a typology of Pioneers and their schemes/initiatives.
  • Use the typology of Pioneers and their schemes/initiatives to inform the selection of interventions and schemes for the WP2 cost-effectiveness evaluations.
  • Compare differences, at Pioneer level, between Pioneers and matched non-Pioneer areas in England in terms of changes in key indicators of the extent of care co-ordination and its consequences. This will be done this using relevant, routine, national and local data sources.
  • Identify similarities and differences between the approaches to system-level (macro) integration adopted by the Pioneers and those in the NHS’s ‘Five Year Forward View’.

WP2 aims to:

  • Assess the cost-effectiveness of systemically important integration initiatives undertaken by Pioneers, using a range of designs, such as quasi-experimental, depending on circumstances.
  • Understand how the initiatives’ impacts are affected by context, by undertaking qualitative interviews and focus groups with key stakeholders in the Pioneers participating in the cost-effectiveness analyses.
  • The first such economic evaluation is looking at community-based integrated health and social care multi-disciplinary teams within three Pioneers.

WP3 aims to:

  • Synthesise findings from WP1 and WP2, by assessing the extent to which integration initiatives’ achievements (WP2) can be related to the higher level approach taken by Pioneers (WP1).
  • Derive and spread evidence-based insights from WP1 and WP2 to improve integrated care, as well as feeding this information back into the research process to encourage reflexive learning. This includes regular interactive workshops involving the research team, Pioneer and patient/user representatives, and other key experts and stakeholders.

The evaluation team involves a collaboration between PIRU researchers based at LSHTM (Nicholas Mays, Mary Alison Durand, Ties Hoomans, Nick Douglas, Tommaso Manacorda, Sandra Mounier-Jack, Bob Erens), along with Gerald Wistow from LSE, Martin Bardsley and Eilis Keeble from Nuffield Trust, and Judith Smith and Robin Miller from the University of Brimngham.

A short summary of the longer-term evaluation can be accessed here>>

Further detail about the Pioneers can be found on the NHS England website. Read more>>


The first project on identifying candidate performance indicators was carried out to a tight timetable between November 2013 and February 2014. Over 30 candidate indicators were identified, classified by six key dimensions:

  • Community well-being and population health
  • Organisational processes and systems
  • Personal outcomes
  • Resource use/balance of care
  • Service proxies for outcomes
  • User/carer experience.

The report was published in April 2014 and can be accessed here >>

The early evaluation was carried out between January 2014 and June 2015. The aims of this qualitative evaluation were to:

  • Describe and understand the scope, objectives, priorities, plans and management of the 14 selected Pioneers
  • Describe the mechanisms and ‘intervention logics’ adopted by the Pioneers to deliver those plans and priorities
  • Identify the financial incentives, contractual forms and budgetary innovations put in place to implement the Pioneers’ plans
  • Describe how the Pioneers’ Better Care Fund (BCF) plans begin to be implemented
  • Assess the extent to which Pioneers are able to address previously identified barriers to the integration of care
  • Examine progress of the Pioneers in the first 12 months in relation to their first year integration objectives.

A report with interim findings from the early evaluation was completed in February 2015.

The final report of the early evaluation was published in April 2016. Key findings of the final report include:

  • By the summer of 2015, most of the 14 first wave Pioneers were still in the relatively early stages of implementing their plans, and it is difficult to draw firm conclusions about their progress; for example, it is too soon to reach definitive conclusions about whether they might provide role models for other parts of the country to learn from.
  • Their early focus on user experience and a shared definition of good integrated care has been helpful in developing a vision for each Pioneer. However, this appears to have been much less useful in supporting the implementation of plans for specific changes to services and professional behaviour.
  • There is considerable diversity in progress between the 14 Pioneer sites. While many Pioneers have largely agreed locally how services should be re-designed, for the majority of sites, much remains to be done to put this in place. There is limited evidence so far of change, despite the expectation that Pioneers would be able to get into delivery mode quickly.
  • One of the ostensible advantages of becoming a Pioneer was not only sharing learning with other sites, but also obtaining access to key decision-makers, and receiving advice and support from national and international experts. Access to external advice and support has continued to be perceived as patchy (at best) by many sites.
  • A number of barriers to greater integration are being gradually resolved at local level, but a number require changes led from the centre that Pioneers cannot initiate, in particular, in relation to workforce and information governance. Some participants in the Pioneers were critical of the extent to which national partners had thus far helped them address the obstacles that related to national policies and systems, such as, for example, data sharing, payment systems, procurement, provider viability and the foundation trust ‘pipeline’.
  • The facilitators of integrated working tended to be related to factors such as leadership, vision, trust and shared values that are largely developed locally, while the barriers were more likely to be features of formal organisational structures and systems only amenable to resolution by national agencies.
  • From the perspective of participants, the environment for whole systems transformation was not becoming easier. There was little evidence that the balance between facilitators and barriers had shifted in favour of the former. If anything, the balance appeared to be shifting in the contrary direction, particularly as the financial situation was deteriorating.
  • This was resulting in an ‘integration paradox’. The context of growing need and declining budgets provided an even stronger imperative for more effective integration. However, at the same time, this context made it more difficult to make progress.
  • On the upside, the shared definition of integrated care as person-centred, coordinated care was helping to frame collective understandings of both the starting point and goal of Pioneer activity in difficult times. However, the priorities associated with the BCF and other policies driven by the financial difficulties of the system provided a competing set of pressures.
  • By the end of the fieldwork for the early evaluation, there were signs of a narrowing of purpose and a greater focus on short-term, financially driven goals, most notably to contain costs through action at the hospital-community interface. The majority of Pioneers appeared to be converging towards a set of specific interventions for older people with substantial needs, such as care navigators, care planning, multi-disciplinary teams, etc.This convergence represents a narrowing of some of the broader ambitions in relation to early intervention envisaged by the Integrated Care Collaorative for the Pioneers.
  • On one scenario, the Pioneers could be seen as laying the foundations to make rapid progress and start sharing learning. On another, he barriers and difficulties they have experienced to date could prevent just such progress. There is little evidence so far to support the first scenario, with the evidence suggesting that integrated care is not getting easier.

The final report for the early evaluation can be accessed here>>

The interim report of the early evaluation can be accessed here>>

An article describing early findings from the evaluation was published in 2017 in the Journal of Integrated Care and can be accessed here>>

Results from the early evaluation were presented at a number of conferences and seminars in 2016:

- Early evaluation of the Integrated Care and Support Pioneers: overview of findings from the draft final report. Pioneer Longer Term Evaluation, Workshop 1: transforming the evaluation into actionable learning. Birmingham: Health Services Management Centre, University of Birmingham, 1 March 2016

- Early evaluation of the Integrated Care and Support Pioneers Programme. Health Policy and Politics Network 2016 Spring Event. Manchester: University of Manchester, Institute of Population Health, 5 May 2016

- Early evaluation of England’s integrated care Pioneers: challenges of implementation and evaluation. 16th International Conference on Integrated Care. Barcelona: Palau de Congressos, 23-25 May 2016

- Findings from the early evaluation and early findings from the longer term evaluation of the integrated care and support Pioneer programme. The King’s Fund Annual Integrated Care Summit 2016. London: King’s Fund, 11 October 2016. Slides from this presentation can be accessed here>>

- Early findings from the longer term evaluation of the integrated care and support Pioneer programme relevant to the New Models of Care (Vanguard) programme. Nuffield Trust-Policy Innovation Research Unit breakfast seminar. London Mathematical Society, 8 November 2016

The report of the results of the first key informant survey can be accessed here>>

The final report of the early evaluation was published in April 2016 and can be accessed here >>

A blog describing the final report is available on the LSE website and can be accessed here>>

An interim report of the early evaluation was published in March 2015 and can be accessed here>>

The final report on candidate indicators and data sources for measuring the performance of the Pioneers was published in April 2014 and can be accessed here >>

Bob Erens presented a methodologal paper on using survey research in evaluations, based on the experience of using surveys in the long-term evaluation, at the European Survey Research Association conference in Lisbon on 18 July 2017.

Mary Alison Durand was invited to present early findings of the longer-term evaluation and an overview of the first economic evaluation at the Pioneer Assembly, hosted by NHS England, on 12 June 2017 in London.

Mary Alison Durand, Ties Hoomans and Robin Miller presented early findings of the long-term evaluation, and outlined plans for the first economic evaluation, at a dedicated workshop on PIRU's long-term Pioneer evaluation at the 17th International Conference on Integrated Care in Dublin on 9th May 2017. The slides from this presentaton can be accessed here >>

Bert Vrijhoef, Professor of Health Systems and Policy at the University of Singapore, gave a talk at LSHTM in September 2016 on 'Evaluating Integration: Learning from International Experience". The slides from this presentationcan be accessed here >>

Findings from the early evaluation were presented by Nick Mays at The Kings Fund Integrated Care Summit on 11th October 2016. Details of the event and slides from Nick's presentation can be found here >>

Nick Mays presented the findings of the early evaluation at the Health Policy and Politics Network Spring event on 5th May 2016 in Manchester. The slides from this presentation can be accessed here >>

Nick also presented the early evaluation findings at the 16th International Conference on Integrated Care in Barcelona on 23-25 May 2016.

The final report of the early evaluation was published in April 2016 and can be accessed here >>

The first interactive workshop with Pioneer sites took place in Birmingham on 1st March 2016.
Notes from the workshop can be accessed here >>
(Please note: You must download the pdf file to access the links within the document.)

The second interactive workshop with Pioneer sites was held on Thursday 15th September 2016. The workshop included a presentation by Prof Bert Vrijhoef on 'Evaluating integration: learning from international experience', which can be accessed here >>

The third interactive workshop with Pioneer sites was held on 29th March 2017 in Birmingham. Notes from the workshop can be accessed here >>

Part of our evaluation involves looking for high level changes over time across the whole system of each Pioneer. We are interested to see how the many different activities within the Pioneer sites translate into changes in commonly used measures relating to service provision and well-being amongst the local population. In order to do this we are building sets of indicators drawing on routinely available data for successive years. For each Pioneer site we will track change in key indicators and enable comparisons to other Pioneer sites and other parts of the country. Our key questions are around which indicators will show a differential trajectory in Pioneer sites.

We have looked at a subset of possible indicators and collated data on change over a few years. The indicators are organised into spreadsheets that allow Pioneers to look at change for their own area, select local, regional or national comparators and present summary changes graphically. We recognise that there are limitations to these high level indicators. Our plan is to add new indicators of interest if they become available and improve the ways we assess the rates of change on specific indicators. We would welcome feedback on the indicators chosen and suggestions for improvements.

We would be interested in knowing how you use the indicator data locally. Please note, however, that the indicators are not designed for the purpose of creating league tables or as a performance management tool, and should not be used as such. The evaluation team will not be using the indicators for such purposes, and furthermore, will not be using them in isolation in undertaking our longer-term evaluation of the Pioneer programme.


The evaluation team holds regular workshops with Pioneer and patient/service user representatives, and other key experts and stakeholders. The aim of the workshops is to derive and spread evidence-based insights from the evaluation in order to improve integrated care, as well as to feed this information back into the research process to encourage reflexive learning.

The first interactive workshop between the Pioneers and the evaluation team was held on 1st March 2016 in Birmingham. The programme for this meeting can be accessed here >>

Notes of the meeting can be accessed here >>
(Please note: You must download the pdf file to access the links within the document.)

The second interactive workshop was held on 15th September 2016 in London. Notes of the second workshop can be accessed here >>

Presentations from this workshop can also be accessed:

Bert Vrijhoef "Evaluating integrated care: learning from international experience" can be accessed here >>

Mary Alison Durand "Longer-term evaluation of the Integrated Care Pioneers: progress update" can be accessed here >>

Deborah Neal "Evaluation: South Somerset experience" can be accessed here >>

Martin Marshall "Negotiating evidence: the researcher-in-residence model" can be accessed here >>

Esther de Weger "Kent evaluation framework" can be accessed here >>

The third interactive workshop was held on 29th March 2017 in Birmingham. Notes and presentations from the third workshop can be accessed here >>

Video blogs from three of the speakers at the third workshop
Dr Richard Lewis (EY) reflects on the national evaluation of the Integrated Care Pilots and lessons for integrated care in 2017.

Leo Lewis (International Foundation for Integrated Care) highlights innovative models of information sharing in Europe.

Bill Jenks (Tower Hamlets CCGs) discusses the progress that has been made in East London on shared care records.



Members area for Pioneer high level indicators

If you have forgotten the password please contact Paula Fry at Paula.Fry@lshtm.ac.uk