By Matthew Wilson, A360 Project Director, PSI
It’s my final day with A360.
It’s almost four years since I first started supporting A360, before taking up the leadership role. I was asked to join the team in Nigeria to help troubleshoot. The project was in its optimization phase as it transitioned from piloting to implementation at scale, and the results had not yet met expectations. The team had created the equivalent of a six-arm study to test different variations of the interventions to see which was the most cost-efficient without losing fidelity to the intended user journey.
This experience was like a microcosm of A360: an incredibly dedicated team, working under intense pressure and scrutiny, moving at breakneck speed, and continuously gathering insights, learning and adapting.
This hasn’t changed in the four years I’ve led the project, though I wish the pressure, scrutiny and pace had eased. But a lot else has changed. We’ve made several important strategic pivots, not least the shift to supporting government-led scaling of A360’s approaches rather than A360-led delivery. The reinvigorated localization agenda has also called me to question whether I’m the right person to be leading a flagship AYSRH project.
And so it’s time to hand over to someone from the African region with deep health systems strengthening experience, who can carry the project forward.
I want to take this opportunity to share a few reflections.
-
Evolving
Over the last decade there have been a multitude of adolescent sexual and reproductive health (ASRH) projects that have focused on trying to determine or prove what works. Many have employed human centered design (HCD) to develop solutions to hard problems like how to position contraception in a way that resonates with adolescents and how to build empathy for adolescents among healthcare providers. The first phase of A360 was one such project.
Fast forward to today and we find a growing consensus on what works, and an array of academic papers and technical publications documenting it.[1]
Each donor/implementer/movement/research organization may package it differently, but they share a lot of the same fundamentals. Fundamentals captured in resources like the High Impact Practices Enhancement Brief on adolescent responsive contraception services.
Given this progress – and faced with scarce resources – one wonders whether it’s time for the sector to embrace ‘good enough’ and increase the emphasis on solving the hardest problem of all: how to scale and sustain ASRH interventions through mixed health systems. Health financing is going to be key.
-
Persevering
“The answer is 17 years, what is the question?” is the brilliantly enticing title of a journal article that examines how long it takes for research to be translated into changes in practice. This reality is clearly juxtaposed with the conventional donor-funded projects which typically last 3-5 years and demand results fast. The time lag illustrates the complexity of health system strengthening.
There are two ways of looking at this. One is to adjust our expectations. The other is to invest more in systems thinking, better understand the evidence around how change happens, and find new levers of change to reduce that lag. We need to do both.
‘To see far is one thing, going there is another,” – Constantin Brancusi, 1876–1957
-
Collaborating
One of my go-to resources is Jeremy Shiffman’s research on what it took to generate political priority for maternal mortality reduction. It identified nine factors, including the need for cohesion across development partners. This is self-evident yet the sector is still plagued by fragmentation of effort. Resolving this, sits first and foremost with the donors and multilaterals like the World Health Organization who have the power to be a convening force and build portfolios that are coherent and complementary. We need to dispel the notion that competition is a force for good and foster deeper collaboration.
As the world wrestles with a global economic crisis, collaboration will become both more important and more challenging. Domestic financing is faltering, many bilaterals are narrowing their focus on nationalist agendas, bilateral aid is increasingly being provided as loans and equity investments rather than grants., and fragility is growing more acute. In many LICs, debt servicing now represents 20-30% of the national budget and defaults are increasing. Less will need to achieve more. Duplication and fragmentation of effort will be inexcusable.
-
Learning
A360 has been labelled a ‘flagship’ project and understandably it has attracted a lot of attention and scrutiny. Our external communications during the first two years did us no favors, prematurely promoting the project’s interventions.
Since then there have been several evaluations of the project. By organizations that have appraised the project without any consultation, by technical experts the donors have consulted, and by organizations we’ve brought in to benefit from their expertise. And of course, by our external evaluators Itad, Avenir Health and the London School of Hygiene and Tropical Medicine (LSHTM).
A360’s 80-page external evaluation summative report will be released this week, alongside our 20-page management response. As these demonstrate, the external evaluation generated a wealth of learning that goes well beyond the effectiveness of the interventions. Learning about the relevance of mCPR as an outcome metric for ASRH interventions; the suitability of using impact evaluations for pilot projects; the implications of designing an outcome evaluation protocol before the interventions have been designed; the challenge of evaluating projects that employ adaptive implementation and continuously evolve; the complexity of reconciling process and outcome evaluations; conducting data collection during a pandemic; and much more. Learning for A360, its evaluators, its donors, and the sector.
“We believe that the evaluation does its best to summarize a complex project and contains important learnings for the project’s stakeholders as well as other donors, governments, advocates, and implementers who are interested in improving adolescent health outcomes….Overall, we support PSI’s response to the evaluation findings; PSI’s efforts to both contextualize and defend its positions are reasonable, and we recommend the evaluation team take note of PSI’s well-argued points and conclusions around shortcomings of the evaluation,” – A360’s Global Advisory Panel, July 2022
There’s often a desire to distill the complexity and nuance of an evaluation like this into a binary conclusion – was the intervention a success or a failure. I encourage you to resist any such urge. It would do a disservice to the learning on offer. And, to use the title of a book by the doctor, academic, writer, and broadcaster Ben Goldacre: “I Think You’ll Find It’s a Bit More Complicated Than That!”
We’ve already started sharing this learning. We were a panelist in an HCD Exchange webinar titled ‘Measuring a Moving Target’ which explored some of these complexities, we’ll be presenting at ICFP in November, and we’ll be hosting a skills building workshop on evaluating HCD programs with the HCD Exchange, Pathfinder, and Itad during the SBCC conference in December. A journal article sharing key learnings from the first phase of A360 has been published, and another providing a commentary on the relevance of mCPR as an outcome metric for adolescent populations is forthcoming. And of course the team will be at ICFP in November.
But most importantly, we’ve already started applying this learning. Indeed much of this learning emerged long ago, with the help of the process evaluation, our adaptive implementation practices, and 18 months of reviewing outcome evaluation data. It has informed course corrections during the first phase of A360 and the design of the second phase of A360. A second phase that will also generate a wealth of learning, this time much of it relating to integrated and multi-sectoral programming and health system strengthening.
* * * * * * * * * * * * * *
I leave a team that is still incredibly dedicated, working under intense pressure and scrutiny, moving at breakneck speed, and continuously gathering insights, learning and adapting. A team I have been immensely privileged to lead and fortunate to learn from.
I encourage you to collaborate with them so together we can advance outcomes for adolescent girls.
[1] A360 was privileged to be part of the group that authored the Journal of Adolescent Health’s seminal December 2019 supplement on the eve of ICPD 2020, which analyzed progress and recommended priorities for the next decade.