The summative report for the external evaluation of A360’s first investment phase was published on 8 November 2022, alongside A360’s management response. This summative report and management response reflect a year and half of deep exploration, dialogue, and reflection. We want to acknowledge the tremendous work that has been done to get to this point. This process has brought A360 and the evaluators on an oftentimes difficult but also positive learning journey. We know there is a tendency to look at the result of these evaluations as a binary conclusion – were the project’s interventions a success or failure? This is something which A360 has challenged, instead working to dig into the nuance and complexity of the evaluation and its implications for our work moving forward and the sexual and reproductive health (SRH) sector at large. We have expressed concerns about the limitations of the evaluation design and whether it was ‘fit for purpose’, while acknowledging that there is an inherent difficulty in evaluating programs that utilize human-centered design (HCD) and take an adaptive approach. We believe that these shortcomings should be considered as a driver of some of the inconclusive evaluation results, explained in further detail in our management response. However, with humility, we acknowledge that there are many lessons to be drawn from this evaluation experience.
These lessons reflect the areas in which A360’s first investment fell short, many of which we have acknowledged in a recently published article, and areas where the broader sector can continue to improve.
Within A360’s management response, we have detailed a few of these lessons. These include:
- Designing a program which effectively supports adolescent girls not just to take up contraception once but to have the agency to use it, when and how they choose, is complex. It is even more difficult to do so without a good understanding of girls’ unique desires and motivations to use contraception. The evaluation pointed out the gaps in our knowledge, which we have committed to address in this next project phase. This is a pre-requisite for us being able to strengthen our interventions to reduce discontinuation while still in need.
- The use of modern contraceptive prevalence rates (mCPR) as a primary outcome metric presents clear challenges when fielded with populations of adolescent girls. We know there is work being done to explore alternative metrics around contraceptive agency and fertility-aligned with contraceptive use, but to date there is no consensus on a more reliable metric. In our next project phase, we are seeking to contribute to testing of these metrics to bring the sector closer to a metric which is more relevant and responsive to adolescents’ SRH needs and experiences.
- Application of traditional evaluation methods to a program which is so fluid and iterative is tremendously difficult. The global community of practice needs alignment on better methods for evaluating programs like A360 – including not just program outcomes but also evaluating a program’s ability to adapt and learn. We have committed to moving this conversation forward to generate better evaluation methodologies.
We hope this investment in the external evaluation has yielded a useful contribution to the adolescent sexual and reproductive health (ASRH) sector and illuminated some of the gaps we continue to face collectively. A360 commits to continued proactive engagement in disseminating these insights and is keen to hear from other partners who feel they can learn and grow with us.
We encourage you to read our management response alongside the itad summative report.