How do we know what works? It seems everyone is saying that better evaluation is the answer.
Last November, Minister for International Development the Hon Pat Conroy MP announced Australia will have ‘a more rigorous, evidence-based approach so we can evaluate whether our aid is effective’. Meanwhile, Andrew Leigh MP has made no secrets about his desire for evaluation to improve government programs more broadly. And 100+ Pulse Check experts delivered their top recommendation for improving DFAT’s capability: ‘establish an independent evaluation mechanism’.
But we know that evaluation isn’t always this popular – it’s something that waxes and wanes. So here’s what the experts have to say on evaluation’s time in the sun and how to make the most of it for improved Australian development.
Development can be complex and difficult. Evaluation helps us find out what works and what needs improvement in Australia’s $4.5 billion spend on aid. Good evaluation gives voice to directly affected people about what they need and value. Together with its sister discipline, monitoring, evaluation is a central component of good management.
That’s why the enthusiasm for evaluation in Canberra is welcome. But let’s ensure we do it right. Here are four ideas.
1. It’s critical that senior DFAT leaders have the incentives to value and demand robust evidence about investment performance. Without this the rest is largely meaningless.
2. DFAT needs more monitoring and evaluation (M&E) specialists on staff – to explain and champion the approach. To ensure timely inputs to budget decisions, designs and investments. To ensure rigour in program M&E. And to work in situations where required security clearances rule out most external consultants.
3. M&E should be proportionate to an investment’s scale and purpose. All investments over $3M would benefit from an independent evaluation. But those which are significantly larger, higher risk or of extra strategic importance need more. This could mean an early review of: 1) a program’s design and contracting arrangements; 2) how well the M&E system is being implemented and; 3) whether data is being used in management and decision-making.
4. Resource meaningful learning across a portfolio of programs related by theme or geography – during their implementation. For example, in DFAT’s work in gender equality and women’s empowerment in the Pacific, or in areas of historic low performance.
And while we’re at it – let’s set up a knowledge management system within DFAT, so that we can access and make use of our growing body of information about good development, and how to evaluate it!
Jess is a multi-award-winning design and MEL specialist who’s led 40+ evaluations and designed well-known programs like MAMPU and Disaster READY. She’s the Founder and Director of Bluebird Consultants which independently validates DFAT’s annual Final Investment Monitoring Reports, has led After Action Reviews for DFAT’s Humanitarian Branch, and supports loads of other monitoring and evaluation work for the Office of the Pacific. At the Lab, we love Jess’ commitment to participatory processes, and the keen eye she takes to strengthening evaluation capacity both within DFAT and beyond.
Australian government appetite for evaluation comes in waves. In a recent trough, DFAT relaxed requirements for development programs to be evaluated and leaders could nominate a select few to be published. Demand for investment evaluations may be on the rise with more rigorous evaluation policies expected, but it doesn’t follow that all leaders will jump on board – some fear media or senate estimates scrutiny, others see evaluations as bringing a low return for scarce resources.
What can proponents of evaluation say to DFAT leaders to encourage them to embrace evaluation of their investments?
1. Findings are not personal – we’re learning
Development is complex and demonstrating results can be tough. You need evaluations to tell you what’s working and what’s not. They give substance to claims about effectiveness and explain the context (including to senate estimates or the media) when progress towards results isn’t as expected.
2. Contestability will improve performance
Incentivise your team to be effective investment managers, including planning for quality evaluations. You have control over the timing, terms of reference, selecting diverse team members, providing quality data based on a properly resourced monitoring, evaluation and learning system, and setting the context for program partners so they know what to expect. But don’t try to control the findings or recommendations. Stay at arm’s length, project an open mind, and invite contestability.
3. Demonstrate accountability through actioning responses
Digest and respond to evaluation findings and recommendations. You don’t need to agree with everything. Publish the evaluation and your management response and, for ongoing investments, hold your team and implementing partners accountable for actioning your response. Annual investment check-ins are a good time to ask your teams to explain their progress.
Building a performance culture needs leadership effort. Improved evaluation policy can’t do it alone.
With an impressive career history across DFAT, World Bank and UNDP, Pip now brings over 20 years of development insights to inform her work at Abt Associates. As Vice President of Program Quality and Strategy, Pip plays a crucial role in supporting the effective and impactful delivery of some of Australia’s largest aid investments. At the Lab, we love Pip’s willingness to share new ideas, and her drive to deliver strong program outcomes.
The increased attention on evaluation is welcome and overdue. But it’s important to remember that the real agenda sitting behind this shouldn’t just be a navel-gazing “let’s improve how we think about ourselves” moment, but a more transformative one where we’re asking, “if Australia’s serious about being a high impact development partner, what will it take to drive lasting reform?”
The risk of a strategic misstep lies in any assumption that doing ‘more’ or ‘better’ evaluation alone will lead to improved performance, because ultimately our development program is managed by busy humans making complex decisions. Perhaps they’re the key to transforming our impact? At a recent Situation Room, we heard the views of 16 of Australia’s top evaluators, and this notion of starting at the human level to drive reform had a lot of traction. Here are two of the key themes that resonated with me the most:
Capacity development across the board will bolster supply of good quality evaluation… It was important to a number of the experts that DFAT embeds evaluation skills into performance and promotion incentives at all levels, while at the same time just get more resources in to make sure that already time-constrained individuals can have the space to do it well. And since external contractors do a huge amount of evaluation work, perhaps it makes sense to send some APS6s out to experience it with them.
… but accountability in senior leadership is the best way to actually transform performance. More than anything, attendees wanted to see changes to the way that Heads of Mission and Secretaries are incentivised to pursue better development outcomes. While some advocated for a return of the Office of Development Effectivess or the appointment of a Chief Evaluator, others felt it was time to look to external best practice by increasing DFAT’s healthy competition with other central agencies that might be ahead in using evaluation outcomes to drive impact.
Mira is an international development practitioner and keen strategic thinker with a background in business development, and policy research and analysis. Prior to joining the Lab, Mira worked in Tetra Tech’s Future Economies Practice to research, design and deliver programs that pursued sustainable economic outcomes in the Indo-Pacific. At the Lab, we love Mira’s ability to connect the strategic dots, and commitment to deeply understanding the perspectives of everyone in the room.