“We bring a lot of content knowledge, you bring technical expertise. If we work together, we should be able to get a better product than if we sit in our different camps and play it like tennis, bouncing things back and forth.”
Constraints, Causation and Commissioning
Eleanor Williams · Managing Director, Australian Centre for Evaluation, Department of Treasury
About this episode
In this debut episode of It Depends, Matt Healey and Tenille Moselen are joined by Eleanor Williams, Managing Director of the Australian Centre for Evaluation, for a wide-ranging conversation on constraints, commissioning, and causation in public policy evaluation. Together, they explore what makes evidence fit for purpose, how to build meaningful relationships between evaluators and commissioners, and the everyday ethical dilemmas evaluators face. This episode is essential listening for anyone navigating the tension between rigour, relevance, and real-world constraints.
Key takeaways
Build relational, not transactional, commissioning relationships.
Trusted relationships between evaluators and commissioners allow for more open conversations that lead to better evaluation design and more useful findings.
In-house evaluation capability adds real value.
Even when working with external evaluators, having internal expertise allows government to be a better commissioner and have more frank, vulnerable conversations about evaluation needs.
Quasi-experimental designs offer practical opportunities.
Methods like stepped wedge designs and waitlist controls can work within real-world constraints such as phased rollouts or service backlogs.
Match methods to questions.
Use frameworks like the UK Government's Magenta Book to identify which evaluation methods best answer specific types of questions, particularly causal questions about impact.
Navigate "everyday ethics" thoughtfully.
Evaluators regularly face ethical challenges beyond formal ethics processes, such as requests to reframe findings. The key is distinguishing between substantive changes and reframing language to increase uptake.
Topics covered
- Evidence-informed policy
- Evaluation ethics
- Quasi-experimental design
- Relational commissioning
- Public sector evaluation
- Impact evaluation