If an in-principle decision is made to proceed with a proposal, further work will usually involve more development of a preferred option, detailed due diligence and implementation planning.
This work will lead to a final decision about whether to proceed with the proposed change.
The next step is taking action, by implementing the agreed proposal. Undertaking due diligence as appropriate throughout the process will help with planning for a successful implementation.
It is also important to evaluate the changes to see whether they are achieving the desired result and what the system can learn from them for the future.
Te arotakenga Evaluation
Evaluating change helps you understand what worked well, and under what circumstances, and it should be an integral part of any structural or governance change. Because changes can be costly and disruptive, it is important to understand if they achieved their objectives, and to identify any lessons for the future.
To determine how valuable the change was, an evaluation may consider:
- what contribution the structural change or joint initiative made to target outcomes and unintended outcomes
- the value for money, cost-effectiveness or cost-benefit that has been realised from the structural change or joint initiative
- how, and how well the structural change or joint initiative has been implemented.
When to evaluate
The need for evaluation should be acknowledged in the early stages of the review, and evaluation planning should begin during the due diligence phase or, at the latest, soon after the final decision is made. The evaluation itself should begin while putting the change in place so you can gather baseline data and address emerging issues.
We have outlined the basic processes for evaluating structural or governance changes in government.
-
Establish roles and resources
- Set or confirm the intention to evaluate
- Engage stakeholders
- Establish decision-making processes
- Decide who will do the evaluation
- Determine resource requirements
For example, if you are evaluating a:
- structural change, ask:
- is there a mandate to evaluate?
- who are the stakeholders and how should they be involved?
- who is commissioning the evaluation?
- how will decisions and sign-off work?
- should the valuation be done internally, externally or with a hybrid approach?
- how much will it cost?
- cross-organisation initiative, ask:
- is there a mandate to evaluate?
- what roles and responsibilities will the different organisations have?
- how will decisions be made and coordinated across the organisations?
- who will do the evaluation? One or organisations or an independent party?
- how much will it cost and who will contribute?
-
Define what is evaluated and why
- Specify or confirm the purpose of the evaluation
- Describe the change being evaluated
- Retrieve or develop an intervention logic
- Identify possible unintended results
For example, if you are evaluating a:
- structural change, ask:
- how will the findings be used?
- what aspects of the structural change are being evaluated?
- do you have an intervention logic that shows inputs, activities, outputs and outcomes? Does it have a sound rationale?
- what unintended outcomes might result from a structural change?
- cross-organisation initiative, ask:
- how will the findings be used?
- does the initiative include multiple interventions? Which will be evaluated?
- do you have an intervention logic that shows inputs, activities, outputs and outcomes? Does it have a sound rationale?
- what unintended outcomes might result from the initiative?
-
Frame the boundaries of the evaluation
- Develop the evaluation’s key questions
- Develop criteria for success
For example, if you are evaluating a:
- structural change, ask:
- what questions will the evaluation answer?
- will it focus on implementation or effectiveness or both? Can you expect long-term outcomes yet?
- what achievements would demonstrate the success of the structural change?
- cross-organisation initiative, ask:
- what questions will the evaluation answer? Given the timing, should it focus on implementation, immediate outcomes, or long-term outcomes?
- what achievements would demonstrate the success of the initiative?
-
Describe what has happened
- Gather information on inputs, activities, outputs, outcomes and context from relevant documents, datasets and peoples
- Analyse information
For example, if you are evaluating a:
- structural change, ask:
- what quantitative and qualitative data can you collate from existing sources?
- what further data do you need to collect?
- how will you identify unanticipated outcomes?
- how will you mitigate the risk of data churn over time?
- cross-organisation initiative, ask:
- can shared measures across the organisations be set up early so that results can be aggregated across the whole initiative?
- what combination of quantitative and qualitative data will answer the evaluation questions?
-
Understand attribution of outcomes
- Use a comparison group if possible
- Ask if the observed outcomes are consistent with what you would expect if the change contributed to them
For example, if you are evaluating a:
- structural change, ask:
- how will you know if the structural change caused the outcomes? Is there evidence for attribution in the timing and nature of the outcomes?
- are alternative explanations a better fit?
- do staff and stakeholders think the change contributed to outcomes?
- cross-organisation initiative, ask:
- can you collect data on a comparison group so as to compare participant outcomes to non-participant outcomes?
- is there evidence for attribution in the timing and nature of the outcomes? Are alternative explanations a better fit?
- do staff and stakeholders think the initiative contributed to outcomes?
-
Synthesise findings
- Assess results against the criteria for success
- Assess value for money, cost-benefit or cost-effectiveness
- Describe what is working well and what can be improved
- Generalise lessons for the wider system
For example, if you are evaluating a:
- structural change, ask:
- what do the results tell us about the success of the change in terms of outcomes and/or implementation quality?
- did the change provide value for money?
- what worked well, and what improvements can be made?
- what lessons can the rest of government learn?
- cross-organisation initiative, ask:
- what do the results tell us about the success of the initiative? Did some interventions do better than others?
- did the initiative provide value for money?
- what has worked well and what improvements can be made?
- what lessons can the wider sector learn?
-
Report on and support the use of findings
- Emphasise positive actions for improvement
- Engage users to encourage take-up and use of the findings
- Publish the report
For example, if you are evaluating a:
- structural change, ask:
- how can your reports be tailored for the audience?
- are reports constructive, stating what’s worked and suggesting improvements?
- how can you socialise the findings and navigate conflicting views?
- when will you publish the reports?
- cross-organisation initiative, ask:
- how can your reports be tailored for the audiences?
- are reports constructive, stating what’s worked and suggesting improvements?
- how can you work with the participating organisations to get the findings used?
- when will you publish the reports?