Monitoring, Evaluation, Accountability & Learning

ANALYZE SITUATION & ASSESS NEEDS

A situational analysis and needs assessment are carried out to determine whether an integrated mental health program is appropriate and feasible and, if so, to inform its planning and design through systematically collecting and analyzing data. This includes information about mental health needs, capacities, resources and national efforts for mental health system strengthening, and overall sustainability of planned services. > Access Toolkit Section: Step 1. Assess & Plan for MH Integration

DESIGN PROGRAM

• Develop/Adapt Program Objectives. Findings from a situational analysis and/or needs assessment are used to design and tailor the program to the local context. This includes being consistent with government guidelines and global best practices, building on any existing or emerging mental health components of a system, and meeting identified needs while also considering existing resources.
• Develop/Adapt Logical Framework & Indicators. This includes the development of a logical framework, and associated activities for monitoring and evaluation (M&E) as well as, program specific quantitative and qualitative indicators. Indicators track key elements of program performance as well as results. Some donors also require specific M&E templates and additional elements such as outlining a Theory of Change (TOC) that helps you check the validity of your assumptions by setting end objectives and working backwards to ensure that activities, outputs and outcomes will link to program objectives. > → See MHIN TOC practical guide & example ToC map for integrating mental health into primary care in India, Nepal, Ethiopia, Uganda and South Africa – the PRIME study. A free 20-minute course by Save the Children on Theory of Change is available on DisasterReady.

COLLECT BASELINE

This is the measurement of the initial conditions as determined by project indicators, that occurs before the start of a program. Baselines can help monitor changes and contribute to program evaluation by making comparisons with later follow up data. Relevant data that is still current and was acquired in the situational analysis and needs assessment phase can also be used as part of the baseline.

When no formal baseline is completed, or the baseline data is incomplete, the following strategies may be employed:

• Consult and utilize existing secondary data (e.g. WHO AIMS, AIMS country reports, WHO proMIND, MH Atlas, MHIN, and mhpss.net) from other NGOs, government or community organizations (e.g. survey data, health registers or project activity records).
• Utilize recall of staff and/or affected populations (e.g. asking staff about past training, activities, practices for addressing mental health problems).

IMPLEMENT & MONITOR

This includes the systematic monitoring of mental health programs over time. It involves the ongoing collection and review of data to provide program managers and other staff with indications of progress against program plans and towards program objectives. This information should be used to improve program implementation through adjustments to programs activities, timelines and stakeholder engagement.

REVIEW AND EVALUATE

This helps determine to what extent programs are meeting or have met their goals. Evaluation is essential to planning strategies, improving programs, demonstrating results, and justifying resource allocations.

• A formative evaluation with a focus on improvement is carried out during the life of the program.
• An end of project evaluation takes place towards the end of the program and is used to reveal its overall impact.

ACCOUNTABILITY & LEARNING

• Learning includes systematic documentation and use of lessons learned, recommendations, and observations, that emerge from monitoring, evaluation and accountability mechanisms to improve program design, and to publish and communicate results internally and externally. Reporting, reflection and learning should occur throughout the whole program cycle.
• Accountability to stakeholders, service users and wider communities involves ongoing participation, information sharing, and development of feedback mechanisms, to guide program design and implementation in line with local priorities.
• Community Participation and Stakeholder Engagement in MEAL is essential to identify and address challenges and gaps as they arise, in ways that are appropriate for the community and context. This can lead to increased relevance of programming, transparency, accountability, sustainability and ownership. Communities and stakeholders can be engaged and participate in:
  • Needs assessments and situational analyses.
  • Resource mobilization.
  • Discussing the program design and planned activities. outcomes and impact.
  • Data collection for monitoring and for evaluation.
  • Providing feedback and suggestions in a safe and dignified manner (e.g. through feedback and complaint mechanisms).
  • Interpretation of MEAL data to make recommendations.

KEY CONSIDERATIONS FOR DEVELOPING INDICATORS:

• Carefully consider the time and resources required to collect specific indicators and ensure that the information collected is practical and useful for documenting and informing programming.
• Ensure that indicators are prioritized to inform programming and demonstrate outcomes; avoid overburdening the community with data collection that is not useful or effective.
• Support the standardization of core indicators for mental health integration programs among agencies and partners.
• Whenever possible, track indicators using >data collection systems that already exist, instead of external systems that will require special efforts to maintain.
• Indicators should be SMART: Specific, Measurable / Quantifiable, Attainable, Relevant, Time-bound; they should be well defined, so that they are understood, and data is collected and interpreted in a consistent manner.

>>> Examples of indicators across levels of MH integration are available in the SAMPLE IMC Mental Health Integration Results Framework.