Cookies on Knowhow Nonprofit

We use cookies in order for parts of Knowhow Nonprofit to work properly, and also to collect information about how you use the site. We use this information to improve the site and tailor our services to you. For more, see our page on privacy and data protection.

OK

Skip to content. | Skip to navigation

Community-made content which you can improve Case study from our community

How to develop a monitoring and evaluation framework

Developing a monitoring and evaluation framework helps clarify which pieces of information to collect to evidence your story of change.

It is good practice to include people who will be collecting the data when you develop your framework. You could also involve beneficiaries, volunteers, trustees, partner organisations or funders.

Ideally, write your framework before your project starts so you can make sure you are collecting appropriate data from the beginning.

Examples provided in this How To are based on a hypothetical monitoring and evaluation framework for a project looking to improve access to employment for people leaving prison.

Things you'll need

  • planning tool
1

Use a planning tool

Be clear about what you are trying to achieve. Most organisations are driven by a clear sense of shared purpose and mission, but this is not always captured on paper. This can lead to problems when you’re deciding what data to collect for your evaluation.

You can use a Planning Triangle or a theory of change to get started.

2

Describe your outputs

Outputs are the goods, products and services that you deliver. If you have developed a Planning Triangle, you can generate a list of outputs by examining your objectives. Your objectives describe a planned area of work – your outputs are all the goods, products and services that fall within these areas.

Example

Outputs include one-to-one support sessions and maths and literacy skills courses.

3

Set output indicators

Output indicators are specific pieces of information that you collect to keep track and report on the work you have delivered. Output indicators give you information about:

  • what outputs you delivered
  • who you delivered them to
  • whether the people or organisations you delivered them to were satisfied and thought your work was good quality.

There will be some outputs that you are more interested in gathering information on than others. Set realistic priorities for measurement and try to limit the number of output indicators you set.

Example

Output (the good, service or product being delivered)

Indicator (what will we measure?)

One-to-one support sessions
  • Number of sessions delivered
  • Location of sessions (which prison)
  • Number of people supported
  • Profile of people supported (age, gender, ethnicity, sexuality, disability, previous work history)
  • Level of user satisfaction

Many funders and commissioners ask for specific profile information about the people you provide your services to. This may include age, gender, sexual orientation, ethnicity or religious belief. More importantly, you will need this information to understand whether or not your service is accessible to everyone you intended to reach.

4

Describe your outcomes

Outcomes are the changes that come about as a result of your work.

If you have developed a Planning Triangle, you can generate a list of outcomes by breaking down your specific aims. Examine your specific aims, which describe the broad areas of change you plan to make, and write down all the individual changes related to these areas. 

Make sure that you:

  • define your outcomes clearly. Avoid woolly outcomes such as ‘service users feel more empowered’. What does this empowerment mean to users? What changes for them as a result? The easier your outcomes are to understand, the easier they will be to measure.
  • have described single changes. Composite outcomes containing more than one idea are difficult to measure.
  • have not articulated big, complex changes. For example, instead of an outcome of ‘improved wellbeing’, think about whether you mean improved physical, mental or emotional wellbeing.

Outcomes should describe the changes that you want to create, not the work that you are delivering. Be cautious of writing outcomes that are about accessing your services. If you are running skills workshops in prisons, for example, there is no real value in measuring the outcome ‘increased access to skills workshops'. This doesn’t tell you anything about what changed for people as a result of your work. There can be an exception to this if your users are very hard to engage, and getting them to access a service signifies a real change for them. This might be the case with street homeless people, for example, who have never accessed a day centre and feel uncomfortable indoors.

It is good practice to include possible negative outcomes in your framework. This will help you to check that your work is not having any undesirable effects. For example, researchers evaluating a project that challenges the behaviour of male perpetrators of domestic violence were aware that doing so could potentially increase the risk of abuse. So they contacted the men’s partners to collect information on whether they had experienced increased physical violence, increased emotional abuse or increased controlling behaviour.

Consider which outcomes are priorities for measurement – it is unlikely that you will be able to spend time measuring everything that changes. Priority outcomes may:

  • be pivotal to your work
  • represent the ‘end result’ of a chain of smaller changes
  • unlock other outcomes
  • be helpful for attracting funding or reporting to existing funders.

Example

Outcomes include: improved maths skills, improved literacy skills, increased job search skills, increased confidence in interviews.

5

Set outcome indicators

Outcome indicators are specific, measurable pieces of information that you can collect to keep track of the difference that your work is making. They tell you whether or not you are achieving your outcomes and how much change has occurred.

Setting outcome indicators is very useful for deciding what to measure and what not to. Writing a clear list of indicators will make developing data collection tools (such as questionnaires) much easier.

Review your list of priority outcomes, and think carefully about what information you need to collect as evidence of change.

Your outcome indicators should:

  • be expressed in neutral language. They might be phrased as ‘level of’, ‘number of’, ‘type of’ or ‘how often’. Unlike outcomes, outcome indicators should not use a word that indicates change, such as ‘increased’ or ‘improved’. This is because you need to collect the data before you know whether or not the intended increase or improvement has happened.
  • be tailored to your context. There are online indicator banks, but they are only helpful if the indicators are suitable for the type of work you are doing, people you are supporting and changes you want to create. Think carefully about your outcomes. What are the signs that you have been successful? How can you tell when you have made progress? The answers to these questions will generally make strong indicators.

It is good practice to set at least two indicators to measure each outcome. When putting your indicators together you might:

  • combine numbers (quantitative data) and descriptions or narrative (qualitative data) to give you a fuller picture of what has changed.
  • combine different perspectives. In our example project, to assess whether or not ex-offenders feel more confident in interviews, we could ask them about how anxious they felt about interviews. Since the project offers mock interviews to help people practice feeling confident, we could also ask the mock interviewer about what they thought of the ex-offender’s performance – how confident did they appear?
  • combine subjective and objective indicators. This can give a more robust picture of whether or not a change has happened. For example, if we wanted to know if people working in voluntary organisations had increased their skills around applying for grants, we could ask them how confident they would feel about making an application for a larger grant – this gives us a subjective view on whether or not the outcome has been achieved. A more objective measure could be the percentage of applications that were successful (although other factors could also influence this).

Example

Outcome (a single, measurable change)

Indicator (what will we measure?)

Improved literacy skills
  • Number of students passing adult Level 2 literacy at C grade or above
  • Student reading age
Increased job search skills
  • Number of online job portals ex-offenders use to search for jobs
  • Suitability of posts applied for (project worker perspective)
6

Plan how to measure soft outcomes

Some of the most important outcomes from your work may be ‘soft’ outcomes – internal changes to the way people feel or think that are impossible to observe directly. For example, improved self-esteem, improved confidence, increased happiness or increased job satisfaction.

To set indicators for soft outcomes, think about how you would tell if the outcome had been achieved. You may be able to measure observable behaviour changes – sometimes known as setting ‘proxy indicators’. For example, when self-confidence goes up, you might see changes in the way people talk, dress, or behave with others. Self-confidence indicators might be: readiness to make new social contacts; how comfortable they are to speak in a group or to make a presentation.

7

Test knowledge and awareness carefully

Simply asking people if they think they know more than before will not necessarily give you meaningful or accurate information; it is better to set more targeted indicators. For example, a campaign that plans to increase awareness of local recycling facilities could set indicators about whether or not people know where to recycle different materials rather than asking about their ‘level of awareness’.

8

Review and prioritise your outcome indicators

After drafting your outcome indicators, you may be left with a long list of information to collect. Think carefully about your priorities for measurement. Which outcomes are most crucial to evidence? If you have set a large number of indicators for each outcome, which are the most interesting or revealing?

Consider whether you are already collecting information that could be used for monitoring. For example, if you are using validated scales and tools, such as the CORE counselling tool, you can use the score as an indicator of overall mental health.

Check your list of outcome indicators. Will they give you accurate information about your outcomes? If you collected evidence against your indicators, would it be easy for someone external to pick holes in your data, or to discount it? Your indicators will be the foundation for your questionnaire questions, your focus group topics, and your interview schedules so it is important to make sure they are measuring what you intend to measure.

9

Consider whether you need to evidence impact

Your impact (the broad or longer-term effects of your work) can be difficult to evidence. It takes time to measure change over time, and a number of different actors and factors will be involved in creating significant, longer-term changes.

In our example project, the impact would be ‘Improved access to employment for people leaving prison’. It is easy to see how other factors, such as the work done with prisoners by similar organisations, the job market, and changes to government policy could all affect whether or not this longer-term change comes about. This makes untangling the specific role of the project more problematic.

To track impact in a meaningful way, you need to be able to demonstrate the role that your work had to play in creating longer-term change. This involves understanding the role that other organisations and wider factors had (attribution). It also involves understanding how much change would have happened if your organisation did not exist (sometimes referred to as the counterfactual, or deadweight).

Getting to grips with these issues often requires a more sophisticated approach to evaluation and a more detailed research design. Generally, evidencing your organisation’s longer-term impact will require more time, skill and resources than evidencing your outcomes.

However, you may decide to set indicators for keeping track of your impact if you feel that you can measure longer-term change in a meaningful way. In this case, review your impact statement and think about which indicators you would need to collect information against. In our example project, impact indicators for tracking improved access to employment for people leaving prison might be:

  • percentage of ex-offenders entering the workforce after leaving prison
  • how soon ex-offenders entered work after leaving prison
  • the type of work ex-offenders take on after leaving prison.
10

Plan your process monitoring

To understand why your project is bringing about change, you need to identify the factors that really matter in the way you do things – your processes. This may include looking at the way you deliver certain outputs, or other aspects of your work such as your relationships with other agencies or the way staff and volunteers work together.

It can be helpful to set specific indicators for processes. Examples might be:

  • amount of team working
  • extent and type of meetings with other agencies
  • level and quality of contact with the media.
11

Finish the framework

To complete your evaluation framework, you will need to decide on your information collection methods. Read more about methods available and how to choose between them.

Further information

This How To was contributed by NCVO Charities Evaluation Services.

Contributors

Page last edited Dec 16, 2016 History

Help us to improve this page – give us feedback.

1 star 2 stars 3 stars 4 stars 5 stars 2.9/5 from 302 ratings