Share on Facebook Share on Twitter Share on LinkedIn Share on Google+ Share by Email

Skip to content

Key resources

  • Bibliography
    Bibliography
  • Health promotion
    Health promotion
  • Health practice
    Health practice
  • Yarning places
    Yarning places
  • Programs
    Programs
  • Organisations
    Organisations
  • Conferences
    Conferences
  • Courses
    Courses
  • Funding
    Funding
  • Glossary
    Glossary
 
Print this page Print

Does your program work?

TIS banner

How will you determine if your program is working? This section will provide you with a general overview of strategies and examples of tools to monitor and evaluate your programs.

For more information see also:

Monitoring and evaluating your project

The processes of monitoring and evaluation use carefully planned and well-thought-out methods to measure the success of a project (or program) in meeting its goals. They are an important part of the project management process, because they provide:

Key terms used when talking about monitoring and evaluation are shown in Box 1.

Box 1: Monitoring and evaluation key terms

  • performance: what the project is achieving (observable results)
  • measurement: how we determine the impact of a project or program on intended outcomes (e.g. using a questionnaire to find out how many people have smoke-free homes or conducting interviews to find out how people keep their homes smoke-free)
  • indicator: measures that show the extent of progress toward outcomes, especially differences in the lives of the people the project is working for
  • data collection: process used to gather evidence (e.g. giving smoke-free event participants a questionnaire survey)
  • output: what the project is producing with its resources (e.g. a specific activity, product or service)
  • outcome: results and impacts of the project (e.g. a percentage reduction in smoking, a change in behaviour).

Monitoring and evaluation are related processes, but each has a different focus:

Table 1 provides examples of different outputs and outcomes for the TIS Programme. Monitoring outputs relies on describing and counting project activities and the number of people who come to events. In the past TIS reporting focused only on what is included in the outputs column – ‘what we do and who we reach.’ Now, however, you are being asked to think more about ‘what difference do our activities make?’ This is a question about your project outcomes.

Table 1: Examples of outputs and outcomes for TIS

Outputs

Outcomes

Smoke-free workshops delivered to 100 workers in 10 community organisations

85 workers have increased knowledge of benefits of smoke-free workplace and increased commitment to being smoke-free at work

 8 Organisations are smoke-free

Brief intervention training provided to all staff (N=20) in grant recipient organisation

20 staff have increased skills to support TIS activities

Most staff describe increased confidence when working with community members

You can find out more about the 'how, who and what' of TIS project evaluation in Box 2.

Box 2: The how, who and what of evaluation

How to plan an evaluation?

It’s a good idea to work out an evaluation plan for your project, so you can keep track of the evaluation. In your plan consider the following points:

  • What are the reasons for evaluating your project?
  • Who will read or listen to your evaluation?
  • What sort of resources will you need for your evaluation?
  • What are your evaluation questions?

Who will do the evaluation?

Work out who will be responsible for organising and writing up the evaluation.

Who should participate in an evaluation?

Good evaluations involve those who are interested and affected by your project. Involving people from the earliest stages of the project’s development to the final evaluation can encourage local communities to set up, control and own a project. Putting together a working group (a group appointed to study or report on a particular question) of community members also helps the evaluation process. A working group brings the values and shared interests of the community into the process. Often they are also the ones who are best placed to talk about the needs of their community.

What should you evaluate?

What aspect of the project do you want to evaluate? For example, do you want  to know if the project’s objectives were met? Or do you want to know what people liked about the project? There are lots of evaluation questions you could use. Examples of the sorts of questions you might use to answer different questions are provided in the Survey question bank.

What sort of resources will you need for your evaluation?

Be realistic about what resources are available to undertake the evaluation. This includes funding, time, staff, salaries, material, equipment and operating costs.

Source: material adapted from Kruger K, McMillan N, Russ P and Smallwood H (2007) Talkin’ up good air: Australian Indigenous tobacco control resource kit. Melbourne: Centre for Excellence in Indigenous Tobacco Control [1]

Or for a more in-depth general discussion of behaviour change evaluation and definitions of some other terms used in monitoring and evaluation, you might find the Evaluation toolbox useful.

When should I think about monitoring and evaluation?

Monitoring and evaluation should be built into your project from the start. It is also a requirement of TIS Programme funding to measure your outcomes every six months for Indigenous Australians’ Health Programme (IAHP) grant agreement progress reporting. The Healthy, deadly and strong: healthy lifestyle worker toolkit developed by HealthInfoNet on behalf of the Australian Government (2012), describes a simple six step process to building evaluation into a project plan:

    1. Plan the project - what do you want to achieve, who do you want to involve, what will you do and how will you do it?
    2. Plan an evaluation - decide and consult about what you will measure, identify key questions, identify local sensitivities, identify good processes, and allocate necessary resources.
    3. Design the evaluation - decide on the methods you will use to collect the information you want, such as counting how many attend a smoke-free community event (output), giving out a short questionnaire asking what people thought of the event, and whether those who attended learnt anything about smoking harms, benefits of quitting, or quit support available (outcomes).
    4. Collect and record your information - do this systemically to get a true picture of what your project is achieving.
    5. Analyse your information - see if the program is achieving what you intended, or whether there are any unexpected outcomes, identify the lessons to learn. Decide if you need to make any changes to your project to keep on track.
    6. Provide feedback on your findings - let the people involved in the project such as the community, your organisation and the participants know about what was achieved. 

Examples of ways to present your data using charts and dashboards for good visual impacts can be found here or on the Tools and resources to monitor your program page. 

Figure 1: Keeping your project on track

Figure 1 shows how monitoring and evaluation forms a central part of your project. You can use the information that you collect to make sure activities are on track and that you are achieving the outcomes you intended, adapting and improving your project as you go. This process of using monitoring and evaluation data to improve your project or activities as you go is known as continual quality improvement (CQI). If you are working in a clinical setting you might be interested in the Tobacco control CQI tool developed by Menzies School of Health Research in association with One21Seventy. More tips for how to use monitoring and evaluation data are provided in Box 3.

Box 3: Using monitoring and evaluation data

Projects do not always run as planned or as expected, and are a learning experience for all those involved. Monitoring your activities and evaluating outcomes will help you keep track of progress, identify important changes and make it easier to see what you have achieved.  Answering the following questions may help keep your project on track:

  • What do the early results mean in terms of what you are trying to achieve?
  • Should you be changing the way you are doing things?
  • How do you continue the positive changes you are achieving?
  • Are you doing things in the best possible way?
  • Are you talking to the right people about the progress of your activities?
  • Are you making progress, making a difference?

Source: adapted from Australian Indigenous HealthInfoNet (2012) [2]

Reporting on your TIS activities: filling out the Department of Health's performance reports

You will need to provide a formal written report to the TIS Programme funder, the Australian Government Department of Health (DoH), using the Indigenous Australian's Health Programme (IAHP) twelve month performance report template and the TIS six month performance report template. The next reporting period for Grant Recipients is 1st January to 30th June 2017, which is due 15 August 2017:

TIS six month performance report template: 1st January - 30th June 2017

If you need assistance with either report, the document, Guidance on filling out your IAHP performance report can be found here. It comprises a good practice example of reporting against all five National TIS Performance Indicators, along with general tips. Reference is made to the TIS Programme logic, which comes from the TIS Programme monitoring and evaluation framework. The TIS Programme logic diagram is useful to keep in a prominent place so all of your team members can refer to the elements and intended outcomes of the TIS Programme.

Related pages

For further information about monitoring and evaluating your TIS activities, please refer to the links below.


[1] This material is used with permission from the Commonwealth of Australia. Talkin’ up good air contains materials that were contributed by Quit Victoria (Anti-Cancer Council of Victoria), Apunipima Cape York Health Council, National Heart Foundation of Australia (NSW Division), Council of Social Services of New South Wales and Queensland Health and which remain their property.

[2] Australian Indigenous HealthInfoNet (2012) Healthy, Deadly and Strong: Healthy Lifestyle Worker Toolkit. Perth, WA: Australian Indigenous HealthInfoNet

More on this topic...

 
Last updated: 26 July 2017
 
Return to top
 
spacing
 


Australia's National Research Centre on AOD Workforce Development National Drug and Alcohol Research Centre National Drug Research Institute