STEP 7: Measuring Achievements

In this section you will map out ways of monitoring the achievement of your objectives to ensure that your program stays on track. This will also help you improve your program on an ongoing basis, and justify requests for project funding.

You will:

  • Consider the use of pilot testing.
  • Decide what measures to monitor frequently.
  • Select the ones you will measure only at major milestones.
  • Consider setting up a control group.

Note for health promoters.

 
1. Whenever practical, pilot test your program before implementing it widely.

Examples

A few years prior to the development of its formal telework initiative, AT&T conducted a few, localized telework pilot projects in three regional offices.

The approach used in the Action by Canadians and Count Me In! programs were first pilot tested at one company.

When Go Boulder first introduced its Guaranteed Ride Home, it pilot-tested it with a small number of companies. No limits were set on its use so that any possible abuses would become apparent. Similarly, its student transit pass was first introduced at the University of Colorado on a one-year trial basis.

The Environment Network pilot tested its starter kits with 150 participants.

Your Program

What groups or locations are available for pilot testing your program?
Login to Save Plans for Tools of Change
Which of the measurement methods that you listed in "Setting Objectives" would be appropriate for such a pilot?
Login to Save Plans for Tools of Change
 
2. Decide what measures you will monitor frequently to ensure that your program stays on track.

Examples

In The Clean Air Commute pilot, results were collected by the coordinator and marked on a display at the end of each month.

On an ongoing basis, JEEP monitored energy outputs at the local power plant and estimated energy savings using information from home visit reports.

Quinte Regional Recycling continuously tracked curbside lift counts, the weight of waste going to landfill, and the weight of Blue Box materials.

Resource Alert for Canadians:
EnerGuide for Houses facilitates the ongoing measurement of energy-efficiency improvements resulting from the Program. Its blower door test provides accurate before and after data on the decreases in air leakage achieved for each house. Even if there is no second (i.e. after) home visit, the programs software can use the collected data to make sound estimates of the likely energy savings from each recommended action taken.

Your Program

Which of the measurement methods that you listed in "Setting Objectives" could you use frequently or on an ongoing basis?
Login to Save Plans for Tools of Change
 
3. Choose the measures you will track less frequently, e. g., at major program milestones.

Examples

The AIDS Peer Education Program used a self-administered questionnaire completed by 70 students (70% of participants) both before and after the intervention.

Two and six months after the ABC / Count Me In! workshop, each participant was contacted by email to report on their progress. Alternative methods of communication were available for those without email.

Go Boulder conducted a Biennial Diary Study, for which 1,000 residents kept a log of their travels for a randomly assigned day in the second week of September. Six years of data on the progress of the program had been accumulated by 1995. Similar studies were undertaken with employees, who were asked to record their travel habits for a full week.

When employees arrived at work on the day of The Clean Air Commute, they marked the activity they undertook on a chart. Pollution Probe collected the charts and tabulated accumulated points. A telephone survey was conducted at the end of the Clean Air Commute pilot to collect information on clean commuting practices undertaken by members of each of the groups during the three-month period. Respondents were also asked about their intentions for repeating these practices the following summer.

Your Program

Which of the measurement methods that you listed in "Setting Objectives" could you use on a less frequent basis?
Login to Save Plans for Tools of Change
 
4. Wherever possible, make use of control groups & randomly selected groups of people who were not exposed to your program or, at least, not to the parts of it that you are planning to monitor. This helps filter out background changes and influences so that you can be more confident in measuring the changes that are actually a consequence of your program. A less reliable alternative is to compare resource use and/ or people's actions before and after implementing your program.

Examples

AT&T conducted annual telephone surveys to measure teleworking, randomly sampled with a good representation of company employees. From the collected survey results, AT&Tcompared the overall changes, year over year, in the various aspects of telework.

The AIDS Peer Education Program used a control group of 74 students attending another high school in another city. The comparison group received the same questionnaire with no intervention. For ethical reasons, the peer educators later visited these students and provided the presentation.

Of the seven companies which agreed to participate in The Clean Air Commute pilot, three were randomly assigned to serve as a control group.

The Environment Network randomly selected one hundred households that had already received a Green Home Tune-Up; 50 received a sustainable landscaping kit, 50 received a household hazardous waste kit. A third group of 50 residents who had participated in the Green Home Tune-Up program were randomly selected as a control group, and received neither kit.

Tip: Plan how you will select your control group before you engage participants, or you may find it difficult to find the control group you need.

Your Program

How might you set up a control group for your measurements?
Login to Save Plans for Tools of Change