Outcome Measurement

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Apply for Grant

Phase II Application

View Timeline


What is Outcome Measurement?


  • Inputs
  • Resources used in a program
  • Money
  • Staff
  • Volunteers
  • Facilities
  • Equipment
  • Supplies

Activities

What program does with inputs:
  • Feed and shelter homeless families
  • Provide job training
  • Educate public about signs of child abuse
  • Counsel pregnant women
  • Create mentoring relationships for youth

Outputs

Products of a program
  • # of classes taught
  • # of counseling sessions conducted
  • # of education materials distributed
  • # hours of service delivered
  • # participants served

Outcomes

Benefits or changes realized by program participants
  • New knowledge
  • Increased skills
  • Changed
  • Attitudes or Values
  • Modified Behavior
  • Improved Condition
  • Altered status

What is Outcome Measurement?


  • Initial Outcomes ñ Benefits participants experience during or directly after
  • Often changes in participants knowledge, attitude or skills
  • Intermediate Outcomes ñ Links Initial Outcomes to Longer term Outcomes
  • Changes in behavior due to participants new knowledge, attitude or skills
  • Longer-Term Outcomes ñ Ultimate Outcomes a program desires to achieve
  • Represent meaningful changes for participants often in their condition or status

Example

  • Initial Outcomes
  • If youth are mentored by adults who stress importance of education
  • Then youth will see education as important
  • Intermediate Outcomes
  • If youth see education as important
  • Then they will attend school more regularly
  • If youth attend school more regularly
  • Then they are more likely to graduate
  • Longer-Term Outcomes
  • If youth graduate
  • Then they are more likely to become employed and not involved in criminal activity
Outcome Targets: Numerical objectives for a program’s level of achievement on it’s outcomes.
SSpeceific – Concrete who or what is expected to change
MMeasureable – Can see hear, count the outcome
A – Attainable – Likely to be achieved
R – Results Oriented – Meaingful valued results
T – Timely – Targeted Date

Why Measure Program Outcomes?


  • Measuring Program Outcomes gives agency ability to demonstrate effectiveness:
  • Retaining/Increasing Funding
  • Enlisting and Motivating Volunteers
  • Attracting new participants
  • Engaging Collaborators
  • Gaining favorable recognition
  • Garnering support for innovative efforts
  • Winning designation as a model or demonstration site

How do we tackle Outcome Measurement?


Step 1: Get Ready

Task 1: Assemble and Orient an Outcome Measurement Workgroup
Task 2: Decide which programs to start with
Task 3: Develop Timeline
Task 4: Distribute your game plan to Key Players

Step 2: Choose the Outcomes You Want to measure

Task 1: Gather ideas for what program outcome are
Task 2: Construct Logic Model for Your Program
Task 3: Select Outcomes that are Important to Measure
Task 4: Get Feedback on your Logic Model & Outcomes Selected for Measurement

Step 3: Specify Indicators for your outcomes

Task 1: Specify at least one indicator for each Outcome
Task 2: Decide what factors could influence participant Outcomes
Level of success on Outcomes will be different for different participant groups based on:
  • Demographics (age, gender, educational level, income level, disability, single parentÖ)
  • Level of Difficulty (very difficult to help, moderate difficulty, minor difficulty)
  • Level of Involvement (high, moderate, low participation)
  • Organizational unit  (if more than one service delivery facility)
  • Service Delivery (group session vs. 1-on-1, live vs. taped)
  • Avoid collecting everything and anything

Step 4: Prepare to Collect Data on your Indicators

Task 1: Identify data sources for your indicators
Where will you get your data? Will data provide useful, reliable information related to the Outcome?
  • Records
  • Specific Individuals
  • General Public
  • Trained Observers
  • Measurement
Task 2: Data Collection instruments
  • Clearly state your intentions with the research.
  • Include instructions with your survey questionnaire
  • Don’t ask for personal information unless you need it
  • Keep the questions short and concise
  • Ask only one question at a time (the double barreled question)
  • Make sure the questions are unbiased
  • Present the questions in a clean and organized layout
  • Test the survey questionnaire
Task 3: Pretest your data collection instruments and procedures
IF ONLYs
  • I would have worded the question differently
  • I should have made the instructions more clear and easier to follow
  • I would have better defined the scale
  • I should have asked more open ended questions
  • I should have dedicated specific persons to do all of the input
  • I should have made the survey one page

Step 5: Try Out your outcome measurement

Task 1: Develop a trial strategy
Task 2: Prepare data collectors
Task 3: Track and collect outcome data
Task 4: Monitor the outcome measurement process

Trial run your system

During trial you will likely identify issues such as :
  • Overlooked outcomes
  • Inadequately defined indicators
  • Cumbersome procedures
  • Analysis and reporting dilemmas

Step 6: Analyze and Report Your Findings

Task 1: Enter the data and check for errors
Easing the data entry process:
  • Code the data
  • Review the completed questionnaires
  • Assign a survey number
  • Determine links
  • Establish a process for open-ended questions
Task 2: Tabulate the data
Steps to data Tabulation
  1. Count total number of participants for whom you have data
  2. Count number achieving each outcome status
  3. Calculate percentage achieving each outcome status
  4. Calculate other descriptive statistics (averages, medians, modes)

Finding Median

Write the numbers in order
Put all the numbers in numerical order
If there is an odd number of results, the†median†is the middle number
Example:  3, 5, 12
What is median?

Finding  Mode

The “mode” is the value that occurs most often.
If no number in the list is repeated, then there is no mode for the list.
3, 7, 5, 13, 20, 23, 39, 23, 40, 23, 14, 12, 56, 23, 29
In order these numbers are:
3, 5, 7, 12, 13, 14, 20,†23, 23, 23, 23, 29, 39, 40, 56
The mode is?
Task 3: Analyze data broken out by key characteristics
Task 4: Provide Explanatory Information related to your findings
Task 5: Present your data in Clear and Understandable form

Consider using Column charts when:

  1. Comparing two or more variables: same unit of measurement, comparable sizes
  2. Show how much
  3. Consider bar chart when your axis labels are too long to fit in a column chart

Consider using Line charts when:

  1. You want to show data trends over a long period of time.
  2. You have too many data points to plot and the column or bar chart clutters the data.
  3. When you want to show how much has changed over a period of time.

Consider using Pie charts when:

  1. You want to show the breakdown of data into its parts.   What ìpiece of the pieî is it?
  2. You have only one data series.
  3. The data points represent the parts of the whole pie.  What does the whole picture look like?
  4. The parts are of comparable sizes.

Consider using Stacked Area charts when:

  1. You want to show the trend of composition.
  2. You want to emphasize the magnitude of change over time.
  3. You have more than 8 data points to plot.
  4. The data points represent the parts of the whole composition.

Consider Using Scatter Chart when:

  1. Analyzing and reporting relationship/correlation between two variables.
  2. When you want to show ëwhyí. For example: # of hours doing homework is related to grade in class
  3. When there are more than 10 data points on the horizontal axis.
  4. There are two variables that depend on each other.

Consider using a heat map when:

  1. You want to demonstrate or identify areas of opportunity.
  2. You want to show where

Tips for Formatting your Reports

  • Consider the needs of your audience: what information are they looking for?
  • Keep it Simple
  • Include a summary of major points
  • Donít crowd too much on a page
  • Define unfamiliar terms
  • Define each outcome indicator
  • Highlight points of interest with bold type, circles   or arrows
  • Use color to help highlight key findings
  • Label charts and tables clearly ñ titles, rows, columns, axes
  • Identify source and date of the data and note limitations
  • Provide context (history or comparisons)
  • Add variety to data presentation by using bar or pie charts
  • Internal repots should be much more detailed than external

Step 7: Improve Your System

Task 1: Review Your Trial Run Experience, Make Necessary Adjustments, and Start Full-Scale Implementation
Task 2: Monitor and Review your system periodically

Step 8: Use Your Findings – Making Data Pay Off

Detecting Needed Improvements
U$E 1: Identify Outcomes that need attention
U$E 2: Identify Client groups that need attention
U$E 3: Identify Procedures/Policies need improvement
U$E 4: Identify improvements in service delivery
Motivating Staff, Volunteers and Clients
U$E 5: Communicate Program Results
U$E 6: Hold regular program reviews
U$E 7: Identify training and technical assistance needs
U$E 8: Recognize staff and volunteers for good outcomes
U$E 9: Motivate clients
Program Planning
U$E 10:  Identify Successful Practices
U$E 11: Test program changes or new programs
U$E 12: Help planning and budgeting
Reporting to Others
U$E 13: Inform Board Members
U$E 14: Inform Funders (current and potential)
U$E 15: Report to Community

Tips on What to include in Executive Summary:

  • A brief description of the outcome measurement process
  • Outcome highlights, including successes and disappointments
  • Explanations of both successes and disappointments
  • Actions the organization is taking or planning to take to address problems.

Cautions and Limitations

  • Don’t jump to conclusions based solely on the data
  • Assess the impact of sample size and composition
  • Assess the response rate (Aim:50%, Scientific:80%)
  • Consider what proxy measurements you are using
  • Confidentiality
#Follow us on Instagram