Monday, April 2, 2007

Arkansas Evaluation Center

The Minority Initiative Subrecipient Grant Office (MISRGO) and the Evaluation Team proposed the creation of an Arkansas Evaluation Center to the Arkansas Legislature on March 19, 2007.

The idea emerged from our tobacco prevention work. The need for more evaluators and more systematic evaluation training throughout the State became increasingly evident. Not only were more evaluators needed for tobacco prevention work but for many other programs as well. Thus our proposal to establish a center to train new evaluators and provide professional development training for working practitioners.

The bill describes the mission or purpose of the center. Highlights include:

  • enhancing the quality of existing services and programs
  • ensuring accountability for funds and services used
  • “raising the bar” concerning standards and expectations
  • increasing the use of data to inform program operation and decision making.

(Click on picture of Bill to display full PDF of the Bill.)

An empowerment evaluation approach and philosophy has been adopted by the Center to shape practice in communities throughout the State. Empowerment evaluation and other collaborative forms of evaluation are particularly useful in complex collaborative endeavors requiring the participation of diverse groups. It is designed to help groups accomplish their objectives using evaluation concepts and tools.

Special thanks are due to all of the MISRGO staff, grantees, and evaluators for helping to identify this need and transform this need into an action step designed to meet the need.

November 4, 2006 - American Evaluation Association (dissemination)

Dr. Fetterman and Ms. Linda Delaney helped to disseminate the Tobacco Prevention evaluation findings at the American Evaluation Association in November, highlighting the amount of money saved the State as a result of these tobacco prevention programs. They also discussed the politics of tobacco prevention evaluations.

(click on photo above to display video)

September 22, 2006 - Refresher, Focus Group, and Black Caucus

The September 22 workshop was designed to be a refresher and provide additional methodological training. Dr. Fetterman provided an overview of the project and its current status from an evaluation perspective, including the evaluation monitoring system and the community survey. Dr. Fetterman also provided focus group training for the Arkansas MISRGO evaluators. Ms. Delaney faciliated the self-assessment process of the training generating a 4th data point of self-assessment for the empowerment evaluation exercise.

Dr. Fetterman was also invited to meet with the Majority Leader, Senator Tracy Steele, and the Black Caucus during this trip.

He provided them with pertinent data, ranging from the amount of money saved as a result of these programs to the wide-spread community support enjoyed by MISRGO across the State (large constituent base). Dr. Fetterman also met with various news organizations to share the evaluation monitoring system findings and the community survey results.

Evaluation Monitoring Tool Update

We now have an interactive spreadsheet that we share on the web. It is our repository of tobacco consumption data. It provides us with an up-to-date view of: number of materials distributed, number pledge to quit, number who quit, cost saved per person, and cost saved by grantee group. We also have a running total of each of these categories including total cost save the State concerning excess medical costs. As of July 4, 2006, the totals across the State are as follows:

We also maintain a more detailed data sheet, enabling us to view the numbers by grantee (see below).

Community Surveys

We conducted a community survey throughout the State of Arkansas focusing on the views of the grantees' community members. We received 946 responses. The findings were overwhelmingly positive.

A few survey responses are provided below to provide some insight into the level of community support for tobacco prevention programs in the State.

In addition to providing information in the aggregate about community views across the State, we produced a report that provides community member viewpoints about their specific grantee or tobacco prevention program.

Clearing the Air in Communities of Color Conference

We shared the evaluation approaches we have been using to measure change with colleagues at the 3rd Annual Clearing the Air in Communities of Color conference. (See video below.)

(click on picture to display video)

We discussed our training efforts, the empowerment evaluation process, our evaluation monitoring system, and our community surveys. It is a useful summary of our efforts up to May 26, 2006. The slides of our presentation are available to the public.

January 2006

We held a local evaluator workshop in January. It focused on how to use the Evaluation Monitoring System. They will work with the local grantees to help them collect critical baseline data, concerning the number of people smoking in their communities. In addition, it will help programs document the number of people they reach and the number who quit. The system will also translate that into costs saved. [The workshop slides are available on this site.]

A brief video of the training is provided below.

Evaluation Monitoring System and Community Survey
Local Evaluator Workshop Training
(click on picture to display video)

In addition, we are working on community surveys of the grantees so that this information can be fed back into the programs to enable them to enhance their service to their communities.

We will continue to use the Evaluation Monitoring System tool, along with many others, and rate our own performance throughout the evaluation. These tools help to guide us through the journey.

Sunday, April 1, 2007

September 2005 - Common Indicators and Qualitative Methods

The September workshop was extremely productive. The grantees agreed upon common indicators to measure program performance, including the size of the target population, the number of smokers, the number reached with educational messages and services, and the number pledged to quit (as well as documented individuals in smoke cessation programs). A sample is provided below (see Figure 1):

Figure 1. Common Indicators

Dr. George Blevins, MISRGO Director,
introducing the September 2005 workshop.

Grantee oriented interviews and surveys were conducted previous to the workshop to solicit grassroots data concerning how grantees collect data, the quality of the data they have, and program implementation. In addition, recommendations were made during the interviews and workshop, including how to collect sign-in and sign-up sheets as well as pledge to quit sheets. Census and At Risk web site were shared to help grantees collect valid and useful data concerning the smoking rates in their communities.

Risk Factor and Census reports shared with grantees

The workshop focused on how to use various qualitative techniques such as ethnographically-oriented observation and interviewing. The workshop also demonstrated how to construct online surveys (with real examples on the web).

Grantees completing an ethnographically oriented observation exercise
and Dr. Fetterman critiquing their work.

The slides from the workshop training are available in pdf format. The Final Report for Year One summarizing work-to-date is online and available on this site. In addition, a brief video is available concerning workshop highlights.

Brief video of September 2005 workshop
focusing on common indicators and
qualitative techniques for data collection.
(Select arrow on bottom left of the screen to display video)

During the meeting we agreed on the need to develop an Evaluation Monitoring System. (See Figure 17 below.) It is described in the September 2005 Annual Report.

We also thought it might be useful to write a Local Evaluator Training Manual. This will help us create greater consistency in practice, ranging from collecting the same type of data to analyzing it in the same format.

We continued to operate as a group or consortium. We have adopted an empowerment evaluation approach to guide our work. This approach enables us to monitor our progress at each stage in our development as a group. We "check in" each time we meet and assess our progress. In this case, we have documented the third data point or assessment of our progress to date (see graph below).