Tobacco Prevention Evaluation (MISRGO)

MISRGO Evaluation: 2nd Quarter Progress Report Issued (mid-year report on progress)

2012-01-23T10:25:00.000-08:00

MISRGO Evaluation: 2nd Quarter Progress Report 2012. The mid-year assessment is available here on the web.

MISRGO grantees have a solid track record of accomplishments. They have contributed to the creationand enactment of smoke free legislation, including Act 811[i]. They educate the public about tobaccoprevention legislation, including Acts 8[ii]& 13[iii]. In addition, they are working acrossthe state to establish smoke free parks and extend perimeter laws. They have made extensive use of socialmedia to positively influence youth concerning tobacco consumption. They have broad-based support acrossthe State for their tobacco prevention work. (See MISRGO Empowerment Evaluation[iv]:June 2011 Annual Report, Fetterman, Tremain, and Delaney, 2011. See also Appendix A for a glossary ofgrantee names and abbreviations.)

Purpose

The purpose of this report is to highlight grantee progressat mid-point in the year. MISRGO grantees use the Center for Disease Controland Prevention’s approved and recommended intervention areas as follows:

Area 1: Eliminate Exposure to Secondhand Smoke

Area 2: Preventing Initiation Among Youth and Young Adults

Area 3: Promoting Quitting Among Youth and Adults

Area 4: Addressing Disparities

MISRGO grantees are making steady progress toward annualgoals and in many cases are significantly exceeding anticipated mid-year goals (or2^nd quarter benchmarks). In brief:

·      21 Exceeded Year-End Goals
·      5 Met Year-End Goals
·      7 Exceeded 2^nd Quarter Benchmarks
·      10 Approximated 2^nd Quarter Benchmarks
·      5 Progress But Below Benchmarks
·      21 Planning Complete – Implementation Later in the Year

Highlights. Granteeachievements made to date (Quarter 2 of 4) are highlighted below. They are organized by intervention area.

Area 1: Eliminate Exposureto Secondhand Smoke

3196 Educated. 3196 parents and law enforcement officialseducated about the benefits of smoke free environments, particularly in cars,e.g. Act 811

608 Pledged. 608 young adults pledged to make theirhomes smoke free environments.

317 Informed. 317 youth and young adults wereinformed about the laws governing smoking in cars, Act 811.

239 Reported. 239 youth reported that smoking was nolonger allowed inside their homes.

In addition, grantees are working to implement smoke freepolicies in parks and churches, convince businesses and landlords to supportbans on smoking, recruit bars and schools to adopt smoke free policies, andrecruit Youth Extinguishing Smoking (YES) team members.

Area 2: PreventingInitiation Among Youth and Young Adults

296 Pledged. 296 minority youth pledged to remaintobacco free

235 Educated. 235 youth educated about the dangers oftobacco

99 Trained. 99 youth trained to educate youth aboutdangers of tobacco

72 Quitline. 72 youth and young adults helped toaccess Quitline

In addition, grantees worked to increase awareness aboutsmoking and pregnancy, conduct storefront surveys, sponsor smoke-free events, andorganize YES teams.

Area 3: Promoting QuittingAmong Youth and Young Adults

114 Educated. 114 African American and Latino youthand young adults educated about the dangers of smoking

70 Quitline. 70 enrolled in SOS Quitline faxreferral program

In addition, grantees worked to enroll youth in the 40 Days to Freedom curriculum, organizeYES rallies, train merchants to deny minors from purchasing tobacco, increasecompliance rate concerning sales to minors, enroll students in school-basedcessation programs, and educate retailers about point of purchase.

Area 4: AddressingDisparities

4005 Educated. 4005 underserved youth educated aboutthe dangers of tobacco

235 Informed. 235 youth participated in workshops toincrease tobacco awareness

In addition, grantees are working to increase the number of minorityorganizations promoting tobacco control, educating about the hazards of secondhand smoke, implementing no smoking programs in churches, convincing youth topledge not to smoke in their homes, removing tobacco advertising from stores, andimplementing the 40 Days to Freedomand Keeping It Real curriculum.

Conclusion

MISRGO grantees mid-year performance is substantial. Many have exceeded their annual goalsin the 2^nd quarter. Many others are on target, meeting their benchmarks. There are grantee activities scheduledlater in the year. Theseactivities should enable them to more closely approximate their annual goalslater in the year. A few havecompleted their planning work and have initiated implementation but may need toreassess how realistic their goals are given their funding capacity.

The evaluation findingsonly scratch the surface of MISRGO grantees’ commitment and dedication totobacco prevention. The details oftheir efforts are explored further in the remainder of this report. (See Appendix B for cumulative grantee activitylogs as well.)

[i] Act 811 builds on Act 13 by protecting“children under age 14 from secondhand smoke while in vehicles.” See: http://arkansased.org/about/legislation/funding_archive.html. See also: http://www.nwaonline.com/news/2011/apr/23/smoking-ban-set-protect-kids-under-14-20110423/

[ii] Act 8 is Arkansas’ Clean Indoor Air Act. It is designed to “protect workers in Arkansas from secondhand smokein the workplace and to protect the citizens of Arkansas from secondhand smokein public places”. For moreinformation see:http://www.healthy.arkansas.gov/programsServices/environmentalHealth/arcleanair/Pages/default.aspx

[iii] Act 13 ArkansasProtection from Secondhand Smoke for Children Act of 2006 (Act 13) “prohibitssmoking in all motor vehicles in which a child, who is less than 6 years of ageand who weighs less than 60 pounds, is restrained in a child passenger safetyseat as required by Arkansas law.” See: http://www.healthy.arkansas.gov/programsServices/tobaccoprevent/Documents/reports/HowSecondhandSmokeAffectsKidsSurvey.pdf

[iv] See Fetterman and Wandersman, 2005concerning more detail about empowerment evaluation.

Additional Note

Please review the entire report for additional details concerning overall progress and individual efforts at mid-year. Grantee progress as a group is presented below. It is organized by intervention area.

MISRGO Evaluation at AEA November 2011

2011-11-17T22:47:00.001-08:00

MISRGO Evaluation at the American Evaluation Association.

The MISRGO evaluation dashboard was presented by Dr. Fetterman, Dr. Tremain, and Ms. Delaney at the American Evaluation Association, November, 2011. (One of the MISRGO evaluation activities, in addition to monitoring and evaluation, is dissemination. The presentation slides are available on this page.) The dashboard is being used by the MISRGO tobacco prevention grantees to monitor their performance.

It consists of: a baseline, a goal, and benchmarks (approximating the goal throughout the year). They document their actual performance and record it on the dashboard and compare it with their benchmarks to determine if they are on track throughout the year. This dashboard signals when a grantee is accomplishing their objectives and on schedule and when they are not. If they are not on track, mid-course corrections are encouraged.

MISRGO Evaluation Dashboard

Dr. Fetterman highlighting the value of the dashboard to monitor progress.

Dr. Tremain discussing the importance of setting realistic goals.

Ms. Delaney emphasizes the role of SMART goals and objectives.

The evaluation dissemination strategies also include publishing articles and books discussing MISRGO evaluation activities. The evaluation is mentioned in Dr. Fetterman's latest book, Ethnography: Step by Step (2010). It is also highlighted in a popular (one of the most downloaded) American Journal of Evaluation articles titled Empowerment Evaluation: Yesterday, Today, and Tomorrow (Fetterman and Wandersman, 2007).

1st Quarter Empowerment Evaluation Workshop

2011-10-17T17:56:00.000-07:00

1^st Quarter Empowerment Evaluation Workshop. The first evaluation workshop of the year was held on October 13, 2011at the Arkansas River Education Service Cooperative in Pine Bluff, Arkansas.

The empowerment evaluation training slides are available here.

Training focused on the:

1) Evaluation Dashboard - Actual Performance, Benchmarks, Goals, and Baseline

The dashboard consists of recording grantee baselines,goals, and benchmarks. It also includedrecording actual performance to facilitate the comparison of their benchmarkswith their actual performance. This helps grantees monitor their performance throughout the year andmake mid-course corrections (and ask for assistance) if they are not matchingtheir benchmarks.

A hands-on exercise was used to ensure all grantees were able to complete the dashboard in a consistent and accurate manner.

2) Empowerment Evaluation

The group is responsible for monitoring and evaluating their own performance, with the assistance of their team of empowerment evaluators (Drs. Fetterman and Tremain and Ms. Delaney). The process has 3 steps: 1) establishing a mission; 2) taking stock of current status; and 3) planning for the future.

The workshop focused on taking stock since the mission has been established as part of the MISRGO grant process.

Taking Stock is a group baseline exercise. It has two steps: Prioritization and Rating.

Prioritization. We began by identifying the most importantactivities for the group to evaluate as a group. (The are the most important activitiesfrom the group's perspective, in order to accomplish their goals at that time). It is impossible to evaluate everything. Therefore everyone was given 5 dots to cast their vote on the most important activities to evaluate.

The result of this exercise is presented below in a prioritization bar chart.

Bar Chart: Prioritization

Rating. We selected the most important activities and rated them. Grantees rated each activity on a 1 (low) to 10 (high) scale and then discussed the reasons for the ratings

This exercise is designed to record the groups view of theirefforts at the beginning of the year. This exercise will be conducted again later in the year to determine ifthe group is making progress (as a group) in areas they determined to becritical to accomplishing their objectives.

A bar chart of the ratings is presented below. Grantee efforts concerning smoke-free workplace policies was rated highly (8.1 on a 10 point scale) - based in large part on their successful efforts to establish smoke free policies, e.g. Act 811.

Bar Chart: Rating

3) Social Media - YouTube, Twitter & Facebook

A number of grantees are using social media to get thetobacco prevention message across, particularly to youth. One of the popular social media tools is YouTube. A demonstration was provided withspecific guidelines on how to produce a quality video.

Example of YouTube Video

of an

Empowerment Evaluation in South Africa

The focus of the presentation was on planning and outliningthe story before beginning the video editing and production stage. Storyboarding was used as a tool to help “lay out thestory”. It was demonstrated using a row of individual pictures and a storyline that had a beginning, middle, and ending.

YouTube - Storyboarding with Individual Slides

YouTube: User-friendly Video Uploading Tool

We produced two YouTube videos during the workshop. The Coalition for Tobacco-Free Arkansas video been uploaded and is available on YouTube.

Coalition for Tobacco Free Arkansas YouTube Video

Katherine Donald

Grantee Collaboration

In addition, grantees discussed common problems andsuccessful strategies. Grantees with expertise in policy, such as the Coalition for TobaccoFree Arkansas, shared their tools with grantees with media expertise,specifically the Mississippi Coalition for Tobacco-Free Arkansas.

Katherine Donald
Coalition for Tobacco Free Arkansas

Elroy Brown
Mississippi Coalition for Tobacco-Free Arkansas

These two grantees served as models for other grantees,encouraging them to share what they learned with each other as well.

Evaluation Consultation: Critical Friends

Grantees were reminded that they were responsible for theirown individual evaluations. However,they were encouraged to use their assigned evaluators, Dr. Beverly Tremain and Ms. Linda Delaney, to helpthem:

review the logic of their program andcorresponding evaluation
make sure the activities contributed to theirgoals
reassess their strategies or activities if theyare not helping them accomplish their goals
prepare their reports for MISRGO
monitor their own performance throughout theyear

Linda (right) consulting with
Imogene (left) about prioritizing

Bev recording ratings

Leroy, Bev & Linda
discussing reporting

MISRGO Reminders

MISRGO reminded the grantees to work with their evaluatorsand turn in their dashboards and quarterly reports on time to help facilitate analysis acrossgrantees and to help MISRGO meet its reporting requirements.

Annual Report - 2011 MISRGO Evaluation Annual Report

2011-07-26T17:06:00.000-07:00

2011 Annual Evaluation Report. The complete 2011 Annual Evaluation Report is available online. It highlights grantee accomplishments throughout the year.

A few highlights from the report are presented below.

Tobacco Use in Arkansas

It begins with a portrait of tobacco use in Arkansas. Over 480,000 individuals smoke cigarettes in the State of Arkansas. That is 22% of the population. Arkansas ranks 44th among states concerning the use of tobacco.

The average tobacco use across MISRGO serviced counties is 24%.

Highlighted Accomplishments

Smoke Free Policy

MISRGO grantees have been effective creating tobacco free policy and environments.

MISRGO grantees adhere to CDC recommendations in this regard. The Coalition for a Tobacco Free Arkansas was instrumental in contributing to the passage of a new law - Act 811. It strengthens Arkansas' smoke-free car law, Act 13 of 2006, by increasing the age of children protected from second hand smoke from 6 to 14 years of age. This impacts 827,411 youth.

Smoke Free Parks and Companies

MISRGO grantees have helped to establish 13 smoke-free parks during 2010-1011. Two grantees have been granted permission to place smoke-free signs in parks in lieu of a formal ordinance and two parks have created voluntary perimeters. This is in addition to the 56 smoke-free parks already in Little Rock, as a result of the Coalition for Tobacco-Free Arkansas' work to help pass the ordinance.

The Mississippi County Coalition for a Tobacco Free Arkansas helped to convince local industry to adopt smoke free policies.

Newspapers and Social Media

MISRGO grantees have used local newspapers and social media to extend the impact of their work. They have sparked the debate about tobacco use and prevention in the press.

MISRGO grantees have also used the press (including television, radio, and newspapers) to highlight their accomplishments

Grantees Tyler Clark and Page Daniel, NWA Tobacco-free Coalition, have used social media, specifically YouTube, to help educate about air quality in smoking bars in Fayetteville (http://www.youtube.com/watch?v=0_2Lhpi07-0). Bruce Lockett of Future Builders, Inc. uses YouTube to educate youth about the dangers of smoking. Kionti Traylor, Garland County CARES Quapaw House, uses radio broadcasts to reach youth (http://web.me.com/profdavidf/nonsmoking.mov).

Also see Page Daniel and Tyler Clark's YouTube video of their 2011 Survey of Registered Voters - used to determine the level of support for their smoke free campaign (http://www.youtube.com/watch?v=3HDDQ9TdFh4). (A brief article about their work and use of this information is provided online in Ozarks unbound.)

Grantees, Carolyn and Eddie Yarborough, used local television to get the tobacco prevention message across. Billboards continue to be an additionally effective tool.

Research

MISRGO grantees have used extensive use of surveys to determine attitudes toward banning smoking from bars and used the data to successfully educate the public about the need for smoke free environments. They are sharing their data including the Fayetteville Arkansas Air Quality Monitoring Study conducted by the Roswell Park Cancer Institute. These kinds of studies have demonstrated how employees and patrons in Fayetteville bars are being exposed to harmful levels of air pollution.

Quit Line

The Quit Line is an important tool in the fight against tobacco consumption.

MISRGO grantees have submitted 948 fax referrals to the Quitline this year. In total MISRGO grantees have submitted over 3,217 fax referrals to the Quitline, excluding grantees no longer funded.

Pledges & Successes

A pledge can help prevent or stop a person from using tobacco.

MISRGO grantees have convinced 1448 youth to pledge to be smoke free this year. They have also convinced 200 parents to pledge not to use tobacco in their homes or in their cars.

Currently funded MISRGO grantees have helped approximately 2,710 members of their communities throughout the state to stop using tobacco.

Education

Education in its various forms represents the backbone or foundation of any effective tobacco prevention program.

MISRGO grantee educational efforts resulted in:

66,766 receiving information about dangers of tobacco use

1,177 receiving information about Acts 8 & 13

2,400 receiving information about smoke-free ordinances

In addition, pregnant, low-income, homeless, and Hispanics or Latinos received tobacco prevention educational materials from MISRGO grantees.

318 pregnant, low-income & homeless people received tobacco prevention information

300 Hispanics received tobacco prevention information

Grantee Activities

MISRGO accomplishments are the result of many activities, monitoring tools, and administrative oversight and technical assistance.

An abbreviated list of grantee activities is provided below to highlight the wide-spectrum of work required to produce MISRGO outcomes.

Working to establish ordinances in parks
Promoting organizations in the community that review tobacco policies (e.g., restaurants and facilities)
Helping to increase compliance rates among retailers
Identifying disparities concerning sales to minors
Providing tobacco education through surveys and media outreach
Providing AHDC Workshops on ACTs 8 and 13
Distributing educational materials & providing tobacco prevention sessions
Promoting the Arkansas Tobacco Quitline
Promoting the fax back referral program
Helping to decrease tobacco advertising
Helping to increase collaboration with other organizations (e.g., organizations serving minorities, clinics/physicians, schools) to promote a tobacco free message
Helping to encourage smoke free households and cars
Training youth advocates or FAST teams in tobacco prevention

Monitoring Progress

MISRGO has implemented a tool to monitor grantee progress and performance. It is designed to help grantees establish their own goals and monitor their own efforts. The tool also enables MISRGO to determine if grantees are making satisfactory progress or if assistance is needed.

Baseline. Grantees established a baseline, such as the number of minority members of the community already receiving educational materials about second hand smoke. In the example below, the baseline was 0.

Goal. Grantees then established a goal or multiple goals, the number of minority community members they would like to receive educational materials about second hand smoke by the end of the year. In this case the number is 1000 for this year and 5,000 over a 5-year period.

Benchmarks. Grantees created benchmarks, or steps toward their goal. This helped them measure their progress toward their goals. In this case the grantee created the following benchmarks for this year: 250 in the 1^st quarter, 500 in the 2^nd quarter, 750 in the 3^rd quarter, and 1000 in the 4^th quarter or end of year goal.

A review of grantee performance based on these data suggests that they are making satisfactory progress toward end-of-the-year goals. Based on a triangulated review of 18 grantees’ performances, the evaluation team found:

Area 1. Eliminating Exposure: 11 grantees meeting or exceeding their benchmarks, 3 near their benchmarks, and 4 not meeting benchmarks.

Area 2. Preventing Initiation: 13 grantees meeting or exceeding their benchmarks, 4 near their benchmarks, and 0 not meeting benchmarks.

Area 3. Promoting Quitting: 14 grantees meeting or exceeding their benchmarks, 2 near their benchmarks, and 1 not meeting benchmarks.

Area 4. Addressing Disparities: 15 grantees meeting or exceeding their benchmarks, 2 near their benchmarks, and 1 not meeting benchmarks.

Public Support (State-wide)

There is State-wide support for the use of public funds for MISRGO tobacco prevention programs. Ninety-nine percent of the 929 people surveyed throughout the state (who were familiar with MISRGO grantee programs), stated they thought that tobacco prevention and education programs were a good use of public money.

Similarly, 99% reported that they would recommend their local MISRGO-related tobacco prevention & education agency to their friends or family members if they smoked or were thinking about smoking.

Community members throughout the State also thought that the MISRGO-related programs have been effective in preventing people from using tobacco and stopping them from using tobacco (95-96% of those surveyed).

Community members rated MISRGO-related tobacco prevention programs effective or very effective (94%).

The majority of community members surveyed thought MISRGO-related tobacco prevention agencies were effective or very effective concerning the following activities:

90% Sending out educational literature

92% Letting people know about the dangers of second hand smoke

91% Telling people about the QUIT Line

86% Educating retailers to not sell tobacco products to minors

84% Getting perimeter laws

86% Getting smoke free parks

79% Getting advertisers to reduce their advertising of tobacco products

90% Building coalitions with other anti-tobacco agencies

88% Helping minorities stop smoking (or not begin)

The State-wide community-based support for MISRGO-related tobacco prevention agencies, highlights MISRGO’s high degree of success working with minority communities. MISRGO’s work is considered credible and valued throughout the state.

Capacity Building

MISRGO has provided numerous workshops, lectures, and technical assistance sessions to improve grantee performance. (See Appendix C.) The MISRGO evaluation team has also provided lectures, demonstrations, exercises, and workshops to enhance evaluation capacity. Workshops focused on tools to conduct both process and outcome evaluations. Technology training was embedded in the evaluation training, ranging from online survey deployment to the use of digital photography to document and disseminate accomplishments. Highlights of the October 13, 2010 and April 28, 2010 workshops are discussed below.

October 13, 2010 MISRGO Empowerment Evaluation Workshop: Baselines and Benchmarks

This MISRGO empowerment evaluation workshop focused on: documenting actual performance in comparison to the baseline, goals, and benchmarks. We critiqued grantees’ five-year plans, using this model. Then we applied the same approach to quarterly reporting for the year. By comparing actual performance with the benchmarks we were able to determine if the grantees were meeting their goals or if they needed to fortify their efforts. This approach helps monitor change over time. Slides from the presentation are available online at the MISRGO Empowerment Evaluation blog (http://tobaccoprevention.blogspot.com).

The evaluation team, Dr. Beverley Tremain, Ms. Linda Delaney, and Dr. Fetterman provided the grantees with individualized assistance as they refined their first quarterly report.

As the grantees volunteered their work for a group critique, it became clear that this was a learning process for the entire group.

Key lessons learned during the workshop included:

separating out goals, instead of clustering them in one sentence, in order to track them more effectively
ensuring that activities are related to the goal
remembering to record actual performance data
stating benchmarks for each year and each quarter; (as opposed to only one year and making the reader assume that the same benchmarks apply to each quarter or year)
using actual numbers verses percentages; stating the goal in terms of a number
completing the calculations for the reader; (instead of stating that the goal is to improve by 20% and making the reader go back to the baseline in a previous report to calculate 20% of the baseline figure)
similarly, avoiding ambiguous statements, e.g. will increase, but not specifying by how much or from what baseline; will improve, without stating how they will improve, etc.
using the matrix instrument to facilitate record keeping, e.g. baseline, goals, benchmarks, and actual performance
creating bar charts that match the matrix data or figures

Grantee discussions were animated and productive. They shared strategies to accomplish goals they had in common.

The grantees also made a “good faith” commitment to focus on fewer things that they all had in common in order to have a greater “dose” effect. Even the breaks were productive, as grantees, MISRGO staff, and empowerment evaluators, shared insights into what works and what has not worked with each other. MISRGO staff were viewed as supportive and helpful.

The Arkansas Department of Health was invited to share their knowledge and resources with the grantees. One of their valuable tobacco related data and report web pages is available online.

They recommended Center for Disease Control reports including:

They also shared their Tobacco Data Deck 2010, providing useful State-wide data to compare with local grantee efforts. The Data Deck is available on the ADH web site and will be available on the GEMS web-site. (GEMS is the web-based tool for grantees to enter their data.)

The workshop concluded with a local “best practices” exchange. Grantees mixed evidenced-based practices with practice-based evidence.

April 28, 2011 Empowerment Evaluation Workshop: Build Capacity to Reduce Tobacco Consumption

The evaluation team conducted an engaging and productive workshop on April 28, 2011 in Pine Bluff. The workshop was held at the Arkansas River Education Co-op.

Dr. Fetterman provided an overview about empowerment evaluation for new members, reminding everyone that they are in charge of conducting the evaluation and that the three empowerment evaluators are facilitators and coaches to keep it rigorous and on track.

We also briefly discussed the MISRGO grantee evaluation dashboard that we developed and agreed to implement. The dashboard is used to let everyone know what grantee goals are, benchmarks (or steps along the way), and actual performance, so that the performance each quarter can be compared with grantee benchmarks (or planned mini-goals for each quarter).

We demonstrated what the data looks like when transformed into a bar chart. The bar charts make it easy for grantees to see their progress towards their goals. If they were not making progress, then the charts represented a red flag. It let everyone know it was time to ask for help with alternative strategies or to rethink the scale of the effort.

Empowerment evaluators Dr. Beverly Tremain and Ms. Linda Delaney provided instruction and coaching during the workshop. They highlighted how the reporting format could be improved, providing constructive suggestions to enhance their presentation.

A detailed presentation of MISRGO evaluation capacity building efforts is available on the web at: http://tobaccoprevention.blogspot.com. (See also Fetterman and Wandersman, 2005 for a detailed discussion about evaluation capacity building.)

Conclusion

MISRGO grantees have had a significant impact on tobacco-free legislation, contributing to the passage of Act 811. This has an impact on over 827,411 youth.

MISRGO grantees continue to have had a significant impact on minority populations in Arkansas They have established a strong rapport with African American, Latino or Hispanic, and Asian communities in the State. This relationship has enabled them to disseminate tobacco prevention literature to youth and adults in these communities. It has also been instrumental in encouraging minorities in these communities to establish tobacco-free environments, particularly in public parks. MISRGO grantees continue to increase the numbers of community members calling the QUIT line. (See Appendix D for a list of the counties served throughout the state.)

MISRGO grantees work in alignment with CDC principles and best practices, focusing on:

1) eliminating exposure to second hand smoke

2) preventing initiation

3) promoting quitting, and

4) addressing disparities

They work in a wide-range of thematic areas including: education, promoting smoke-free environments, contributing to policy, and cultivating legislative relationships. They use conventional media tools, such as television, radio, and newspapers, to generate awareness and a dialogue about the issues. MISRGO grantees have also used social media, including YouTube to reach out to youth.

Minority community members throughout the State support the use of public funds for MISRGO-based tobacco prevention programs. This support is based in part on their perception of the grantees’ effectiveness in preventing and stopping people from using tobacco in their communities.

MISRGO grantees are making progress toward their stated goals and objectives. Continued support will enhance their capacity to reduce tobacco consumption in minority communities throughout the State of Arkansas.

_/_/_/_/_/_/_/ _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

References

Centers for Disease Control and Prevention. Best Practices for Comprehensive Tobacco Control Programs – 2007. Atlanta: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; October, 2007.

Fetterman, D., Tremain, B., and Delaney, L. (2011a). MISRGO Empowerment Evaluation. January 2011. Report: Accomplishments. San Jose: Fetterman & Associates (January 6, 2011).

Fetterman, D. and Wandersman, A. (2005). Empowerment evaluation principles in practice. New York: Guilford Publications.

Shultz, J.M., Novotny, T.E., and Rice, D.P. (1991). Quantifying the disease impact of cigarette smoking with SAMMEC II. Software. Public Health Reports, May-June, 106(3):326-333.

Starr, G., Rogers, T., Schooley, M., Porter, S., Wiesen, E., Jamison, N. Key Outcome Indicators For Evaluating Comprehensive Tobacco Control Programs. Atlanta, GA: Center for Disease Control and Prevention, 2005.

Substance Abuse and Mental Health Services Administration, Office of Applied Studies (2008). Results from the 2007 National Survey on Drug Use and Health: National Findings (NSDUH Series H-34, DHHS Publication No. SMA 08-4343). Rockville, MD.

See the 2011 Annual Evaluation Report for additional details and the Appendices.

MISRGO Empowerment Evaluation Workshop April 28, 2011

2011-04-30T09:43:00.000-07:00

MISRGO Empowerment Evaluation Workshop to Build Capacity and Reduce Tobacco Consumption. We conducted an engaging and productive workshop on April 28, 2011 in Pine Bluff. The workshop was held at the Arkansas River Education Co-op.

We also briefly discussed the MISRGO grantee evaluation dashboard that we developed and agreed to implement. The dashboard is used to let everyone know what our goals are, our benchmarks (or steps along the way), and our actual performance, so we can compare our performance each quarter with our benchmarks (or planned mini-goals for each quarter).

This is what the data look like when transformed into a bar chart. It makes it easy for us to see our progress towards our goals. If we are not making progress, then the chart is like a red flag. It lets us know it is time to ask our colleagues for help with alternative strategies or to rethink the scale of our efforts.

Empowerment evaluation coaching: Beverly Tremain is facilitating the section of the session about reporting and how we can improve on our reporting format.

Here is Linda Delaney facilitating a session on goals with her grantees.

Here are the three coaches together (what a team):

Beverly Tremain, David Fetterman, and Linda Delaney.

The session show stoppers were the grantees themselves. They presented their accomplishments. One accomplishment involved passing a new law - Act 811- that extended the protection to children, concerning second hand smoke. (Contact Katherine Donald at the Coalition for a Tobacco Free Arkansas for more details.)

Other educational accomplishments included getting our tobacco prevention message across to youth and adults using YouTube videos (Tyler Clark, Page Daniel, Bruce Lockett) and radio broadcasts (Kionti Traylor).

Tyler Clark, Project Coordinator

NWA Tobacco Free Coalition

(click on picture below to view YouTube Video)

Speaking about Air Quality Study in Smoking Bars in Fayetteville

Also see Page Daniel and Tyler Clark's YouTube video of their 2011 Survey of Registered Voters - used to determine the level of support for their smoke free campaign. (A brief article about their work and use of this information is provided online in ozarks unbound.)

We also have a host of additional grantees using social media to get the message across.

Bruce Lockett - Future Builders, Inc.

Kionti Traylor - Garland County CARES

Kionti produced her own radio broadcast
(see below for a video based on the radio broadcast)

Kionti Traylor Garland County CARES Quapaw House

Our grantees have also used local television to get the message across. Contact Carolyn Yarborough for more information about her television advertisements.

Our grantees have used extensive use of surveys to determine attitudes toward banning smoking from bars and used the data to successfully educate the public about the need for smoke free environments. They are sharing their data including the Fayetteville Arkansas Air Quality Monitoring Study conducted by the Roswell Park Cancer Institute. These kinds of studies have demonstrated how employees and patrons in Fayetteville bars are being exposed to harmful levels of air pollution. They have also won awards for their work.

Page Daniel from Fayetteville

NWA Tobacco-Free Coalition

We also continue to collect our own data about our process and our efforts. We continue to use the dots to help us prioritize what we want to focus on and evaluate at this time. Participants help record the data and analyze it as well (typically using Excel for most of the ratings).

After using the dots to prioritize what we want to evaluate as a group at this time, we rate the selected activities on a 1 (low) to 10 (high) point scale. Then we engage in a dialogue about the ratings. We provide evidence for our ratings and learn how the reasons we give provide us with the most efficient path toward planning for the future and solving our problems. In other words, the reasons we give for our ratings guide us toward the solutions, e.g. if communication is a problem and specifically the lack of an agenda is listed we know we better start using agendas. It makes our problem solving more efficient and focused.

It is a team effort.

The entire group is engaged and productive. We are also sharing data and resources across agencies and grantees to make the entire group stronger and more effective. We have learned that if one group does well it makes us all look good. If one group falls behind it hurts the overall portrait or picture of what we are trying to accomplish.

The excitement of the day was infectious. The group is positive, reinforcing, and a pleasure to be with now and throughout the year.

homepage.mac.com/profdavidf/GCCARESsmoking-boys-031411.mp3http://web.me.com/profdavidf/GCCARESsmoking-boys-031411.mp3

Report for the Legislature - January 2011 Report

2011-01-10T10:02:00.000-08:00

Empowerment Evaluation Report - January 2011. The January 2011 MISRGO Empowerment Evaluation report was issued. It was prepared for MISRGO, ADH, and the State Legislature.

The front cover highlights one of MISRGO's achievements - securing smoke free parks in Arkansas. (The cover highlights Elroy Brown's effective work in Gosnell.)

The report is available at this site, just download it at your leisure. A few points drawn from the report are presented below:

Tobacco Use in Arkansas (with a national comparison)

There are over 480,000 individuals who smoke cigarettes in the State of Arkansas. That is approximately 22.3% of the population. Arkansas ranks 44^th among all states concerning the prevalence of cigarette smoking among adults. Smoking attributable mortality among adults is 45^th in the nation.[i]

Arkansas ranks 49^th among the states concerning youth consumption of tobacco.

Graphs from Centers for Disease Control and Prevention: Smoking & Tobacco Use 2010

MISRGO Grantee Counties (smoking prevalence rates)

Tobacco use in MISRGO grantee counties range from 16% to 37%. The average tobacco use is 24% across MISRGO serviced counties. The number of grantees working in the same counties ranges from 1 to 7, depending on the grantees’ specific capacities and the types of intervention or assistance needed.

Public Support (State-wide)

There is State-wide support for the use of public funds for MISRGO tobacco prevention programs. 99% of the 929 people surveyed throughout the state (who were familiar with MISRGO grantee programs), stated they thought that tobacco prevention and education programs were a good use of public money.

Similarly, 99% reported that they would recommend their local MISRGO-related tobacco prevention & education agency to their friends or family members if they smoked or were thinking about smoking.

Community members rated MISRGO-related tobacco prevention programs effective or very effective (94%).

The majority of community members surveyed thought MISRGO-related tobacco prevention agencies were effective or very effective concerning the following activities:

90% Sending out educational literature

92% Letting people know about the dangers of second hand smoke

91% Telling people about the QUIT Line

86% Educating retailers to not sell tobacco products to minors

84% Getting perimeter laws

86% Getting smoke free parks

79% Getting advertisers to reduce their advertising of tobacco products

90% Building coalitions with other anti-tobacco agencies

88% Helping minorities stop smoking (or not begin)

Highlighted Accomplishments

Tobacco-free policy

MISRGO grantees have been effective in creating tobacco free environments. According to the Centers for Disease Control and Prevention (Starr, et al, 2005), “Creating tobacco-free policies in workplaces, other public places, and homes and vehicles not only protects nonsmokers from involuntary exposure to toxins in tobacco smoke, but also may have the added benefit of reducing tobacco consumption by smokers and increasing the number of smokers who quit” (p. 147).

MISRGO grantees adhere to CDC recommendations in this regard. They have helped to establish 6 smoke free parks. They have also helped to convince local industry to adopt smoke free policies, including Nibco of Blytheville and American Greeting and Denso, both in Osceola. They have also persuaded 4 churches to adopt smoke-free policies, a well as Sweet Home Community Park and the Charles Strong Recreation Center.

MISRGO grantees have used local newspapers and media to extend the impact of their work. They have sparked the debate about tobacco use and prevention in the press. This is important in itself, as the CDC points out: “Experience shows that the education that occurs when a community debates whether it wants a local tobacco-free law – a debate that typically generates extensive media coverage – can greatly facilitate enforcement of the law, sometimes making it largely self-enforcing” (p. 147). MISRGO grantees have also used the press (including television[i], radio[ii], and newspapers) to highlight their accomplishments, another useful tool in educating the public to adhere to tobacco-free policies.

[i] 3 television spots aired concerning the ills of tobacco use.

[ii] 35,000 people were exposed to radio adds about tobacco consumption.

Quit Line

The Quit Line is an important tool in the fight against tobacco consumption. According to the CDC “Tobacco control programs need to foster the motivation to quit through policy changes and media campaigns and promote their quitline services” (2007, p. 41). MISRGO grantees have submitted 948 fax referrals to the Quitline this year. In total MISRGO grantees have submitted over 3,217 fax referrals to the Quitline, excluding grantees no longer funded.

Pledges & Successes

There are many steps needed to convince people not to begin smoking and to promise to stop smoking. MISRGO grantees have been effective in convincing minority members of the communty to take one of the first significant steps in this direction, pledging to quit. Although a pledge is not as compelling as a referral to the Quitline or larger community-based change, such as creating tobacco free environments, a pledge is an important stepping-stone. According to the CDC “Attempting to quit is an essential step in the process of becoming tobacco-free” (Starr, et al, 2005, p. 246). A pledge can help prevent or stop a person from using tobacco.

MISRGO grantees have convinced 1448 youth to pledge to be smoke free this year. They have also convinced 200 parents to pledge not to use tobacco in their homes or in their cars.

Currently funded MISRGO grantees have helped approximately 2,710 members of their communities throughout the state to stop using tobacco.

Education

The type of success described in this report is predicated on a number of factors. One of the most important contributing factors is education. Education may range from disseminating educational information at health fairs and in schools to measuring knowledge retention using pre- and post-tests. Most MISRGO educational interventions preclude extensive testing due to the nature of the informal community and home settings. However, there is no single educational intervention that works for everyone and thus multiple educational approaches are used to convince people not to use tobacco, including youth performances. According to the CDC, “State program involvement in community-level interventions should include: Supporting local strategies or efforts to educate the public and media not only about the health effects of tobacco use and exposure to secondhand smoke, but also about available cessation services” (2007, p. 23). Education in its various forms represents the backbone or foundation of any effective tobacco prevention program.

MISRGO grantee educational efforts resulted in:

66,766 receiving information about dangers of tobacco use

1,177 receiving information about Acts 8 & 13

2,400 receiving information about smoke-free ordinances

In addition, pregnant, low-income, homeless, and Hispanics or Latinos received tobacco prevention educational materials from MISRGO grantees.

318 pregnant, low-income & homeless people received tobacco prevention information

300 Hispanics received tobacco prevention information

Grantee Activities

MISRGO accomplishments are the result of many activities, monitoring tools, and administrative oversight and technical assistance.

An abbreviated list of grantee activities is provided below to highlight the wide-spectrum of work required to produce MISRGO outcomes.

· Working to establish ordinances in parks
· Promoting organizations in the community that review tobacco policies (e.g., restaurants and facilities)
· Helping to increase compliance rates among retailers
· Identifying disparities concerning sales to minors
· Providing tobacco education through surveys and media outreach
· Providing AHDC Workshops on ACTs 8 and 13
· Distributing educational materials & providing tobacco prevention sessions
· Promoting the Arkansas Tobacco Quitline
· Promoting the fax back referral program
· Helping to decrease tobacco advertising
· Helping to increase collaboration with other organizations (e.g., organizations serving minorities, clinics/physicians, schools) to promote a tobacco free message
· Helping to encourage smoke free households and cars
Training youth advocates or FAST teams in tobacco prevention

Monitoring Progress

MISRGO has implemented a useful tool to monitor grantee progress and performance. It is designed to help grantees establish their own goals and monitor their own efforts. The tool also enables MISRGO to determine if grantees are making satisfactory progress or if assistance is needed.

A review of grantee performance based on this data suggests that they are making satisfactory progress toward end-of-the-year goals. (See Appendix B for an example.)

Capacity Building

MISRGO has provided numerous workshops, lectures, and technical assistance sessions to improve grantee performance. (See Appendix C.) The MISRGO evaluation team has also provided lectures, demonstrations, exercises, and workshops to enhance evaluation capacity. Workshops focused on tools to conduct both process and outcome evaluations. Technology training was embedded in the evaluation training, ranging from online survey deployment to the use of digital photography to document and disseminate accomplishments. (See Fetterman and Wandersman, 2005 for a detailed discussion about evaluation capacity building.) A detailed presentation of MISRGO evaluation capacity building efforts is available on the web at: http://tobaccoprevention.blogspot.com.

Conclusion

MISRGO grantees have had a significant impact on minority populations in Arkansas. They have established a strong rapport with African American, Latino or Hispanic, and Asian communities in the State. This relationship has enabled them to disseminate tobacco prevention literature to youth and adults in these communities. It has also been instrumental in encouraging minorities in these communities to establish tobacco-free environments, particularly in public parks. MISRGO grantees have also been effective in encouraging community members to call the QUIT line.

Minority community members throughout the State support the use of public funds to support MISRGO-based tobacco prevention programs. This support is based in part on their perception of the grantees’ effectiveness in preventing and stopping people from using tobacco in their communities.

MISRGO grantees are making progress toward their stated goals and objectives. Continued support will only enhance their capacity to reduce tobacco consumption in minority communities throughout the State of Arkansas.

[See the MISRGO Empowerment Evaluation Report for additional details, January 2011].

2010-10-26T11:30:00.000-07:00

Online Survey Training. Dr. Beverly Tremain provided an online survey training webinar for the grantees on October 26, 2010.

She provided an overview of the free version of SurveyMonkey - a popular and user-friendly online survey program.

We started with creating a survey, including yes/no, multiple choice, and open-ended questions.
Then she explained how to use the Question Logic option or skipping option - to jump from one question to another string of questions (depending on your response to the initial question).

Bev continued the demonstration with a focus on data collection, primarily by sending the survey URL to everyone in the community or placing the URL on Facebook. She concluded the demonstration with suggestions for sharing the findings, including saving the bar charts as an html or pdf file. One member of the webinar suggested taking a computer screen snapshot and emailing that out to everyone. (It can be edited with PhotoShop if needed.)

Bev highly recommended piloting the survey before launching. She also reminded everyone that this tool is designed to create surveys and help with data collection and display. It still requires a separate and concerted effort to construct quality survey questions.

An added bonus of the demonstration is that it was conducted using free DimDim software, which facilitates group online demonstrations and webinars. It allows the presenter to share their desktop with all of the webinar participants. This allows the presenter to demonstrate the software, in this case SurveyMonkey, on her screen and simultaneously project it to the computer screens of every webinar participant.

Bev also mentioned another resource during the demonstration, which is called FreeConferenceCall.com. This enables you to conduct a conference call for fee for up to 20 people at a time.

Dr. Tremain plans on providing monthly training sessions to promote capacity building this year. Please contact her for upcoming training events for MISRGO grantees. Her email is: btremain@publichealthconsulting.net

“Using Evaluation to Inform Policy” - Dr. David M. Fetterman

2010-10-21T23:01:00.000-07:00

Dr. Fetterman, Fetterman & Associates, was our MISRGO University of Arkansas at Pine Bluff guest speaker on October 14, 2010. The presentation was made at the American Cancer Society, 901 N. University Avenue, Little, Rock, Arkansas.

Dr. Fetterman presented 5 case examples to illustrate how ethnographic and empowerment evaluations have been used to influence policy and policy decision-making. The case examples included:

Arkansas Department of Education: Schools in Academic Distress
Hewlett-Packard $15 Million Digital Village Project
MISRGO: Tobacco Prevention
Stanford University School of Medicine
U.S. Department of Education: Dropout and Gifted Programs

The examples demonstrated how to use evaluation to inform policy on the local, state, and national levels.

Small groups were organized to:

1) discuss what they had heard during the presentation

2) share their own experiences linking evaluation to policy; and

3) apply what they had learned to their own experience and generate new ideas for future work.

The presentation concluded with a demonstration of useful technological tools to facilitate evaluations, specifically designed to inform policy decision-making.

Dr. Fetterman’s slide presentation is available here.

October 13, 2010 MISRGO Empowerment Evaluation Workshop

2010-10-21T13:33:00.000-07:00

Empowerment Evaluation Workshop. This MISRGO empowerment evaluation workshop focused on: documenting actual performance in comparison to the baseline, goals, and benchmarks. We critiqued each other’s five year plans, using this model. Then we applied the same approach to quarterly reporting for this year. By comparing actual performance with the benchmarks we were able to determine if we were meeting our goals or if we needed to fortify our efforts. This approach helps us monitor change over time. (Slides from the presentation are available here.)

Quarterly Measurements of Performance:

Benchmark and Actual Performance Comparisons

Our evaluation team, Dr. Beverley Tremain, Ms. Linda Delaney, and Dr. Fetterman provided the grantees with individualized assistance as they refined their first quarterly report.

Bev (above) and Linda (below) facilitating
the discussion and critique

As we volunteered our work for a group critique, it became clear that this was going to be a learning process for all of us.

Grantees Actively Listening

Key lessons learned during the workshop included:

separating out goals, instead of clustering them in one sentence, in order to track them more effectively
ensuring that activities are related to the goal
remembering to record actual performance data
stating benchmarks for each year and each quarter; (as opposed to only one year and making the reader assume that the same benchmarks apply to each quarter or year)
using actual numbers verses percentages; stating the goal in terms of a number
completing the calculations for the reader; (instead of stating that the goal is to improve by 20% and making the reader go back to the baseline in a previous report to calculate 20% of the baseline figure)
similarly, avoiding ambiguous statements, e.g. will increase, but not specifying by how much or from what baseline; will improve, without stating how they will improve, etc.
using the matrix instrument to facilitate record keeping, e.g. baseline, goals, benchmarks, and actual performance
creating bar charts that match the matrix data or figures

Our discussions were animated and productive. We shared strategies to accomplish goals we had in common.

Light Moments and Serious Moments
During Our Workshop Exchange and Critique Session

We also made a “good faith” commitment to focus on fewer things that we all had in common in order to have a greater “dose” effect. Even the breaks were productive, sharing insights into what works and what has not worked with each other. MISRGO staff were viewed as supportive and helpful.

Bev, Ruthie, Linda, and Marian
at the Table During a Break

We invited the Arkansas Department of Health to share their knowledge and resources with us. One of their valuable tobacco related data and report web pages is available here.

Sherry Johnson and
ADH Colleagues

They recommended Center for Disease Control reports including:

We concluded the workshop with our own local “best practices” exchange. We mixed evidenced-based practices with practice-based evidence.

Dr. Fetterman
Concluding the Workshop

2010 MISRGO Annual Evaluation Report

2010-08-26T00:26:00.000-07:00

2010 Annual Evaluation Report. Individual grantees are making significant progress concerning the implementation of their programs.

The annual report was presented to the MISRGO Advisory Board on August 24, 2010. (See the report for detailed findings and a list of accomplishments.) Officials in attendance from the Arkansas Department of Health included:

Dr. Paul Halverson, Director and State Health Officer
Dr. Nathaniel Smith, Deputy Director for Public Health Programs and State Epidemiologist
Dr. Carolyn Dresler, Program Director of the Tobacco Prevention and Cessation Program
Ms. Sherry Johnson, Section Chief, Surveillance and Evaluation

University of Arkansas at Pine Bluff officials included:

Dr. Mary Benjamin, Vice Chancellor for Academic Affairs
Dr. Calvin Johnson, Dean, School of Education
Dr. Marian Evans, Coordinator, MISRGO (and staff)

Advantage Communications personnel were also present, including:

Ms. E'van Steele, Executive Vice President and Whitney Albert, Account Executive

Dr. David Fetterman, President and CEO of Fetterman & Associates, presented the evaluation findings. The highlights of the evaluation are presented below.

Highlights

All of the grantees are operating within 6 major MISRGO intervention areas, including:

Promoting Quitting Among Youth and Adults
Identifying and Eliminating Tobacco-Related Disparities Among Population Groups
Eliminating Exposure to Second Hand Smoke
Preventing Initiation Among Youth and Adults
Creating Tobacco Free Environments
Reducing Youth Access to Tobacco

The number of grantees working in each area are displayed below:

There were 5 themes that were identified across the intervention areas, including:

One of the themes, tobacco-free policy, provides an insight into the number of grantee accomplishments documented this year:

Recommendations

The evaluation recommendations are presented below:

Evaluation Dashboard
The MISRGO empowerment evaluation dashboard was introduced to the Advisory Board during this session. The key elements of the dashboard include: goals, benchmarks, actual performance, and baseline.

Metaphor

A weight lifting metaphor was used in an initial training session with grantees. It resonated with grantees, Advisory Board Members, and many of the Arkansas Department of Health officials.

Measurement and Monitoring

The report also described the plan for the future, concerning the use of evaluation to help grantees accomplish their objectives and to help MISRGO monitor grantee progress. The bar chart below illustrates the user-friendly format adopted for the future, e.g. baseline, benchmarks compared with actual performance, and the goal. This format helps to keep the process transparent. It enables anyone to see where individual grantees are at any time.

The evaluation report was well received and will be used to guide the next MISRGO grantee evaluation training session in October 2010. (The presentation slides are also available.)

Clearing the Air in Communities of Color April 28-29, 2010

2010-05-05T11:26:00.000-07:00

7th Annual Conference at UAPB. The annual conference highlighted MISRGO grantees' work.

Drs. Johnson, Benjamin, and Evans welcomed the participants and invited them to learn from each other.

The grantees presented many of their approaches to reduce tobacco consumption in their counties. Billboards were used to send the message with a referral to the QUIT line.

Speaking before the Senate was another tool used by MISRGO grantees.

Another approach involves organizing youth. In this case, they were trained on how to:

pick up cigarette butts off the ground (from just outside the city council building)
present the cigarette butts to the city council (during a scheduled session)
explain how the chemicals in the cigarettes are leaching into the soil
recommend swift action, in this case recommending smoke free parks policies

Grantee Posters
(highlighted many of their activities)

The conference also provided grantees with an opportunity to consult with Dr. Fetterman about specific evaluation issues and concerns. They also discussed the new evaluation plans focusing on establishing baselines, benchmarks, and goals.

Additional details about the conference, including digital video of the proceedings are available through MISRGO and UAPB TV.

MISRGO Evaluation Workshop - March 25, 2010

2010-03-25T08:40:00.000-07:00

Evaluation Goals, Benchmarks, Actual (current status), and Baselines. We agreed that evaluation data was a powerful way to "tell your story."

It can be used to monitor and track our progress. It can be a signal that we need help and that we should contact each other first, MISRGO second, and Dr. Fetterman third. The rationale for this is that we want to foster collegial sharing and cooperation among peers first.

We used a weight lifting analogy to highlight the tools we use in evaluation: establish a baseline (how much can you lift now), set a goal (how many pounds do you want to be able to lift in 6 months), set benchmarks (the weight you want to be able to lift by the end of month one, two, three, four, five, and six). Measurement - record the amount of weight you can lift at each interval and compare it with the benchmarks you set to determine if you are making progress or if it is a signal that you need help.

We agreed on a common tool or matrix for collecting and keeping track of our data. It included: goals, benchmarks, actual or current status, and baseline data. It also specified dates for each benchmark as well as numbers (in this case of tobacco-free environments).

We also demonstrated how easily it is to transform this data into bar charts and other easily accessible (and highly visual) formats. We highlighted a comparison of actual performance or accomplishments with our benchmarks to help us determine if we were still on track.

This was a working session and we completed a number of exercises to prepare ourselves for applying these tools to our work.

We practiced presenting our results in front of each other and providing a diplomatic, but honest critique of our work and presentations.

We reminded each other of how important it is to keep track of the number of people their agencies have helped to stop consuming tobacco. We discussed how their was power in numbers since we began merging our data to demonstrate our effectiveness as a MISRGO group across the State (as compared with our individual efforts).

This discussion helped to plant the seed for future collaborations and consolidations. We discussed the potential of merging our data concerning tobacco free environments. We created a model for what that might look like - which would be, once again, more persuasive and impressive that our individual efforts. We also discussed the possibility of doing fewer things but having more activities in common to create a "larger dose effect".

We critiqued our latest reports with the aim of improving them and ensuring that we all had comparable data.

The workshop concluded with a demonstration of our web-based resources including our MISRGO Empowerment Evaluation Blog and Collaborative Web Site. (We added to both the blog and collaborative web site during the workshop to demonstrate how user-friendly both tools are and to document our work while we were in the middle of conducting the training.)

Dr. Fetterman's slides are available on our blog and collaborative web site.

MISRGO Evaluation - February 2010 Progress Report

2010-02-03T12:19:00.002-08:00

MISRGO Evaluation Progress Report. The 2010 Progress Report has been issued. It is currently being reviewed by the MISRGO Advisory Board (February 3, 2010).

The report highlights MISRGO's and the grantees' efforts to launch a new evaluation monitoring and reporting system. Each grantee is now required to establish a baseline, specify program goals, and develop realistic benchmarks (to help guide and monitor program performance).

It represents a systematic approach to monitoring and evaluating grantee performance, building on the existing framework.

KEY FINDINGS include:

1. All grantees are operating within the 4 MISRGO domains, including:

Creating Tobacco-Free Environments
Reducing Youth Access to Tobacco
Promoting Quitting Among Youth & Adults
Identifying and Eliminating Tobacc0-Related Disparities Among Population Groups

2. Grantees have established baselines, specific goals, and annual (as well as quarterly) benchmarks
For example, the Mississippi County Coalition for Tobacco-Free Arkansas (MCCTFA) has a goal of increasing the number of public buildings with smoke-free policies by 15% (22 of 157) by 2015. They divided their long-term goal of 22 by 5 years. This resulted in equal benchmarks of 4.5 each year. (Approximately 5% of public buildings in their area have already been identified as having smoke-free policies, representing their baseline).

3. Many grantees are using the Evaluation Monitoring System (to record the number of people they help to stop using tobacco and translate that figure into dollars saved in terms of excess medical expenses)

4. Many grantees are using the Visual Alert System (to plan their activities and compare their plans with their actual performance)

5. Although, most grantees and evaluators have made a conscientious effort to create accurate baselines, realistic benchmarks, and meaningful goals, there is considerable variation among grantees concerning the use of the new evaluation monitoring and reporting tools.

6. In addition, grantees are currently implementing a wide-variety of activities. At one point consideration might be given to consolidating and focusing on fewer activities held in common by the group. This would result in a "bigger dose effect" - with more grantees conducting the same type of activities across the State.

7. The data also suggest that additional training is merited (and is planned for the March 25, 2010 evaluation training workshop). It will address issues associated with the variability or quality of the data.

The report is available online at 2010 Progress Report or in the right hand column of resources and MISRGO evaluation documents.

MISRGO Dissemination "Jackpot" - 2009

2009-12-29T13:44:00.000-08:00

Dissemination "Jackpot" 2009. One of the charges of the evaluation is dissemination - getting the word out about MISRGO work. An effort is made throughout the year to disseminate the findings and the tools used in the evaluation. However, November and December 2009 represented rare opportunities for maximum exposure and dissemination of MISRGO work.

In addition to highlighting the MISRGO evaluation in a recent professional association training workshop (in the picture above), the three key dissemination outlets were:

Professional Association Annual Meeting Presentation
Scholarly Book Publication
World-wide Recognition Notice About Scholarly Article Publication

Professional Association Presentation

Dr. Fetterman presented many of the technological tools employed in the MISRGO empowerment evaluation at the American Evaluation Association (November 2009).

The presentation was called:

Empowerment Evaluation: Technological Tools of the Trade

The following tools were discussed:

Wordle. Wordle is a powerful qualitative data analysis breakthrough. It is a form of content analysis that produces a visual image of the frequency of key words used in a text. The Wordle image below highlights comments about a MISRGO evaluation workshop. This tool, among others, has been introduced to the MISRGO grantees and evaluators (to help them gain an additional insight into the effectiveness of their programs). It is also used to communicate evaluation findings in a user-friendly and visually appealing manner. In addition, the use of MISRGO evaluation online surveys was discussed.

Digital Photography. The use of digital photography was also explored, using MISRGO case examples. The examples ranged from, a Clearing the Air Conference in Communities of Color to a MISRGO evaluation workshop.

Blogs. Blogs, like this one, were also shared to emphasize the principles of transparency and openness associated with the MISRGO evaluation.

Scholarly Book

Dr. Fetterman published the 3rd edition of Ethnography: Step by Step. This updated version of his popular methods book highlighted some of the MISRGO evaluation work. (The timing of the book revision represented a window of opportunity to highlight MISRGO evaluation activity.)

The examples highlighted in his book include: the Tobacco Prevention (MISRGO) blog, the MISRGO Empowerment Evaluation Collaborative Web Page, and the use of MISRGO online surveys.

World-wide Recognition About Scholarly Article

Sometimes exposure is a byproduct of a publication or presentation, instead of the presentation or publication itself. In this case, the American Evaluation Association recognized the popularity of Drs. Fetterman and Wandersman's 2007 article titled, Empowerment Evaluation: Yesterday, Today, and Tomorrow.

According to the American Evaluation Association, their article was the most accessed American Journal of Evaluation online.

Since the article highlighted MISRGO evaluation work, it not only documented a very effective dissemination strategy, it also served as a "marketing tool" to enhance exposure.

[Reproduced from the American Journal of Evaluation,
Vol. 28, No. 2, June 2007, pp. 179-198]

MISRGO Mapping Project

2009-10-13T16:56:00.001-07:00

Grantee Map. We have a shared map called the:

Tobacco Prevention in Arkansas - MISRGO Map.

You can view and edit this map (if you are a MISRGO tobacco prevention grantee or evaluators) at:

http://maps.google.com/maps/ms?ie=UTF8&hl=en&oe=UTF8&msa=0&msid=118091722891591549115.000475d6af7a195c624e2

Inspired by Sarah Hammer's map work - in which she highlights the smoke free places in her area - and Dr. Tarek's workshop on using maps, a map of all of us was created. In addition to letting us know where we are across the State, it provides us with each others' contact information to facilitate sharing and assistance.

Take a look and feel free to edit any of the information on the maps - make any corrections and additions. Right now, however, this information will help us help each other as we work on our baseline evaluation efforts.

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Note 1: For a tutorial on creating your own map of locations of interest go to the following site:
http://maps.google.com/support/bin/answer.py?hl=en&answer=67842

Click here for an abbreviated tutorial with computer screen snapshots

You may want to map out tobacco free areas, tobacco billboards, stores with multiple violations, and so on in your county. This is a very useful tool. You can map out resources in the community including smoke cessation programs, as well as negative influences concerning the use of tobacco by youth.

Note 2: You should already be able to edit this map. However, if not be sure to reply to the posting that was sent to you on October 13, 2009 and accept Dr. Fetterman's invitation to become a collaborator on this map.

If you don't have a Google account, you can create one at
http://www.google.com/accounts/NewAccount?reqemail=.

Baseline, Benchmark, and Goals Workshop

2009-09-26T16:10:00.001-07:00

Baseline, Benchmark, and Goals Workshop. Our September 24, 2009 evaluation workshop received rave reviews. A few grantee and evaluator comments included:

"It wet our appetite for more".
"This is precisely what we needed."
"Timely."
"Clear."

MISRGO staff also valued the day: "We've gotten good reviews about the workshop. Thanks so much for all you do!!!"

Dean Johnson

Dean Johnson took time out of his busy schedule to join us and help strategize about next steps in the evaluation.

Dean Calvin Johnson

Three Evaluation Speakers/Facilitators

We had three speakers/facilitators for the day. The day began with Dean Stewart Donaldson from Claremont Graduate University. He facilitated a session entitled:

Practical Program Evaluation: A Program Theory Approach

He highlighted the role of program theory. It is used to make sure everyone agrees with the logic of their program in an evaluation.For example, if you did X (connected them with the Quitline and heard back from them) it would be likely to produce Z (help people stop smoking).
He conducted an exercise where everyone was asked to evaluate the room. It was practical and down-to-earth and emphasized the role of an individual's perspective in evaluation.

Grantee and Evaluator Presenting Insights Based on Dr. Donaldson's Exercise

The second facilitator was Dr. Tarek Azzam, also from Claremont. He presented the tech session. It was titled:

Using Technology to Enhance Applied Research & Evaluation

He highlighted the use of technology to conduct evaluations and to more effectively communicate about evaluation findings.
Tarek covered the use of: GIS, online surveys, and various qualitative data display tools.

Use of GIS to Identify Problem Issues and Areas: Tobacco Billboards & Income

Qualitative Data Display: Frequency of Words Used Visually Represented in a Word Cloud

The third speaker was Dr. David Fetterman, from Fetterman & Associates. He built on the first two speakers ideas and techniques. Dr. Donaldson focused on program theory - does the program make sense as designed. Dr. Tarek highlighted the use of technology to display evaluation findings.

Dr. Fetterman highlighted a visually powerful way of highlighting grantee program baselines, benchmarks, and goals. This will be the focus of next year's reporting - to help us all keep track of our progress throughout the year and for the next 5 years. The title of his session was:

Baseline, Benchmarks, and the Visual Alert System

The MCCTFA Baseline and Quarterly Evaluation Assessment Form was used as a case example. Dr. Fetterman explained how to pull the baseline figure from the narrative. Then he demonstrated how to identify the goal, for example, to increase the number of public buildings with smoke free policies by 15%. From there it was simply a matter of pulling the current year objective, 5%, and establishing benchmarks for the next 5 years. The slides for his presentation are available here online. (Select Baseline.)

Dr. Fetterman presented a translation of the evaluation language we are using at this stage of the empowerment evaluation to help make the process jargon free:

Normal Question (Evaluation Terms)

Where are you now? (Baseline)
Where do you want to go? (Objective or Goal)
How do you plan to get there? (Activities or Strategies)
How do you know where you are along the way? Are we there yet? (Benchmarks)
How do you know when you have arrived at your destination? (Comparison of baseline with long-term objective or outcome)

Here is a bar chart that the group used as an example of how to visually report their own work.

Bar Chart: Baseline, Benchmarks, and Goal

Dr. Fetterman providing one-on-one technical assistance

Dr. Fetterman, Ms. Evans, and Ms. Delaney went from table to table, helping grantees and evaluators link activities to their baselines and benchmarks. It was a rewarding and productive day.

House & Senate Health Services Subcommittees Report

2009-08-06T14:18:00.000-07:00

House & Senate Health Services Committee report. MISRGO presented the "MISRGO Best Practices & Outcomes" report to the House & Senate Health Services Committee of the House and Senate Public Health, Welfare, and Labor Committees on August 4, 2009. (It is available online here and on the right column.)

The report summarizes the best practices and outcomes of the Minority Initiative Sub-Recipient Grant Office's (MISRGO) tobacco prevention efforts. It begins with:

$94 Million Saved in Excess Medical Costs

MISRGO grantees have saved the State of Arkansas $94,944,281.96 in excess medical costs. This is a significant outcome by any standard. It is also an increase from previous years.

The formula for calculating this savings is straightforward. The figures used in the formula come from the Arkansas Department of Health, Campaign for Tobacco-Free Kids, Centers for Disease Control and Prevention, and the U.S. Census Bureau.

The formula is the number of people who stop smoking, times the per person cost of smoking in the State of Arkansas. Then each grantee’s total is added together to produce the final reported figure. The dollars saved are in terms of excess medical costs saved the State.

A table of the formula is presented below:

A complex formula is used to provide precise data by county. However, a simpler approach can be used to calculate the dollars saved. The overall State average can be used to calculate the excess medical costs saved. The formula is based on the following information:

1. $812 million annual health care costs in Arkansas directly caused by smoking

Source: Campaign for Tobacco-Free Kids (The New York Times Magazine called the Campaign the "the country's leading anti-tobacco lobby.")

2. 4900 people who die each year from their own smoking in Arkansas

Source: Campaign for Tobacco-Free Kids

3. $812,000,000 divided by 4900 = $165,714 approximate per person cost

4. $165,714 times 572 (the number of people who have quit smoking as a result of grantee efforts) = $94,788,408 saved in terms of excess medical cost

STATE-WIDE SUPPORT FOR MISRGO TOBACCO PREVENTION PROGRAMS

A state-wide survey was conducted to solicit community views of MISRGO tobacco prevention programs. The most significant survey finding was the overwhelmingly positive value placed on these tobacco prevention programs in communities throughout the State. When asked if:

“Tobacco prevention & education programs are a good use of public money?”

The response from an overwhelming majority of Arkansans sampled was positive (98% answered yes).

In addition, community members believe grantees’ tobacco and prevention efforts have been effective in PREVENTING people from using tobacco and in getting people to STOP using tobacco.

CDC RECOMMENDED PROGRAM ACTIVITIES

MISRGO grantee programs use CDC recommended tobacco prevention activities. These activities contribute to the outcome data reported earlier, e.g. reduction in the number of smokers in the State and the millions saved in terms of excess medical expenses. A series of bar charts illustrate how MISRGO monitors and supports these efforts across programs and grantees. This tool has also been useful for grantees, placing their activities in comparison with other grantees. It has influenced them to "join in" the collective effort, increasing the "dose" effect with more grantees working on the same tasks and concerns.

The bar charts below provide an insight into the activities held in common across grantees, eg. minority initiatives, coalition building, Quitline promotion, newspaper articles, radio and TV advertisements, and educational materials.

Detailed findings of program activities are presented in the following report: Introducing the Visual Alert System: A Graphical Analysis of the MISRGO Mid-Year Report (Fetterman, Fitch, and Delaney, 2008).

TRAINING

Training is a critical component of MISRGO grantee effectiveness. Grantees and evaluators continually learn more about effective tobacco prevention practices. They also receive evaluation training in order to monitor and assess their effectiveness. (See earlier postings on this site for more information about the evaluation training grantees and evaluators receive to support their efforts.)

The pictures below highlight grantees and evaluators engaged in self-evaluation exercises and Dr. Fetterman providing training in interviewing and observation skills.

POLITICAL SUPPORT FOR THE PROGRAM AND EVALUATION TRAINING

Political support for the program is extensive including:

University of Arkansas academic officials (Drs. Johnson and Benjamin)
Senators (Henry "Hank" Wilkins, IV and Tracey Steele)
Representatives (Stephanie Flowers and Earnest Brown)

MISRGO has received a citation from the State of Arkansas for its evaluation training.

DISSEMINATION OF WORK

One of MISRGO's mandates is to disseminate promising practices, evaluation findings, and program outcomes. MISRGO has made presentations at professional meetings such as the following:

In addition, MISRGO staff and colleagues have published articles and books concerning "lessons learned."
CONCLUSION

The report summarizes many of MISRGO's tobacco prevention best practices and outcomes. A few of these practices and outcomes include:

94 million saved in excess medical costs between 2005-present
94% engaged in minority initiatives to prevent tobacco consumption
94% using educational materials as part of their tobacco prevention campaigns
88% engaged in coalition building to combat tobacco use
76% promoting the use of Quitline
76% using radio and TV ads to combat tobacco use

MISRGO has provided strong leadership and technical assistance and grantees and evaluators remain vibrant and committed. The prognosis is good. MISRGO and its grantees are well positioned to continue to have a positive impact on the minority community in Arkansas in the foreseeable future.

2008-2009 Annual MISRGO Evaluation Report

2009-07-11T20:24:00.000-07:00

2008-2009 Annual MISRGO Evaluation Report. The report was issued July 11, 2009. The 2008-2009 MISRGO Evaluation Annual Report highlights many of the accomplishments of the year. They include a more focused view of evaluation objectives and the power of consolidating grantee interests and activities.

The Evaluation Monitoring System and Visual Alert System are tools that have been successfully employed in the field and sparked an interest in common data collection and reporting efforts in the future.

Ironically, the evaluation tools themselves have helped unite a diverse group of grantees and evaluators and initiate a dialogue about the power of speaking with a single united evaluation voice.

This report also discusses the range of evaluation training provided to build evaluation capacity among grantees and evaluators, ranging from formal workshops and exercises to informal one-on-one critiques and consultations. The aim of these training exercises is to enhance the quality of the current tobacco prevention work. It is also designed to enhance the quality of tobacco prevention evaluators in the State, building a stronger and more sustainable evaluation workforce for the future.

April 30, 2009 - EMS Online Collaborative Web Site Workshop

2009-06-07T17:01:00.000-07:00

Evaluation Monitoring System Collaborative Web Site Training Workshop.

We focused on how to calculate the excess medical cost saved by each grantee.

Then we demonstrated how to post the bar charts, pictures, text, and related information on the Collaborative Web Site. This web site allows each of us to create and update our own web page on our shared site. The workshop was a useful training opportunity for new grantees/evaluators and a useful refresher for those who already generate the Evaluation Monitoring System information and use the collaborative site. (The first page summarizes our collective data and can be seen in the top left corner. Collectively we have saved the State of Arkansas over $94 million in excess medical costs (by helping people stop using tobacco).

Below is the Contact Information page of our collaborative web site, highlighting the individual accomplishments of our grantees (as compared with our collective data in the top left corner of this blog).

Dr. Calvin Johnson, Dean of the School of Education at the University of Arkansas at Pine Bluff, welcomed the grantees and evaluators at the workshop (see below).

Evaluators and grantees contemplated the potential of this site and the power of combining our data instead of reporting only by each individual grantee.

We had a guest speaker during the workshop training. Oluwakemi Talabi (Kemi) is a Tobacco Cessation Program Analyst from the Arkansas Department of Health. She provided us with an update on the type of information we could expect to receive from the new vendors operating the Quitline. It was impressive and and excellent opportunity to link up with the Quitline contact people in the Tobacco Prevention and Cessation Program in the Arkansas Department of Health. (See the contact information cards below.)

This is Kemi speaking to the group (below). She was gracious and informative.

The Clearing the Air in Communities of Color Conference was held concurrently with the evaluation training workshop. We were all able to attend parts of the conference including the skits (see below) which were very effective tools to reach minority youth.

After the skits concluded, many of the grantees and evaluators reconvened and discussed next steps with Dr. Fetterman. Dr. Fetterman explained that he would be providing his comments on each of the mid-year reports (which he has already completed). In addition, he would continue to provide additional technicals assistance until the Evaluation Monitoring System and Visual Alert System were used consistently by each grantee. Dr. Fetterman also agreed to continue to provide additional one-on-one assistance concerning posting on the Collaborative Web Site (so that the information would be up-to-date and visible to the public).

Summer Evaluation Workshop

2008-08-01T00:28:00.000-07:00

Evaluation Workshop. The free summer evaluation workshop was considered a tremendous success, according to participants, workshop facilitators, and legislative officials. Inquiries came as from as Ethiopia and Liberia. However, most of the interest came from Arkansas and the mid-west. There were approximately 100 participants.

The workshop began with welcome messages from University of Arkansas at Pine Bluff academic officials, including Drs. Johnson and Benjamin. Mayor Redus also welcomed workshop participants.

Dr. Fetterman provided an outline of the workshop, including plenary speakers and workshop facilitators. He also highlighted the local Arkansas evaluation group's meeting during the workshop.

Plenary speakers included: Senator Tracey Steele, Representative Flowers, Kevin Collins (Centers for Disease Control), Charles Gasper, Missouri Foundation for Health, and Charlotte Lewellen-Williams (University of Arkansas Clinton School of Public Service).

The House of Representatives presented a Citation for our work during the workshop:

The topics and speakers were as follows:

Dr. Stewart Donaldson - Practical Program Evaluation
Dr. Tarek Azzam - Technological Tools
Dr. David Fetterman - Empowerment Evaluation
Dr. Rosalie Torres - Interviewing Skills
Dr. Abraham Wandersman - Getting to Outcomes

The Arkansas Group of Evaluators held its inaugural meeting during the workshop (at night). The chair, Linda Delaney, orchestrated the evening meeting and plans to continue to build this organization (an official local affiliate of the American Evaluation Association).

Center for Disease Control and Prevention literature was made available to participants free of charge, including:

A detailed portrait of the workshop is provided at the Arkansas Evaluation Center blog.

MISRGO Annual Evaluation Report 2008

2008-06-20T16:25:00.000-07:00

Report. The Annual Report provides a brief summary of evaluation activities and accomplishments throughout the year. The most significant achievement by the group is presented in their Evaluation Monitoring System. To-date, the group has save the state $94,944,281.96 in excess medical costs. This is an increase of approximately $10,188,113.56 since last year. In addition, the Visual Alert System is being implemented this year. It involves the use of bar charts to graphically illustrate what kinds of activities the grantees are engaged in, as well as the frequency of these activities - across grantees. Additional accomplishments include workshop training, blogs, and publications. Plans for the Summer Evaluation Workshop (which is part of MISRGO training efforts) have been extensive, tailoring national talents to local needs. It has attracted prospective participants from as far away as Liberia and Ethiopia, as well as throughout the United States. It will be held July 10-11, 2008. In addition to these efforts, a certificate and masters program in evaluation is being planned. For a more detailed account of evaluation efforts see the complete report or contact Dr. Fetterman.

Clearing the Air in Communities of Color 2008

2008-06-20T16:10:00.000-07:00

Clearing the Air. This year's conference began with Bishop Miles and a talk about "A Vision of a Smokefree Society." The theme was maintained throughout the conference with Kevin Collins (CDC) emphasizing capacity building and Onjewel Smith discussing tobacco industry techniques. Success stories were shared including presentations by Elroy Brown, Larry Owens, and Carolyn Yarbrough. Dr. David Fetterman provided a status update and workshop about MISRGO's empowerment evaluation efforts. He highlighted both the Evaluation Monitoring System and the Visual Alert Systems. Other speakers included: Lloyd Franklin (law enforcement), Ruby Trotter (youth), Misty Paschall (parent), and Bob Mathis (state government). Special presentations were made by: Wilma Houston (Asian Pacific), J.R. "Pinky" Few (Rural Community), Keisha Grigsby (African American Community), and Ruben Arana (Hispanic Community), as well as Advantage Communications. Dr. Benjamin helped to open and close the conference with the AR State PRIDE Youth Performance.

Spring Evaluation Training Workshop - April 24, 2008

2008-05-01T14:36:00.000-07:00

Spring Evaluation Training Workshop

Grantees and evaluators came together on April 24, 2008 to participate in evaluation training. Bringing both of them together ensures that they are on the same wave-length about data needs, roles, and relationships. (A copy of the slides used to guide the training day are available on this blog.)

The training was geared toward a new monitoring tool – the Visual Alert System. (This tool is described in the Visual Alert System posting below this one.)

This tool helps monitor grantee progress across programs, providing a readily available update upon demand. For example, bar charts are used to visually document how many grantees are using the Quitline.In addition, grantees and evaluators agreed that the same bar chart format could be used to document individual grantee progress. For example, a grantee would have:

The first bar chart would document their baseline of 4 news articles printed so far this year.
The second bar chart would document their aspirations – a benchmark number such as 10 new news articles by the end of the year.
A third bar chart would document the actual number of articles they had published that year, which might be 8.
A fourth chart would compare the first baseline bar chart of 4 with a final end of the year bar chart of 8 – documenting their progress throughout the year.

These bar charts could be forwarded to MISRGO and Dr. Fetterman and Ms. Delaney to document progress across grantees based on activities held in common, such as the Quitline or newspaper articles.

Dr. Johnson, Dean of the School of Education, was present to introduce new points and reinforce existing recommendations, including the need include activities directly related to tobacco prevention efforts even if the activities were not originally listed in the plan. In addition, he reminded grantees of the need to seek external funding.

The Visual Alert System provided an insight into patterns across grantees, alerting grantees and evaluators to successes that should be reinforced and continued, as well as areas meriting attention. The issue of whether we were implementing the “right” or most effective activities was discussed.

The Center for Disease Control document concerning Best Practices helped guide this part of the discussion. A summary of best practices was distributed during the workshop. CDC’s complete document is available on line.
One of the Visual Alert System findings initially suggested that few grantees were applying CDC Best Practices. However, after some discussion it was clear that grantees were in fact applying CDC Best Practices without knowing it, e.g. Quitline, coalition building, and so on. This finding and discussion highlighted the importance of sharing CDC documentation and engaging in a dialogue about what was being implemented in practice.

The Evaluation Monitoring System was also emphasized during the workshop, highlighting the importance of the group continuing to work together as a group. This system has demonstrated how much money they have saved as a unified tobacco prevention coalition - $84 million.

The workshop concluded with a few announcements including the:

establishment of the Arkansas Evaluation Center
development of an evaluation certificate program (and master’s program) at UAPB
upcoming Arkansas Summer Evaluation Institute Workshop – July 10th and 11th

The group responded enthusiastically to the announcements.

Dr. Fetterman pointed to the LCD projection to let them know the details associated with the Summer Evaluation Workshop.

The Summer Evaluation Workshop will include:

Senator Tracey Steele
CDC’s Health Education Specialist Kevin Collins
Dr. Stuart Donaldson, Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University - Introduction to Practical Program Evaluation (Program Theory)
Dr. Tarek Azzam, Senior Research Associate, School of Behavioral and Organizational Sciences, Claremont Graduate University - Introduction to Technology and Evaluation: Focus on the use of Geographic Information Systems and Online Surveys
Dr. Rosalie Torres, Torres Consulting Group - Interviewing Skills for Evaluators
Dr. Fetterman, Director of Evaluation, School of Medicine, Stanford University and Co-Director, Arkansas Evaluation Center, School of Education, UAPB
Mr. Charles Gasper, Director of Evaluation, Missouri Foundation for Health - A Foundation Perspective about the Significance of Program Evaluation
Dr. Nacaman Williams, Walton Foundation – A Foundation Perspective about the Significance of Program Evaluation

The Summer Evaluation Workshop Program and Schedule is listed in under Useful Links on the left column of the blog. For your convenience, the program is also available by selecting this Summer Workshop Program 2008 link.

The Spring Evaluation Workshop concluded with an announcement about the new Arkansas Group of Evaluators. AGES is an official affiliate of the American Evaluation Association.

Their first official meeting as an affiliate will be on July 10th in the evening at UAPB. It will be held during the Summer Evaluation Workshop. Ms. Linda Delaney is the chair of the State-wide evaluation group.

Visual Alert System

2008-03-28T21:51:00.000-07:00

Visual Alert System. We have developed a system that enables us to monitor our progress quickly. We are using a simple visual tool - bar charts. The MISRGO Mid-Year and Annual Evaluation Reports can be difficult to navigate. There are 18 reports covering varying topics ranging from coalition building to promoting the use of the Quitline. It is a time consuming and laborious task to compare grantee efforts throughout the State by reading the text in each report alone. We have applied this tool to the current Mid-Year Evaluation Reports (see Visual Alert System listed on the right hand column of this blog). The bar charts enable us to quickly scan all of the reports and determine how many of us are building coalitions, using the Evaluation Monitoring System, working with churches to promote tobacco prevention activities, and so on. Grantees can look at the bar charts and decide to shift their efforts to strengthen the groups' efforts as a collective. Policymakers and legislators can also see where we are putting our emphasis to meet our goals. We will provide training on the use of this tool during our upcoming training session (April 24, 2008). This tool among others will be highlighted during our Arkansas Summer Evaluation Institute Workshop, which will be held at the University of Arkansas at Pine Bluff on July 9-10, 2008.

Evaluation Technology Workshop Sept 27, 2007

2007-09-28T16:45:00.000-07:00

Evaluation Technology Workshop. MISRGO provided a technology training day at the University of Arkansas at Pine Bluff. The training was designed to familiarize grantees and evaluators with a variety of web-based tools that will help them with their programs and assessments of their programs.

Dr. Fetterman was invited to facilitate the session.

He provided an overview of useful tools, including online surveys, blogs, digital photography, picture file sharing, internet telephony (Skype), online documents and spreadsheets, and videoconferencing on the net.

Grantees and evaluators will be using the online spreadsheet to keep track of the Evaluation Monitoring System data. The Evaluation Monitoring system keeps track of the number of people who have stopped using tobacco as attributed to grantee efforts and translates that figure into dollars saved concerning excess medical costs for the State). This will allow MISRGO, grantees, and evaluators to access the data on demand.

The day was energizing as we interactively created online surveys and blogs and demonstrated the power of the net.

We also demonstrated the value of online calendars to announce event and schedule meetings across grantees and evaluators. Dr. Fetterman's slides are available here. In addition, Dr. Fetterman created a pre-test survey and a post-test survey for participants to help evaluate the session. The results were extremely positive, documenting gains in familiarity and knowledge about online surveys, blogging, online documents and spreadsheets, file sharing (pictures), telephony, and videoconferencing on the net.

In addition, a few emails provide some insight into the participants’ assessment of the workshop: “Thank you, Dr. David, for the session last week. I will put into practice all I have learned.” Another said: “Hi. Thank you for your patience and thoughtfulness with us!!! Thanks also for the blogs!!!” A copy of the Evaluation Technology Workshop Report is available from this blog.
The participants co-created another survey. It was the Arkansas Evaluation Center Needs Assessment survey. The results of the brief needs assessment is being used to inform the Center's curricular development .

MISRGO EVALUATION PLAN - 2007-2008

2007-08-17T13:50:00.001-07:00

MISRGO Evaluation Plan. This document is divided into two sections. The first half highlights accomplishments to-date, including creating the Evaluation Monitoring System.

The second half of the report presents the evaluation plan for this coming year. It charts the way for us. We plan to use the report to help us navigate through often mirky and tumultuous waves. It also provides an insight into what's in store for us in the future.

For a copy of the report click here.

MISRGO Evaluation - Grantee Satisfaction with Office Services

2007-06-04T15:17:00.000-07:00

MISRGO provides administrative and support services to tobacco prevention programs in Arkansas. Fetterman & Associates are currently facilitating an empowerment evaluation of the program, helping grantees use evaluation to accomplish their objectives (with a continuous cycle of assessment and action).

One part of the evaluation includes assessing MISRGO itself, as a supporting office or set of services. Refinements and improvements in the Office directly benefit and enhance grantee work. MISRGO staff embraced this part of the assessment and helped generate questions for the online survey because they sincerely wanted to know what grantees thought about their services. In addition, they wanted to improve the quality of their services to grantees. They understood the chain of reasoning involved, enhancing services increases the probability of improving grantee success in meeting their tobacco prevention objectives.

The report is available to the public. It was generally favorable, they valued the services they have been receiving. However, they were also critical and suggested areas meriting attention. They also recommended a list of ideas for future workshops and services.

MISRGO is already processing this information and planning to adopt many of the recommendations associated with the report, including soliciting more involvement from grantees in future conferences. This is the first step in evaluating the office, additional activities will include a site visit, interviews with staff members and grantees, and a review of relevant documentation.

4th Annual Conference Clearing the Air in Communities of Color May 23-24, 2007

2007-06-01T16:52:00.000-07:00

4th Annual Conference" Clearing the Air in Communities of Color

The conference received letters of support from Governor Mike Beebe, Mayor Carl Redus, Chancellor Lawrence Davis, Jr., Dean Calvin Johnson, and Program Coordinator Marian Evans.

Keynote Speakers included: Senator Henry "Hank" Wilkins, Broadway Joe Booker, and Kenya Masala.

Sessions included: Tobacco Use in the Asian Pacific Community, The Oxygen Project, Tobacco Use Among African American 13 Year Olds in Jefferson County,

Cease Project, Best Practices, Advocacy and You, Fire and Burns Related to Cigarettes, Improve Service Delivery by Embracing Youth Culture, Empowerment Evaluation Principles, Tobacco Control in the Latino Community of Arkansas, and Just Say Y.E.S. on the Move for Better Choices.

Dr. Fetterman presented the Closing Luncheon Presentation.

He provided conference participants (particularly grantees and evaluators), with an update on the status of the evaluation, including the Evaluation Monitoring System, the State-wide Community Survey Results, and the new Arkansas Evaluation Center.

In addition, a special video was presented highlighting the accomplishments of each grantee. (It was both inspiring and humorous as well.)

Number Quit Smoking, MISRGO Office Evaluation & the Arkansas Evaluation Center - April 26, 2007

2007-05-01T14:39:00.000-07:00

Evaluation Workshop in Little Rock - April 26, 2007

The April MISRGO Evaluation Workshop in Little Rock was very productive. Dr. Fetterman and Ms. Linda Delaney facilitated the workshop. The participants brought the session alive. They were completely engaged. (Click on picture above to display a brief video of our workshop.)

There were three key topics covered: the evaluation monitoring system (focusing on the number who quit smoking or quit using tobacco); the evaluation of the MISRGO office; and the Arkansas Evaluation Center. (Click on Fetterman & Associates below to access the slide presentation during the workshop.)

Evaluation Monitoring System

The evaluation team was reminded of the importance of continuing to collect tobacco consumption (or in this case cessation) data. They agreed an ideal data collection effort would collect monthly figures. However, a realistic goals was to assist the grantees as they collect quarterly and bi-annual cumulative data. Evaluators plan on reminding grantees how important it is to continue to collect the data, even if they only record one person who stopped smoking, since that represents $150,000 in excess medical costs saved.

Dr. Christine Sheffer
University of Arkansas for Medical Sciences
Fay W. Boozman College of Public Health
Health Behavior and Health Education Department
SOSQuitline 1.866.NOW.QUIT
SOSWorks Fax 1.888.827.7057

Marian Evans, MISRGO Coordinator, stated she would follow up with Dr. Christine Sheffer, University of Arkansas for Medical Sciences, concerning the SOS data referral plan, to help us track grantee referrals to smoke cessation programs.

MISRGO Office Evaluation

This was one of the most rewarding parts of the workshop. The focus was on plans to evaluate the effectiveness of the MISRGO office, in terms of supporting and facilitating grantee and evaluator activity.

MISRGO office staff members and MISRGO/grantee evaluators worked together to generate a list of useful questions to assess ourselves. Instead of the typical defensiveness associated with endeavor, the MISRGO office staff jumped in, posing difficult questions that they wanted the grantees and evaluators to ask about their performance. They did this because they want to know what they can do better, to better serve the entire community. It was a rewarding and fruitful exercise. An online survey will be developed in the next few weeks and grantees and evaluators will be asked to provide their views concerning the MISRGO office's efforts. This is in the true spirit of empowerment evaluation and self-assessment (to improve and enhance).

Sample Questions

Arkansas Evaluation Center

Dr. Fetterman and Ms. Delaney announced and described the new Arkansas Evaluation Center (and highlighted the new blog page about the Center). The bill (now an act) passed the House and Senate and was approved by the Governor.

Dr. Fetterman and Ms. Delaney took this time to solicit input from the workshop participants concerning the overall design of the Center, possible courses, and the strengths and weaknesses of the online distance learning component (using WebCT).

The participants were generative and productive. They recommended specific courses for a certificate program, such as research design, survey design, focus groups, interviews, and statistics (light). They also recommended a course about ethics, guidelines, and standards.

Based on a straw vote, everyone said they would be interested in participating in a evaluation certificate program. The next steps will include: the development of a needs assessment survey - to determine the level of interest in the program (above and beyond our group), a review of comparable programs (to identify the best features across programs that we might incorporate in our program), and course development (including WebCT distance learning programs).

Arkansas Evaluation Center

2007-04-02T01:20:00.000-07:00

The Minority Initiative Subrecipient Grant Office (MISRGO) and the Evaluation Team proposed the creation of an Arkansas Evaluation Center to the Arkansas Legislature on March 19, 2007.

The idea emerged from our tobacco prevention work. The need for more evaluators and more systematic evaluation training throughout the State became increasingly evident. Not only were more evaluators needed for tobacco prevention work but for many other programs as well. Thus our proposal to establish a center to train new evaluators and provide professional development training for working practitioners.

(Click on Picture to Display Full Size Poster of Arkansas Evaluation Center)

The bill describes the mission or purpose of the center. Highlights include:

enhancing the quality of existing services and programs
ensuring accountability for funds and services used
“raising the bar” concerning standards and expectations
increasing the use of data to inform program operation and decision making.

(Click on picture of Bill to display full PDF of the Bill.)

An empowerment evaluation approach and philosophy has been adopted by the Center to shape practice in communities throughout the State. Empowerment evaluation and other collaborative forms of evaluation are particularly useful in complex collaborative endeavors requiring the participation of diverse groups. It is designed to help groups accomplish their objectives using evaluation concepts and tools.

Special thanks are due to all of the MISRGO staff, grantees, and evaluators for helping to identify this need and transform this need into an action step designed to meet the need.

November 4, 2006 - American Evaluation Association (dissemination)

2007-04-02T01:10:00.000-07:00

Dr. Fetterman and Ms. Linda Delaney helped to disseminate the Tobacco Prevention evaluation findings at the American Evaluation Association in November, highlighting the amount of money saved the State as a result of these tobacco prevention programs. They also discussed the politics of tobacco prevention evaluations.

(click on photo above to display video)

September 22, 2006 - Refresher, Focus Group, and Black Caucus

2007-04-02T01:06:00.000-07:00

The September 22 workshop was designed to be a refresher and provide additional methodological training. Dr. Fetterman provided an overview of the project and its current status from an evaluation perspective, including the evaluation monitoring system and the community survey. Dr. Fetterman also provided focus group training for the Arkansas MISRGO evaluators. Ms. Delaney faciliated the self-assessment process of the training generating a 4th data point of self-assessment for the empowerment evaluation exercise.

Dr. Fetterman was also invited to meet with the Majority Leader, Senator Tracy Steele, and the Black Caucus during this trip.

He provided them with pertinent data, ranging from the amount of money saved as a result of these programs to the wide-spread community support enjoyed by MISRGO across the State (large constituent base). Dr. Fetterman also met with various news organizations to share the evaluation monitoring system findings and the community survey results.

Evaluation Monitoring Tool Update

2007-04-02T00:50:00.000-07:00

We now have an interactive spreadsheet that we share on the web. It is our repository of tobacco consumption data. It provides us with an up-to-date view of: number of materials distributed, number pledge to quit, number who quit, cost saved per person, and cost saved by grantee group. We also have a running total of each of these categories including total cost save the State concerning excess medical costs. As of July 4, 2006, the totals across the State are as follows:

We also maintain a more detailed data sheet, enabling us to view the numbers by grantee (see below).

Community Surveys

2007-04-02T00:45:00.000-07:00

We conducted a community survey throughout the State of Arkansas focusing on the views of the grantees' community members. We received 946 responses. The findings were overwhelmingly positive.

A few survey responses are provided below to provide some insight into the level of community support for tobacco prevention programs in the State.

In addition to providing information in the aggregate about community views across the State, we produced a report that provides community member viewpoints about their specific grantee or tobacco prevention program.

Clearing the Air in Communities of Color Conference

2007-04-02T00:40:00.000-07:00

We shared the evaluation approaches we have been using to measure change with colleagues at the 3rd Annual Clearing the Air in Communities of Color conference. (See video below.)

(click on picture to display video)

We discussed our training efforts, the empowerment evaluation process, our evaluation monitoring system, and our community surveys. It is a useful summary of our efforts up to May 26, 2006. The slides of our presentation are available to the public.

January 2006

2007-04-02T00:32:00.000-07:00

We held a local evaluator workshop in January. It focused on how to use the Evaluation Monitoring System. They will work with the local grantees to help them collect critical baseline data, concerning the number of people smoking in their communities. In addition, it will help programs document the number of people they reach and the number who quit. The system will also translate that into costs saved. [The workshop slides are available on this site.]

A brief video of the training is provided below.

Evaluation Monitoring System and Community Survey
Local Evaluator Workshop Training
(click on picture to display video)

In addition, we are working on community surveys of the grantees so that this information can be fed back into the programs to enable them to enhance their service to their communities.

We will continue to use the Evaluation Monitoring System tool, along with many others, and rate our own performance throughout the evaluation. These tools help to guide us through the journey.

September 2005 - Common Indicators and Qualitative Methods

2007-04-01T22:49:00.000-07:00

The September workshop was extremely productive. The grantees agreed upon common indicators to measure program performance, including the size of the target population, the number of smokers, the number reached with educational messages and services, and the number pledged to quit (as well as documented individuals in smoke cessation programs). A sample is provided below (see Figure 1):

Figure 1. Common Indicators

Dr. George Blevins, MISRGO Director,

introducing the September 2005 workshop.

Grantee oriented interviews and surveys were conducted previous to the workshop to solicit grassroots data concerning how grantees collect data, the quality of the data they have, and program implementation. In addition, recommendations were made during the interviews and workshop, including how to collect sign-in and sign-up sheets as well as pledge to quit sheets. Census and At Risk web site were shared to help grantees collect valid and useful data concerning the smoking rates in their communities.

Risk Factor and Census reports shared with grantees

The workshop focused on how to use various qualitative techniques such as ethnographically-oriented observation and interviewing. The workshop also demonstrated how to construct online surveys (with real examples on the web).

Grantees completing an ethnographically oriented observation exercise
and Dr. Fetterman critiquing their work.

The slides from the workshop training are available in pdf format. The Final Report for Year One summarizing work-to-date is online and available on this site. In addition, a brief video is available concerning workshop highlights.

Brief video of September 2005 workshop
focusing on common indicators and
qualitative techniques for data collection.

(Select arrow on bottom left of the screen to display video)

During the meeting we agreed on the need to develop an Evaluation Monitoring System. (See Figure 17 below.) It is described in the September 2005 Annual Report.

We also thought it might be useful to write a Local Evaluator Training Manual. This will help us create greater consistency in practice, ranging from collecting the same type of data to analyzing it in the same format.

We continued to operate as a group or consortium. We have adopted an empowerment evaluation approach to guide our work. This approach enables us to monitor our progress at each stage in our development as a group. We "check in" each time we meet and assess our progress. In this case, we have documented the third data point or assessment of our progress to date (see graph below).