Tuesday, July 26, 2011

Annual Report - 2011 MISRGO Evaluation Annual Report

2011 Annual Evaluation Report.  The complete 2011 Annual Evaluation Report is available online.  It highlights grantee accomplishments throughout the year.

A few highlights from the report are presented below.

Tobacco Use in Arkansas

It begins with a portrait of tobacco use in Arkansas.  Over 480,000 individuals smoke cigarettes in the State of Arkansas.  That is 22% of the population.  Arkansas ranks 44th among states concerning the use of tobacco.

The average tobacco use across MISRGO serviced counties is 24%.




Highlighted Accomplishments

Smoke Free Policy

MISRGO grantees have been effective creating tobacco free policy and environments. 


MISRGO grantees adhere to CDC recommendations in this regard.  The Coalition for a Tobacco Free Arkansas was instrumental in contributing to the passage of a new law - Act 811.  It strengthens Arkansas' smoke-free car law, Act 13 of 2006, by increasing the age of children protected from second hand smoke from 6 to 14 years of age. This impacts 827,411 youth.


Smoke Free Parks and Companies

MISRGO grantees have helped to establish 13 smoke-free parks during 2010-1011.  Two grantees have been granted permission to place smoke-free signs in parks in lieu of a formal ordinance and two parks have created voluntary perimeters.  This is in addition to the 56 smoke-free parks already in Little Rock, as a result of the Coalition for Tobacco-Free Arkansas' work to help pass the ordinance.

The Mississippi County Coalition for a Tobacco Free Arkansas helped to convince local industry to adopt smoke free policies.

Newspapers and Social Media

MISRGO grantees have used local newspapers and social media to extend the impact of their work.  They have sparked the debate about tobacco use and prevention in the press.


MISRGO grantees have also used the press (including television, radio, and newspapers) to highlight their accomplishments


Grantees Tyler Clark and Page Daniel, NWA Tobacco-free Coalition, have used social media, specifically YouTube, to help educate about air quality in smoking bars in Fayetteville (http://www.youtube.com/watch?v=0_2Lhpi07-0).  Bruce Lockett of Future Builders, Inc. uses YouTube to educate youth about the dangers of smoking.  Kionti Traylor, Garland County CARES Quapaw House, uses radio broadcasts to reach youth (http://web.me.com/profdavidf/nonsmoking.mov).


Also see Page Daniel and Tyler Clark's YouTube video of their 2011 Survey of Registered Voters - used to determine the level of support for their smoke free campaign (http://www.youtube.com/watch?v=3HDDQ9TdFh4).  (A brief article about their work and use of this information is provided online in Ozarks unbound.)

Grantees, Carolyn and Eddie Yarborough, used local television to get the tobacco prevention message across. Billboards continue to be an additionally effective tool.

Research

MISRGO grantees have used extensive use of surveys to determine attitudes toward banning smoking from bars and used the data to successfully educate the public about the need for smoke free environments. They are sharing their data including the Fayetteville Arkansas Air Quality Monitoring Study conducted by the Roswell Park Cancer Institute.  These kinds of studies have demonstrated how employees and patrons in Fayetteville bars are being exposed to harmful levels of air pollution.

Quit Line

The Quit Line is an important tool in the fight against tobacco consumption. 




MISRGO grantees have submitted 948 fax referrals to the Quitline this year.  In total MISRGO grantees have submitted over 3,217 fax referrals to the Quitline, excluding grantees no longer funded.
Pledges & Successes

There are many steps needed to convince people not to begin smoking and to promise to stop smoking.  MISRGO grantees have been effective in convincing minority members of the community to take one of the first significant steps in this direction, pledging to quit.  Although a pledge is not as compelling as a referral to the Quitline or larger community-based change, such as creating tobacco free environments, a pledge is an important stepping-stone.  


A pledge can help prevent or stop a person from using tobacco. 

MISRGO grantees have convinced 1448 youth to pledge to be smoke free this year.  They have also convinced 200 parents to pledge not to use tobacco in their homes or in their cars. 

Currently funded MISRGO grantees have helped approximately 2,710 members of their communities throughout the state to stop using tobacco.

Education

The type of success described in this report is predicated on a number of factors.  One of the most important contributing factors is education.  Education may range from disseminating educational information at health fairs and in schools to measuring knowledge retention using pre- and post-tests.  Most MISRGO educational interventions preclude extensive testing due to the nature of the informal community and home settings.  However, there is no single educational intervention that works for everyone and thus multiple educational approaches are used to convince people not to use tobacco, including youth performances.


Education in its various forms represents the backbone or foundation of any effective tobacco prevention program.

MISRGO grantee educational efforts resulted in:

66,766   receiving information about dangers of tobacco use
1,177      receiving information about Acts 8 & 13
2,400      receiving information about smoke-free ordinances

In addition, pregnant, low-income, homeless, and Hispanics or Latinos received tobacco prevention educational materials from MISRGO grantees.

318         pregnant, low-income & homeless people received tobacco prevention information
300          Hispanics received tobacco prevention information

Grantee Activities

MISRGO accomplishments are the result of many activities, monitoring tools, and administrative oversight and technical assistance.

An abbreviated list of grantee activities is provided below to highlight the wide-spectrum of work required to produce MISRGO outcomes.

  •          Working to establish ordinances in parks
  •           Promoting organizations in the community that review tobacco policies (e.g., restaurants and facilities)
  •          Helping to increase compliance rates among retailers
  •          Identifying disparities concerning sales to minors
  •          Providing tobacco education through surveys and media outreach
  •          Providing AHDC Workshops on ACTs 8 and 13
  •          Distributing educational materials & providing tobacco prevention sessions
  •          Promoting the Arkansas Tobacco Quitline
  •          Promoting the fax back referral program
  •          Helping to decrease tobacco advertising
  •          Helping to increase collaboration with other organizations (e.g., organizations serving minorities, clinics/physicians, schools) to promote a tobacco free message
  •          Helping to encourage smoke free households and cars
  •          Training youth advocates or FAST teams in tobacco prevention

 Monitoring Progress

MISRGO has implemented a tool to monitor grantee progress and performance.  It is designed to help grantees establish their own goals and monitor their own efforts.  The tool also enables MISRGO to determine if grantees are making satisfactory progress or if assistance is needed.

Baseline.  Grantees established a baseline, such as the number of minority members of the community already receiving educational materials about second hand smoke.  In the example below, the baseline was 0.

Goal.  Grantees then established a goal or multiple goals, the number of minority community members they would like to receive educational materials about second hand smoke by the end of the year.  In this case the number is 1000 for this year and 5,000 over a 5-year period.

Benchmarks.  Grantees created benchmarks, or steps toward their goal.  This helped them measure their progress toward their goals.  In this case the grantee created the following benchmarks for this year:  250 in the 1st quarter, 500 in the 2nd quarter, 750 in the 3rd quarter, and 1000 in the 4th quarter or end of year goal.
 

A review of grantee performance based on these data suggests that they are making satisfactory progress toward end-of-the-year goals.  Based on a triangulated review of 18 grantees’ performances, the evaluation team found: 

Area 1.  Eliminating Exposure:  11 grantees meeting or exceeding their benchmarks, 3 near their benchmarks, and 4 not meeting benchmarks.
Area 2.  Preventing Initiation:  13 grantees meeting or exceeding their benchmarks, 4 near their benchmarks, and 0 not meeting benchmarks.
Area 3.  Promoting Quitting:  14 grantees meeting or exceeding their benchmarks, 2 near their benchmarks, and 1 not meeting benchmarks.
Area 4.  Addressing Disparities:  15 grantees meeting or exceeding their benchmarks, 2 near their benchmarks, and 1 not meeting benchmarks.


Public Support (State-wide)

There is State-wide support for the use of public funds for MISRGO tobacco prevention programs.  Ninety-nine percent of the 929 people surveyed throughout the state (who were familiar with MISRGO grantee programs), stated they thought that tobacco prevention and education programs were a good use of public money.


Similarly, 99% reported that they would recommend their local MISRGO-related tobacco prevention & education agency to their friends or family members if they smoked or were thinking about smoking.


Community members throughout the State also thought that the MISRGO-related programs have been effective in preventing people from using tobacco and stopping them from using tobacco (95-96% of those surveyed).


Community members rated MISRGO-related tobacco prevention programs effective or very effective (94%).


The majority of community members surveyed thought MISRGO-related tobacco prevention agencies were effective or very effective concerning the following activities:

90%    Sending out educational literature
92%    Letting people know about the dangers of second hand smoke
91%    Telling people about the QUIT Line
86%    Educating retailers to not sell tobacco products to minors
84%    Getting perimeter laws
86%    Getting smoke free parks
79%    Getting advertisers to reduce their advertising of tobacco products
90%    Building coalitions with other anti-tobacco agencies
88%    Helping minorities stop smoking (or not begin)

The State-wide community-based support for MISRGO-related tobacco prevention agencies, highlights MISRGO’s high degree of success working with minority communities.  MISRGO’s work is considered credible and valued throughout the state.

Capacity Building

MISRGO has provided numerous workshops, lectures, and technical assistance sessions to improve grantee performance.  (See Appendix C.) The MISRGO evaluation team has also provided lectures, demonstrations, exercises, and workshops to enhance evaluation capacity.  Workshops focused on tools to conduct both process and outcome evaluations.  Technology training was embedded in the evaluation training, ranging from online survey deployment to the use of digital photography to document and disseminate accomplishments.  Highlights of the October 13, 2010 and April 28, 2010 workshops are discussed below.

October 13, 2010 MISRGO Empowerment Evaluation Workshop: Baselines and Benchmarks

This MISRGO empowerment evaluation workshop focused on:  documenting actual performance in comparison to the baseline, goals, and benchmarks.  We critiqued grantees’ five-year plans, using this model.  Then we applied the same approach to quarterly reporting for the year.   By comparing actual performance with the benchmarks we were able to determine if the grantees were meeting their goals or if they needed to fortify their efforts.  This approach helps monitor change over time. Slides from the presentation are available online at the MISRGO Empowerment Evaluation blog (http://tobaccoprevention.blogspot.com).



The evaluation team, Dr. Beverley Tremain, Ms. Linda Delaney, and Dr. Fetterman provided the grantees with individualized assistance as they refined their first quarterly report.   


As the grantees volunteered their work for a group critique, it became clear that this was a learning process for the entire group.



Key lessons learned during the workshop included:
  • separating out goals, instead of clustering them in one sentence, in order to track them more effectively
  •  ensuring that activities are related to the goal
  •  remembering to record actual performance data
  •  stating benchmarks for each year and each quarter; (as opposed to only one year and making the reader assume that the same benchmarks apply to each quarter or year)
  • using actual numbers verses percentages; stating the goal in terms of a number
  • completing the calculations for the reader; (instead of stating that the goal is to improve by 20% and making the reader go back to the baseline in a previous report to calculate 20% of the baseline figure)
  • similarly, avoiding ambiguous statements, e.g. will increase, but not specifying by how much or from what baseline; will improve, without stating how they will improve, etc.
  • using the matrix instrument to facilitate record keeping, e.g. baseline, goals, benchmarks, and actual performance
  • creating bar charts that match the matrix data or figures
Grantee discussions were animated and productive.  They shared strategies to accomplish goals they had in common. 



The grantees also made a “good faith” commitment to focus on fewer things that they all had in common in order to have a greater “dose” effect.  Even the breaks were productive, as grantees, MISRGO staff, and empowerment evaluators, shared insights into what works and what has not worked with each other.  MISRGO staff were viewed as supportive and helpful.



The Arkansas Department of Health was invited to share their knowledge and resources with the grantees.  One of their valuable tobacco related data and report web pages is available online.


They recommended Center for Disease Control reports including:
They also shared their Tobacco Data Deck 2010, providing useful State-wide data to compare with local grantee efforts.  The Data Deck is available on the ADH web site and will be available on the GEMS web-site. (GEMS is the web-based tool for grantees to enter their data.)

The workshop concluded with a local “best practices” exchange.  Grantees mixed evidenced-based practices with practice-based evidence.


April 28, 2011 Empowerment Evaluation Workshop: Build Capacity to Reduce Tobacco Consumption  

The evaluation team conducted an engaging and productive workshop on April 28, 2011 in Pine Bluff.  The workshop was held at the Arkansas River Education Co-op.
Dr. Fetterman provided an overview about empowerment evaluation for new members, reminding everyone that they are in charge of conducting the evaluation and that the three empowerment evaluators are facilitators and coaches to keep it rigorous and on track.


We also briefly discussed the MISRGO grantee evaluation dashboard that we developed and agreed to implement.  The dashboard is used to let everyone know what grantee goals are, benchmarks (or steps along the way), and actual performance, so that the performance each quarter can be compared with grantee benchmarks (or planned mini-goals for each quarter).


We demonstrated what the data looks like when transformed into a bar chart.  The bar charts make it easy for grantees to see their progress towards their goals.  If they were not making progress, then the charts represented a red flag.  It let everyone know it was time to ask for help with alternative strategies or to rethink the scale of the effort.


Empowerment evaluators Dr. Beverly Tremain and Ms. Linda Delaney provided instruction and coaching during the workshop.  They highlighted how the reporting format could be improved, providing constructive suggestions to enhance their presentation.


A detailed presentation of MISRGO evaluation capacity building efforts is available on the web at:  http://tobaccoprevention.blogspot.com.  (See also Fetterman and Wandersman, 2005 for a detailed discussion about evaluation capacity building.) 

Conclusion

MISRGO grantees have had a significant impact on tobacco-free legislation, contributing to the passage of Act 811.  This has an impact on over 827,411 youth.

MISRGO grantees continue to have had a significant impact on minority populations in Arkansas They have established a strong rapport with African American, Latino or Hispanic, and Asian communities in the State.  This relationship has enabled them to disseminate tobacco prevention literature to youth and adults in these communities.  It has also been instrumental in encouraging minorities in these communities to establish tobacco-free environments, particularly in public parks.   MISRGO grantees continue to increase the numbers of community members calling the QUIT line.  (See Appendix D for a list of the counties served throughout the state.) 

MISRGO grantees work in alignment with CDC principles and best practices, focusing on:

1) eliminating exposure to second hand smoke
2) preventing initiation
3) promoting quitting, and
4) addressing disparities 

They work in a wide-range of thematic areas including:  education, promoting smoke-free environments, contributing to policy, and cultivating legislative relationships.  They use conventional media tools, such as television, radio, and newspapers, to generate awareness and a dialogue about the issues.  MISRGO grantees have also used social media, including YouTube to reach out to youth.

Minority community members throughout the State support the use of public funds for MISRGO-based tobacco prevention programs.  This support is based in part on their perception of the grantees’ effectiveness in preventing and stopping people from using tobacco in their communities.

MISRGO grantees are making progress toward their stated goals and objectives.  Continued support will enhance their capacity to reduce tobacco consumption in minority communities throughout the State of Arkansas.



_/_/_/_/_/_/_/ _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

References

Centers for Disease Control and Prevention.  Best Practices for Comprehensive Tobacco Control Programs – 2007.  Atlanta:  U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; October, 2007.

Fetterman, D., Tremain, B., and Delaney, L. (2011a).  MISRGO Empowerment Evaluation.  January 2011. Report:  Accomplishments.  San Jose:  Fetterman & Associates (January 6, 2011).

Fetterman, D. and Wandersman, A. (2005).  Empowerment evaluation principles in practice.  New York: Guilford Publications.

Shultz, J.M., Novotny, T.E., and Rice, D.P. (1991).  Quantifying the disease impact of cigarette smoking with SAMMEC II. Software.  Public Health Reports, May-June, 106(3):326-333.

Starr, G., Rogers, T., Schooley, M., Porter, S., Wiesen, E., Jamison, N.  Key Outcome Indicators For Evaluating Comprehensive Tobacco Control Programs.  Atlanta, GA:  Center for Disease Control and Prevention, 2005. 

Substance Abuse and Mental Health Services Administration, Office of Applied Studies (2008). Results from the 2007 National Survey on Drug Use and Health: National Findings (NSDUH Series H-34, DHHS Publication No. SMA 08-4343). Rockville, MD.


See the 2011 Annual Evaluation Report for additional details and the Appendices.