Wednesday, May 20, 2015

3rd Quarter MISRGO Empowerment Evaluation Report is Out


MISRGO Evaluation
3rd Quarter Report 2014-2015
Prepared for:   MISRGO, Grantees & Legislative Offices
Prepared by:   Dr. Fetterman and Ms. Delaney
Date:               May 19, 2015
                       





Introduction

Tobacco use is the single most preventable cause of disease, disability, and death in the United States” (Centers for Disease Control and Prevention, November 2014).  The annual health care costs in Arkansas directly caused by smoking is $1.2 billion, according to the Arkansas Department of Health (2009) and the Campaign for Tobacco Free Kids (2013).  Arkansas’ Master Tobacco Settlement revenue is being used in part to address this serious health issue.

The University of Arkansas at Pine Bluff’s Minority Initiative Sub-Recipient Grant Office (MISRGO) has received Settlement funds through the Arkansas Department of Health to focus on tobacco use in minority communities. MISRGO’s mission is specifically to prevent and reduce tobacco use in minority communities.

MISRGO has awarded more than 50 Arkansas organizations with funding for tobacco prevention and cessation programs.  MISRGO currently sponsors and provides technical assistance for 15 grantees across the State of Arkansas.

MISRGO grantees have broad-based support across the State for their tobacco prevention and cessation work.  (See MISRGO Empowerment Evaluation: June 2011 Annual Report, Fetterman, Tremain, and Delaney, 2011.)
 
MISRGO grantees also have a strong track record of success based on past performance.  For example, in the last 3 years, the majority of MISRGO grantees met or exceeded their annual performance goals (see MISRGO Evaluation Annual Report 2013-2014 Fetterman, Delaney, and Tremain, July 11, 2014; 2012-2013 Fetterman, Delaney, Tremain, July 18, 2013a; and 2011-12 Fetterman, Tremain, and Delaney, July 12, 2012). 

Purpose

The purpose of this report is to present MISRGO tobacco prevention grantee progress at this point in the year - the 3rd  Quarter 2014-2015.  The fiscal and programmatic end of the year is June 30, 2015.  (See Appendix A for a glossary of grantee names.)

MISRGO Grantee Intervention Areas

MISRGO grantees use the Centers for Disease Control and Prevention’s (CDC) approved and recommended intervention areas.  They include:

Area 1:  Eliminate Exposure to Secondhand Smoke
Area 2:  Preventing Initiation Among Youth and Young Adults
Area 3:  Promoting Quitting Among Youth and Adults
Area 4:  Addressing Disparities

Grantee progress and activities are reported on both a quarterly and annual basis, according to these CDC intervention areas. 
 
Quarterly Progress

During the first quarter, MISRGO grantees established their goals, benchmarks, and baselines.   Goals are the planned outcomes or results at the end of the year, e.g. number of smoke-free parks.  Benchmarks are intermediate objectives or approximations of outcomes.  Baselines represent, for example, the number of smoke-free environments that existed before the grantee activity or intervention. 

During the second quarter, grantees report on their mid-year progress toward annual goals.  This provides grantees with an opportunity to reflect on their performance.  Based on these data, grantees continue as planned, make mid-course changes, and request assistance as needed.

The third quarter focuses on any gaps between benchmarks and annual goals.  Grantees share techniques with each other, MISRGO staff provide additional recommendations, and Fetterman & Associates notify grantees about gaps and make specific recommendations to close the gap. 

The fourth quarter is the annual comparison of grantee goals with their actual performance throughout the year.

3rd Quarter Findings

Between 73% and 86% of the grantees have met or exceeded their 3rd quarter evaluation benchmarks.

Area 1:  Eliminate Exposure to Secondhand Smoke  (73% met or exceeded 3rd Qtr benchmarks)
Area 2:  Preventing Initiation Among Youth and Young Adults (80% met or exceeded 3rd Qtr benchmarks)
Area 3:  Promoting Quitting Among Youth and Adults (80% met or exceeded 3rddQtr benchmarks)
Area 4:  Addressing Disparities (86% met or exceeded 3rd Qtr benchmarks)


CDC Area 1:  Eliminate Exposure to Secondhand Smoke

73% of the grantees have met or exceeded their 3rd  quarter evaluation benchmarks concerning Area 1:  Eliminate Exposure to Secondhand Smoke.

Grantee activities focused on:

·      Encourage voluntary smoke-free perimeter policies
·      Conduct secondhand smoke tobacco prevention messaging
·      Help to establish smoke-free park policies
·      Encourage tobacco-free policies
·      Encourage establishment of campus free ordinances and/or workplace smoking policies
·      Implement anti-smoking media campaigns (via radio, social media and television)
·      Encourage "No Smoking within 25 feet" policies
·      Promote smoke-free home and car campaign
·      Educate about the impact of marketing strategies
·      Educate about the benefits of smoke free environments in the car (ACT 811) and in the home

CDC Area 2:  Preventing Initiation Among Youth and Young Adults

80% of the grantees have also met or exceeded their 3rd  quarter evaluation benchmarks concerning Area 2: Preventing Initiation Among Youth and Young Adults.

Grantee activities focused on:

·      Conduct Operation Store Front surveys
·      Conduct retailer visits regarding sales to minors
·      Encourage smoke-free or tobacco-free church campus policy
·      Conduct Open Storefront youth education campaign, including practice refusal skills when offered tobacco
·      Use media, include radio campaigns, social media and promotions, to inform youth about dangers of smoking
·      Educate youth about the effects of smoking
·      Encourage signed tobacco related pledges
·      Help youth reach "Past 30-day chewing tobacco use" goals
·      Help to implement point of purchase policy
·      Partner with organizations to prevent initiation of tobacco use by youth (presentations)
·      Encourage youth to pledge to be tobacco free
·      Train youth in prevention strategies through media outreach, gorilla marketing campaigns, and youth rallies

CDC Area 3:  Promoting Quitting Among Youth and Adults

80% of the grantees have met or exceeded their 3rd quarter evaluation benchmarks concerning Area 3:  Promoting Quitting Among Youth and Adults 

Grantee activities focused on:

·      Encourage use of Arkansas Quit-line Fax Referral System
·      Encourage youth to pledge to quit
·      Work with churches, schools and peers to help friends quit smoking
·      Partner with community organizations to host or take part in events, presentations, and/or activities to help people quit smoking
·      Help to increase the number of participants in cessation programs
·      Educate through radio/social media and print about the dangers of tobacco products
·      Educate people about cessation services
·      Present information (tobacco facts) concerning the effects of tobacco
·      Encourage people to participate in the established cessation programs
·      Educate about the dangers of tobacco through evidence-based strategies
·      Educate about cessation services using surveys, education activities, and media outreach

CDC Area 4:  Addressing Disparities

86% of the grantees have also met or exceeded their 3rd quarter evaluation benchmarks concerning Area 4:  Addressing Disparities. 

Grantee activities focused on:

·      Provide information about dangers of tobacco use, tobacco laws, dangers of secondhand smoke, and the importance of the Quitline (focus on Hispanics)
·      Encourage people to call and enroll with the Arkansas Tobacco Quitline (focus on African American and Hispanic women)
·      Help to increase the number of voluntary smoke-free home policies
·      Encourage people to make smoke-free home pledges
·      Educate about tobacco disparities using focus groups and media campaigns (addressing tobacco-related health disparities, focusing on African American and Latino tobacco users)
·      Encourage people to call the Quit Line (focusing on homeless, LGBT, substance abusers and aging populations)
·      Help to increase knowledge about tobacco policies and regulations regarding youth
·      Conduct media campaigns to increase awareness of tobacco dangers and tobacco industry advertising tactics using radio, social media, billboards, and flyers
·      Encourage smoke-free home/car pledges
·      Educate about the hazards of secondhand smoke through radio, print and social media campaigns (focusing on African American males)
·      Encourage people to participate in focus groups and use Facebook to learn about tobacco ads and tactics as well as facts about tobacco use
·      Educate tobacco retailers about the laws associated with selling tobacco to underage youth
·      Educate underserved on the ills of tobacco products
·      Recruit youth to become community advocates
·      Encourage smoke-free policies or point of purchase policies to reduce advertising to youth and disparity groups
·      Help to increase the number of minorities who acknowledge tobacco related disparities and the practices used by the tobacco industry to create these disparities
·      Educated underserved about the ills of tobacco products through surveys and media outreach

Evaluation Dashboard

An evaluation dashboard for each grantee’s annual performance is provided to facilitate communication and collaboration among grantees, enhance MISRGO accountability, and build knowledge.  The dashboards compare grantee’s actual performance with their planned quarterly and annual outcomes.  This helps stakeholders monitor progress and ensure internal and external accountability.  It is organized by CDC intervention area (see Appendix B). The MISRGO Grantee Activity Log provides an additional insight into the list of grantee activities across sites and intervention areas (see Appendix C).  



Technical Assistance

MISRGO grantees receive evaluation technical assistance throughout the year, including assistance monitoring and evaluating their progress.  

New and long-term grantees participated in an evaluation workshop on October 24, 2014.  The workshop highlights included:  1) tobacco prevention resources and 2) the surge of e-cigarette use.  In addition, evaluation methodological assistance was provided focusing on: 1) the use of pre- and post-tests; and 4) evaluation dashboards. 


The majority of grantees met for a second technical assistance workshop on April 30, 2015.  This workshop focused on closing the gap between 3rd quarter accomplishments and final end-of-the-year goals.  In addition technical skills were enhanced. Excel and social media training was provided.


Tobacco Prevention Resources

Evaluation resources were also shared. They included: Campaign for Tobacco-Free Kids; County Health Ranking; County Health Calculator; Arkansas Department of Health; CDC State Tobacco Activities, Tracking, and Evaluation System; and the MISRGO Evaluation blog.


We also demonstrated the use of the Community Commons (an on line social mapping tool) to document tobacco use in each county throughout Arkansas.


In addition, grantees discussed a chapter titled “Empowerment Evaluation and Evaluation Capacity Building in a 10-Year Tobacco Prevention Initiative.”  The chapter was about 10 years of MISRGO tobacco prevention evaluation. It was published October 2014.  The chapter was published in: Fetterman, D.M., Kaftarian, S. and Wandersman, A. (2015) (eds.) Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage. 
 

Use of Pre- and Post-Tests

During the first evaluation workshop, grantees participated in an online survey before and after lunch to demonstrate the value of pre- and post-tests.  Grantees were asked to rate how hungry they were before lunch and then after the intervention (lunch) to assess their hunger level again.  We used SurveyMonkey to demonstrate the utility of this technique. The same approach was recommended for grantee tobacco prevention awareness sessions, to document knowledge acquisition or experiential changes in attitude.


Evaluation Dashboards

Grantees participated in another exercise to ensure the appropriate use of their evaluation dashboards (discussed earlier), specifically to monitor their own progress throughout the year. The exercise highlighted the value of establishing goals, baselines, and benchmarks.  Once these were established only the actual performance measures were needed throughout the year – comparing actual performance with benchmarks and goals to determine if sufficient progress was made.

 

MISRGO and evaluation team members also use it to enhance accountability, signaling precisely when assistance is required to increase the probability of grantees accomplishing their end-of-year goals. 

The evaluation dashboards are designed to build capacity and produce outcomes.  They are in accordance with an empowerment evaluation approach.  (Fetterman, 2013; Fetterman, Kaftarian, and Wandersman, 2015; Fetterman and Wandersman, 2005.)



e-cigarette

The e-cigarette was discussed as well during both workshops and off-line after the workshops.  We discussed the urgency of the issue, since CDC has found that:  E-cigarette use tripled among middle and high school students in just one year.”  We also highlighted the lack of regulation concerning the product and potential dangers.  The FDA and World Health Organization have called for stricter regulation of e-cigarettes.  Additional observations were made during the workshop, including: how e-cigarettes looks like cigarettes and following the same “script” as earlier marketing of cigarettes.  This is considered part of the tobacco industry’s efforts to “normalize” the use of cigarettes in the public again.

Dialogue

In addition, during both technical assistance workshops grantees openly discussed obstacles pursuing their work and generously shared advice and best practices, based on their own experience.  The collaborative spirit helped grantees solve problems, including soliciting help from each other in translating tobacco prevention material into Spanish.

Technical Assistance Workshop:  October 24, 2014

  
Technical Assistance Workshop:  April 30, 2015


Grantees were provided with additional tobacco prevention tools and updates to facilitate their programmatic and evaluative efforts throughout the year.  (See Appendix D for an example of periodic technical assistance communications.) 

Details concerning evaluation technical assistance are provided on the MISRGO tobacco prevention blog at: http://tobaccoprevention.blogspot.com

Conclusion

MISRGO grantees have established their goals, benchmarks, and baselines.  In addition, they have documented their actual performance.  The grantees’ 3rd quarter evaluation report documents progress meeting or exceeding quarterly benchmarks, according to CDC intervention areas.  The percentages are presented below:

 
The evaluation dashboards help grantees monitor their own performance.   They are being used to confirm grantee performance and alert MISRGO and grantees concerning the need for corrections throughout the year.

The evaluation findings reported in this 3rd quarter evaluation report provide evidence of MISRGO grantee dedication and commitment, concerning tobacco prevention.  They also document progress toward annual programmatic and evaluation goals.




References

Arkansas Department of Health (2009). Tobacco Prevention and Cessation Program
ARKANSAS STRATEGIC PLAN TO PREVENT AND REDUCE TOBACCO USE 2009 – 2014.  Little Rock, Arkansas:  Arkansas Department of Health, p. 3. http://www.healthy.arkansas.gov/programsServices/tobaccoprevent/Documents/TPCPStrategicPlan.pdf

Centers for Disease Control and Prevention National Center for Chronic Disease Prevention and Health Promotion (2014). Tobacco Use.  Targeting the Nation’s Leading Killer
- At A Glance 2011.  Atlanta, Georgia:  Centers for Disease Control and Prevention.

Campaign for Tobacco Free Kids (2013). Annual health care costs in Arkansas directly caused by smoking. http://www.tobaccofreekids.org/facts_issues/toll_us/arkansas

Fetterman, D.M. (2013).  Empowerment Evaluation in the Digital Villages:  Hewlett-Packard’s $15 Million Race Toward Social Justice.  Stanford:  Stanford University Press.

Fetterman, D.M., Delaney, L., Triana-Tremain, B., and Evans-Lee, M. (2014). Empowerment Evaluation and Capacity Building in a 10-Year Tobacco Prevention Initiative.  In Fetterman, D.M., Kaftarian, S., and Wandersman, A. (eds). Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.

Fetterman, D.M., Delaney, L., and Tremain, B. MISRGO Evaluation Annual Report 2013-2014.  San Jose:  Fetterman & Associates (July 11, 2014).

Fetterman,  D., Delaney, L., Tremain, B. (2013a).  MISRGO Evaluation Annual Report 2012-2013.  San Jose:  Fetterman & Associates (July 18, 2013).

Fetterman, D.M., Delaney, L., and Tremain, B. (2013b).  MISRGO Evaluation 2nd Quarter Progress Report 2013.  San Jose:  Fetterman & Associates.

Fetterman, D.M., Delaney, L., and Tremain, B. (2013c).  MISRGO Evaluation 3rd  Quarter Progress Report 2013.  San Jose:  Fetterman & Associates

Fetterman, D.M. , Kaftarian, S., and Wandersman, A. (2015).  Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.

Fetterman, D.M., Tremain, B., and Delaney, L. (2012).  MISRGO Evaluation Annual Report 2011-2-12.  July 12, 2012. San Jose: Fetterman & Associates.

Fetterman, D.M., Tremain, B., and Delaney, L. (2011).  MISRGO Empowerment Evaluation:  June 2011 Annual Report.  San Jose: Fetterman & Associates.


Fetterman, D.M. and Wandersman, A. (2005). Empowerment Evaluation Principles in Practice.  New York: Guilford Publication.

(Full report available here online.)