Sunday, July 17, 2016

2015-2016 MISRGO Annual Empowerment Evaluation Report


MISRGO Evaluation
Annual Report 2015-2016
Prepared for:   MISRGO, Grantees & Legislative Offices
Prepared by:   Dr. Fetterman and Ms. Delaney
Date:               July 16, 2016
                       


 
Percent of Grantees Meeting Annual Goals





Introduction

Tobacco use is the single most preventable cause of disease, disability, and death in the United States” (Center for Disease Control and Prevention, November 2014).  The annual health care costs in Arkansas directly caused by smoking is $1.21 billion, according to the Campaign for Tobacco Free Kids (2015).  The tobacco industry spends over $119 million marketing tobacco in Arkansas.  Arkansas’ Master Tobacco Settlement revenue is being used in part to address this serious health issue.  (See also Arkansas Department of Health (2009).)  However, “as of fiscal year 2010, Arkansas allocated 54.4% of the CDC-recommended funding for tobacco control ($19.8 million of $36.4 million).”[i]

The University of Arkansas at Pine Bluff’s Minority Initiative Sub-Recipient Grant Office (MISRGO) has received a portion of the Settlement funds through the Arkansas Department of Health to focus on tobacco use in minority communities. MISRGO’s mission is specifically to prevent and reduce tobacco use in minority communities.

MISRGO has awarded more than 50 Arkansas organizations with funding for tobacco prevention and cessation programs.  MISRGO currently sponsors and provides technical assistance for 14 grantees across the State of Arkansas.

MISRGO grantees have broad-based support across the State for their tobacco prevention and cessation work.  (See MISRGO Empowerment Evaluation: June 2011 Annual Report, Fetterman, Tremain, and Delaney, 2011.)
 
MISRGO grantees also have a strong track record of success based on past performance.  For example, in the last 4 years, the majority of MISRGO grantees met or exceeded their annual performance goals. (See MISRGO Empowerment Evaluation Annual Report, 2014-2015, Fetterman and Delaney; MISRGO Evaluation Annual Report 2013-2014 Fetterman, Delaney, and Tremain, July 11, 2014; 2012-2013 Fetterman, Delaney, Tremain, July 18, 2013a; and 2011-12 Fetterman, Tremain, and Delaney, July 12, 2012). 

Purpose

The purpose of this report is to present MISRGO tobacco prevention grantee progress at the end of the year (2015-2016).  The fiscal and programmatic end of the year is June 30, 2016.  (See Appendix A for a glossary of grantee names.)

MISRGO Grantee Intervention Areas

MISRGO grantees use the Centers for Disease Control and Prevention’s (CDC) approved and recommended intervention areas.  They include:

Area 1:  Eliminate Exposure to Secondhand Smoke
Area 2:  Preventing Initiation Among Youth and Young Adults
Area 3:  Promoting Quitting Among Youth and Adults
Area 4:  Addressing Disparities

Grantee progress and activities are reported on both a quarterly and annual basis, according to these CDC intervention areas. 
 
Quarterly Progress

During the first quarter, MISRGO grantees established their goals, milestones, and baselines.   Goals are the planned outcomes or results at the end of the year, e.g. number of smoke-free parks.  Milestones are intermediate objectives or approximations of outcomes.  Baselines represent, for example, the number of smoke-free environments that existed before the grantee activity or intervention. 

During the second quarter, grantees report on their mid-year progress toward annual goals.  This provides grantees with an opportunity to reflect on their performance.  Based on these data, grantees continue as planned, make mid-course changes, and/or request assistance as needed.

The third quarter highlights any gaps between milestones and annual goals.  Grantees share techniques with each other, MISRGO staff provide additional recommendations, and Fetterman & Associates notify grantees about gaps and make specific recommendations to close the gap.  

The fourth quarter, this report, is the annual comparison of their goals with their actual performance throughout the year.

4th Quarter Findings

100% of MISRGO grantees established their goals, milestones, and baselines for the year during the 1st quarter.   Between 50% and 79% of the grantees met[ii] or exceeded their 2nd quarter evaluation milestones.  Between 69% and 82% of the grantees met or exceeded their 3rd quarter evaluation milestones.  Between 77% and 86% of the grantees met or exceeded their 4th quarter or annual evaluation goals.


Area 1:  Eliminate Exposure to Secondhand Smoke (86% met or exceeded annual goals)
Area 2:  Preventing Initiation Among Youth and Young Adults (77% met or exceeded annual goals)
Area 3:  Promoting Quitting Among Youth and Adults (86% met or exceeded annual goals)
Area 4:  Addressing Disparities (77% met or exceeded annual goals)



Percent of Grantees Meeting Met or Exceeded Annual Goals

CDC Area 1:  Eliminate Exposure to Secondhand Smoke

86% of the grantees have met or exceeded their annual evaluation goals concerning,
Area 1:  Eliminate Exposure to Secondhand Smoke.

Grantee activities focused on:

·      Encourage voluntary smoke-free perimeter policies
·      Conduct secondhand smoke tobacco prevention messaging
·      Help establish smoke-free park policies
·      Encourage tobacco-free policies
·      Implement anti-smoking media campaigns (via radio, social media and television)
·      Promote smoke-free home and car campaign
·      Educate about the impact of marketing strategies
·      Educate about the benefits of smoke free environments in the car (ACT 811) and in the home
·      Conduct operation storefront/STARTS surveys
·      Organize tobacco free workshops/coalition meetings

CDC Area 2:  Preventing Initiation Among Youth and Young Adults

77% of the grantees have also met or exceeded their annual evaluation goals concerning, Area 2: Preventing Initiation Among Youth and Young Adults.

Grantee activities focused on:

·      Conduct Operation Store Front/STARS surveys
·      Conduct retailer visits regarding sales to minors
·      Encourage smoke-free or tobacco-free church campus policy
·      Use media, include radio campaigns, social media and promotions, to inform youth about dangers of smoking (focus on African Americans and Latina youth)
·      Educate youth about the effects of smoking
·      Help to implement point of purchase policy
·      Partner with organizations to prevent initiation of tobacco use by youth (presentations)
·       Encourage youth to pledge to be tobacco free
·      Train youth in prevention strategies through media outreach and youth rallies
·      Use media to educate about prevention strategies

CDC Area 3:  Promoting Quitting Among Youth and Adults

86% of the grantees have met or exceeded their annual evaluation goals concerning,
Area 3:  Promoting Quitting. 

Grantee activities focused on:

·      Encourage use of Arkansas Quit-line Fax Referral System
·      Encourage youth to pledge to quit
·      Work with churches, schools and peers to help friends quit smoking
·      Partner with community organizations to host or take part in events, presentations, and/or activities to help people quit smoking
·      Help increase the number of participants in cessation programs
·      Educate through radio/social media and print about the dangers of tobacco products
·      Educate people about cessation services
·      Present information (tobacco facts) concerning the effects of tobacco
·      Encourage people to participate in the established cessation programs
·      Educate about the dangers of tobacco through evidence-based strategies
·      Educate about cessation services using surveys, education activities, and media outreach

CDC Area 4:  Addressing Disparities

77% of the grantees have also met or exceeded their annual evaluation goals concerning, Area 4:  Addressing Disparities. 

Grantee activities focused on:

·      Provide information about dangers of tobacco use and secondhand smoke and the importance of tobacco laws
·      Encourage people to call and enroll with the Arkansas Tobacco Quitline (focus on African American and Hispanic women)
·      Help increase the number of voluntary smoke-free home policies
·      Encourage people to make smoke-free home pledges
·      Educate about tobacco disparities using focus groups and media campaigns (addressing tobacco-related health disparities (focusing on African American and Latino tobacco users)
·      Help to increase knowledge about tobacco policies and regulations regarding youth
·      Conduct media campaigns to increase awareness of tobacco dangers and tobacco industry advertising tactics using radio, social media, billboards, and flyers (disproportionately targeting minorities)
·      Educate about the hazards of secondhand smoke through a radio, print, and social media campaign (focusing on African American males)
·      Educate tobacco retailers about the laws associated with selling tobacco to underage minority youth
·      Encourage smoke-free policy or point of purchase policy to reduce advertising to youth and disparity groups
·      Help increase the number of minorities who acknowledge tobacco related disparities and the practices used by the tobacco industry to create these disparities
·      Educate underserved about the ills of tobacco products through surveys and media outreach

An evaluation dashboard for each grantee’s annual performance is provided to facilitate communication and collaboration among grantees, enhance MISRGO accountability, and build knowledge.  It is organized by CDC intervention area (see Appendix B). The MISRGO Grantee Activity Log provides an additional insight into the list of grantee activities across sites and intervention areas (see Appendix C).  

Technical Assistance

MISRGO grantees receive evaluation technical assistance throughout the year, including assistance monitoring and evaluating their progress.  

New and long-term grantees participated in an evaluation workshop on September 11, 2015, highlighting:  1) tobacco prevention resources; 2) establishing financial sustainability; 3) e-cigarettes; and 4) evaluation dashboards.  (See Fetterman, 2015 for details concerning approaches to establish financial stability and sustainability.)


Tobacco Prevention Resources

The evaluation resources shared with the grantees included: Campaign for Tobacco-Free Kids; County Health Ranking; County Health Calculator; Arkansas Department of Health; CDC State Tobacco Activities, Tracking, and Evaluation System; as well as the MISRGO Evaluation blog.



In addition, a chapter about 10 years of MISRGO tobacco prevention evaluation was published in October 2014.  The title is “Empowerment Evaluation and Evaluation Capacity Building in a 10-Year Tobacco Prevention Initiative.” The chapter was highlighted during the workshop in part to applaud the works of the group.  It was published in: Fetterman, D.M., Kaftarian, S. and Wandersman, A. (2014) Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.


Evaluation Dashboards

Grantees participated in another exercise to ensure the appropriate use of their evaluation dashboards, specifically to monitor their own progress throughout the year. The exercise highlighted the value of establishing goals, milestones, and baselines.  Once these were established, only the actual performance measures have been needed throughout the year – comparing actual performance with milestones and goals to determine if adequate progress has been made.


MISRGO and evaluation team members also use the evaluation dashboard to enhance accountability, signaling precisely when assistance is required to increase the probability of grantees accomplishing their end-of-year goals. 

The evaluation dashboards are designed to build capacity and produce outcomes.  They are in accordance with an empowerment evaluation approach.  (Fetterman, 2013; Fetterman and Wandersman, 2005; Fetterman, Kaftarian, and Wandersman, 2015.)

e-cigarette

The e-cigarette was discussed as well, highlighting the tobacco industry’s financial backing for the device.  In addition, the FDA and World Health Organization have called for stricter regulation of e-cigarettes.  Additional observations were made about the marketing of e-cigarettes during the workshop. It is no accident that the industry is using the same script used in the past to market cigarettes to now sell “e-cigarettes” to the public, e.g. use of celebrities, entertainment venues, and media.  This is part of the tobacco industry’s efforts to “normalize” the use of cigarettes in the public again. Recent statistics also highlight the dramatic increase in the use of e-cigarettes by youth.  The CDC has found that: “E-cigarette use tripled among middle and high school students in just one year” (CDC, 2015) 
  


Dialogue

In addition, grantees openly discussed obstacles and generously shared advice and best practices, based on their own experience.


Grantees are provided with additional tobacco prevention tools and updates to facilitate their programmatic and evaluative efforts between workshops.  For example, MISRGO’s TFA grantee built on our workshop discussion about smokeless tobacco by highlighting, in an email listserve, that the FDA announced that it was expanding its campaign to educate rural, white male teenagers about the negative health consequences associated with smokeless tobacco use (April 19, 2016).  Dangers include:  nicotine addiction, gum disease, tooth loss, and multiple kinds of cancer Their advertisements were posted in 35 U.S. selected markets. (See Appendix D for other examples of peer and evaluation team technical assistance communication.) 

Details concerning evaluation technical assistance are provided on the MISRGO tobacco prevention blog at: http://tobaccoprevention.blogspot.com

Conclusion

MISRGO grantees established their goals, milestones, and baselines during the 1st quarter.  In addition, they documented their actual performance for the 1st and 2nd quarter.  Between 69% and 82% of the grantees met[iii] or exceeded their 3rd Quarter evaluation milestones in each of the CDC intervention areas.  Between 77% and 86% of the grantees met or exceeded their annual goals.  The evaluation dashboards help grantees monitor their own performance.   They are being used to confirm effective performance and alert MISRGO and grantees concerning the need for mid-course corrections throughout the year.

The evaluation findings reported in this annual evaluation report provide evidence of MISRGO grantee dedication and commitment, concerning tobacco prevention.  They also document significant accomplishments concerning annual programmatic and evaluation goals.





References

Arkansas Department of Health (2009). Tobacco Prevention and Cessation Program
ARKANSAS STRATEGIC PLAN TO PREVENT AND REDUCE TOBACCO USE 2009 – 2014.  Little Rock, Arkansas:  Arkansas Department of Health, p. 3. http://www.healthy.arkansas.gov/programsServices/tobaccoprevent/Documents/TPCPStrategicPlan.pdf

Centers for Disease Control and Prevention (2015).  E-cigarette use triples among middle and high school students in just one year


Centers for Disease Control and Prevention National Center for Chronic Disease Prevention and Health Promotion (2014). Tobacco Use.  Targeting the Nation’s Leading Killer
- At A Glance 2011.  Atlanta, Georgia:  Centers for Disease Control and Prevention.

Centers for Disease Control and Prevention.  Prevention Status Report, 2013.  http://www.cdc.gov/psr/tobacco/2013/ar-tobacco.pdf

Campaign for Tobacco Free Kids (2013). Annual health care costs in Arkansas directly caused by smoking. http://www.tobaccofreekids.org/facts_issues/toll_us/arkansas

Fetterman, D.M. (2013).  Empowerment Evaluation in the Digital Villages:  Hewlett-Packard’s $15 Million Race Toward Social Justice.  Stanford:  Stanford University Press.

Fetterman, D.M. (2015).  MISRGO:  Sustainability Report (Empowerment Evaluation Workshop).  San Jose: Fetterman & Associates.

Fetterman, D.M. and Delaney, L. (2015).  MISRGO Empowerment Evaluation Annual Report 2014-2015.  San Jose:  Fetterman & Associates.

Fetterman, D.M., Delaney, L., Triana-Tremain, B., and Evans-Lee, M. (2014). Empowerment Evaluation and Capacity Building in a 10-Year Tobacco Prevention Initiative.  In Fetterman, D.M., Kaftarian, S., and Wandersman, A. (eds). Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.

Fetterman, D.M., Delaney, L., and Tremain, B. MISRGO Evaluation Annual Report 2013-2014.  San Jose:  Fetterman & Associates (July 11, 2014).

Fetterman,  D., Delaney, L., Tremain, B. (2013a).  MISRGO Evaluation Annual Report 2012-2013.  San Jose:  Fetterman & Associates (July 18, 2013).

Fetterman, D.M., Delaney, L., and Tremain, B. (2013b).  MISRGO Evaluation 2nd Quarter Progress Report 2013.  San Jose:  Fetterman & Associates.

Fetterman, D.M., Delaney, L., and Tremain, B. (2013c).  MISRGO Evaluation 3rd  Quarter Progress Report 2013.  San Jose:  Fetterman & Associates.

Fetterman, D.M., Kaftarian, S., and Wandersman, A. (2015).  Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.

Fetterman, D.M., Tremain, B., and Delaney, L. (2012).  MISRGO Evaluation Annual Report 2011-2-12.  July 12, 2012. San Jose: Fetterman & Associates.

Fetterman, D.M., Tremain, B., and Delaney, L. (2011).  MISRGO Empowerment Evaluation:  June 2011 Annual Report.  San Jose: Fetterman & Associates.

Fetterman, D.M. and Wandersman, A. (2005). Empowerment Evaluation Principles in Practice.  New York: Guilford Publication.




[i]  See CDC site:  http://www.cdc.gov/psr/tobacco/2013/ar-tobacco.pdf
[ii] Grantees either met or closely approximated their benchmarks.
[iii] Grantees either met or closely approximated their benchmarks.