Wednesday, November 5, 2014

Tobacco Prevention Empowerment Evaluation - 1st Quarter Report 2014-2015

MISRGO Evaluation
1st Quarter Report 2014-2015

Prepared for:            MISRGO, Grantees & Legislative Offices
Prepared by:             Dr. Fetterman and Ms. Delaney
Date:                           November 4, 2014
                       





Introduction

Tobacco use is the single most preventable cause of disease, disability, and death in the United States” (Center for Disease Control and Prevention, November 2014).  The annual health care costs in Arkansas directly caused by smoking is $1.2 billion, according to the Arkansas Department of Health (2009) and the Campaign for Tobacco Free Kids (2013).  Arkansas’ Master Tobacco Settlement revenue is being used in part to address this serious health issue.

 The University of Arkansas at Pine Bluff’s Minority Initiative Sub-Recipient Grant Office (MISRGO) has received Settlement funds through the Arkansas Department of Health to focus on tobacco use in minority communities. MISRGO’s mission is specifically to prevent and reduce tobacco use in minority communities.

MISRGO has awarded more than 50 Arkansas organizations with funding for tobacco prevention and cessation programs.  MISRGO currently sponsors and provides technical assistance for 15 grantees across the State of Arkansas.

MISRGO grantees have broad-based support across the State for their tobacco prevention and cessation work.  (See MISRGO Empowerment Evaluation: June 2011 Annual Report, Fetterman, Tremain, and Delaney, 2011.)
 
MISRGO grantees also have a strong track record of success based on past performance.  For example, in the last 3 year, the majority of MISRGO grantees met or exceeded their annual performance goals (see MISRGO Evaluation Annual Report 2013-2014 Fetterman, Delaney, and Tremain, July 11, 2014; 2012-2013 Fetterman, Delaney, Tremain, July 18, 2013a; and 2011-12 Fetterman, Tremain, and Delaney, July 12, 2012). 

Purpose

The purpose of this report is to present MISRGO tobacco prevention grantee progress at this point in the year - the end of the 1st Quarter 2014-2015.  The fiscal and programmatic end of the year is June 30, 2015.  (See Appendix A for a glossary of grantee names.)

MISRGO Grantee Intervention Areas

MISRGO grantees use the Centers for Disease Control and Prevention’s (CDC) approved and recommended intervention areas.  They include:

Area 1:  Eliminate Exposure to Secondhand Smoke
Area 2:  Preventing Initiation Among Youth and Young Adults
Area 3:  Promoting Quitting Among Youth and Adults
Area 4:  Addressing Disparities

Grantee progress and activities are reported on both a quarterly and annual basis, according to these CDC intervention areas. 
 
Quarterly Progress

During the first quarter (the focus of this report), MISRGO grantees established their goals, benchmarks, and baselines.   Goals are the planned outcomes or results at the end of the year, e.g. number of smoke-free parks.  Benchmarks are intermediate objectives or approximations of outcomes.  Baselines represent, for example, the number of smoke-free environments that existed before the grantee activity or intervention. 

During the second quarter, grantees report on their mid-year progress toward annual goals.  This provides grantees with an opportunity to reflect on their performance.  Based on these data, grantees continue as planned, make mid-course changes, and request assistance as needed.

The third quarter focuses on any gaps between benchmarks and annual goals.  Grantees share techniques with each other, MISRGO staff provide additional recommendations, and Fetterman & Associates notify grantees about gaps and make specific recommendations to close the gap. 

The fourth quarter is the annual comparison of their goals with their actual performance throughout the year.

1st Quarter Findings

100% of MISRGO grantees have established their goals, benchmarks, and baselines for the year.   Between 27% and 53% of the grantees have met[i] or exceeded their 1st Quarter evaluation benchmarks:

Area 1:  Eliminate Exposure to Secondhand Smoke  (33% met or exceeded 1st Qtr benchmarks)
Area 2:  Preventing Initiation Among Youth and Young Adults (27% met or exceeded 1st Qtr benchmarks)
Area 3:  Promoting Quitting Among Youth and Adults (53% met or exceeded 1st Qtr benchmarks)
Area 4:  Addressing Disparities (33% met or exceeded 1st Qtr benchmarks)






CDC Area 1:  Eliminate Exposure to Secondhand Smoke

33% of the grantees have met or exceeded their 1st Quarter evaluation benchmarks concerning Area 1:  Eliminate Exposure to Secondhand Smoke.

Grantee activities focused on:

·      Encourage voluntary smoke-free perimeter policies
·      Conduct secondhand smoke tobacco prevention messaging
·      Help to establish smoke-free park policies
·      Encourage tobacco-free policies
·      Encourage establishment of campus free ordinances and/or workplace smoking policies
·      Implement anti-smoking media campaigns (via radio, social media and television)
·      Encourage "No Smoking within 25 feet" policies
·      Promote smoke-free home and car campaign
·      Educate about the impact of marketing strategies
·      Educate about the benefits of smoke free environments in the car (ACT 811) and in the home

CDC Area 2:  Preventing Initiation Among Youth and Young Adults

27% of the grantees have also met or exceeded their 1st Quarter evaluation benchmarks concerning Area 2: Preventing Initiation Among Youth and Young Adults.

Grantee activities focused on:

·      Conduct Operation Store Front surveys
·      Conduct retailer visit regarding sales to minors
·      Encourage smoke-free or tobacco-free church campus policy
·      Conduct Open Storefront youth education campaign, including practice refusal skills when offered tobacco
·      Use media, include radio campaigns, social media and promotions, to inform youth about dangers of smoking
·      Educate youth about the effects of smoking
·      Encourage signed tobacco related pledges
·      Help youth reach "Past 30-day chewing tobacco use" goals
·      Help to implement point of purchase policy
·      Partner with organizations to prevent initiation of tobacco use by youth (presentations)
·      Encourage youth to pledge to be tobacco free
·      Train youth in prevention strategies through media outreach, gorilla marketing campaigns, and youth rallies

CDC Area 3:  Promoting Quitting Among Youth and Adults

53% of the grantees have met or exceeded their 1st Quarter evaluation benchmarks concerning Area 3:  Promoting Quitting. 

Grantee activities focused on:

·      Encourage use of Arkansas Quit-line Fax Referral System
·      Encourage youth to pledge to quit
·      Work with churches, schools and peers to help friends quit smoking
·      Partner with community organizations to host or take part in events, presentations, and/or activities to help people quit smoking
·      Help to increase the number of participants in cessation programs
·      Educate through radio/social media and print about the dangers of tobacco products
·      Educate people about cessation services
·      Present information (tobacco facts) concerning the effects of tobacco
·      Encourage people to participate in the established cessation programs
·      Educate about the dangers of tobacco through evidence-based strategies
·      Educate about cessation services using surveys, education activities, and media outreach

CDC Area 4:  Addressing Disparities

33% of the grantees have also met or exceeded their 1st Quarter evaluation benchmarks concerning Area 4:  Addressing Disparities. 

Grantee activities focused on:

·      Provide information about dangers of tobacco use, tobacco laws, dangers of secondhand smoke, and the importance of the Quitline (focus on Hispanics)
·      Encourage people to call and enroll with the Arkansas Tobacco Quitline (focus on African American and Hispanic women)
·      Help to increase the number of voluntary smoke-free home policies
·      Encourage people to make smoke-free home pledges
·      Educate about tobacco disparities using focus groups and media campaigns (addressing tobacco-related health disparities (focusing on African American and Latino tobacco users)
·      Encourage people to call the Quit Line (focusing on homeless, LGBT, substance abusers and aging populations)
·      Help to increase knowledge about tobacco policies and regulations regarding youth
·      Conduct media campaigns to increase awareness of tobacco dangers and tobacco industry advertising tactics using radio, social media, billboards, and flyers
·      Encourage smoke-free home/car pledges
·      Educate about the hazards of secondhand smoke through a radio, print and social media campaign (focusing on African American males)
·      Encourage people to participate in a focus group and use Facebook to learn about tobacco ads and tactics as well as facts about tobacco use
·      Educate tobacco retailers about the laws associated with selling tobacco to underage youth
·      Educate underserved on the ills of tobacco products
·      Recruit youth to become community advocates
·      Encourage smoke-free policy or point of purchase policy to reduce advertising to youth and disparity groups
·      Help to increase the number of minorities who acknowledge tobacco related disparities and the practices used by the tobacco industry to create these disparities
·      Educated underserved about the ills of tobacco products through surveys and media outreach

An evaluation dashboard for each grantee’s annual performance is provided to facilitate communication and collaboration among grantees, enhance MISRGO accountability, and build knowledge.  It is organized by CDC intervention area (see Appendix B). The MISRGO Grantee Activity Log provides an additional insight into the list of grantee activities across sites and intervention areas (see Appendix C).  

Technical Assistance

MISRGO grantees receive evaluation technical assistance throughout the year, including assistance monitoring and evaluating their progress.  

New and long-term grantees participated in an evaluation workshop on October 24, 2014, highlighting:  1) tobacco prevention resources; 2) the use of pre- and post-tests; 3) e-cigarettes; and 4) evaluation dashboards. 





Tobacco Prevention Resources

The evaluation resources included: Campaign for Tobacco-Free Kids; County Health Ranking ; County Health Calculator; Arkansas Department of Health; CDC State Tobacco Activities, Tracking, and Evaluation System; as well as the MISRGO Evaluation blog.


In addition, a chapter about 10 years of MISRGO tobacco prevention evaluation was published in October 2014.  The title is “Empowerment Evaluation and Evaluation Capacity building in a 10-Year Tobacco Prevention Initiative.” The chapter was published in: Fetterman, D.M., Kaftarian, S. and Wandersman, A. (2014) Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage. 


Use of Pre- and Post-Tests

Grantees participated in online survey before and after lunch to demonstrate the value of pre- and post-tests.  Grantees were asked to rate how hungry they were before lunch and then after the intervention (lunch) to assess their hunger level again.  We used SurveyMonkey to demonstrate the utility of this technique. The same approach was recommended for grantee tobacco prevention awareness sessions, to document knowledge acquisition or experiential change in attitude.


Evaluation Dashboards

Grantees participated in another exercise to ensure the appropriate use of their evaluation dashboards, specifically to monitor their own progress throughout the year. The exercise highlighted the value of establishing goals, baselines, and benchmarks.  Once these are established only the actual performance measures are needed throughout the year – comparing actual performance with benchmarks and goals to determine if adequate progress has been made.


MISRGO and evaluation team members also use it to enhance accountability, signaling precisely when assistance is required to increase the probability of grantees accomplishing their end-of-year goals. 

The evaluation dashboards are designed to build capacity and produce outcomes.  They are in accordance with an empowerment evaluation approach.  (Fetterman, 2013; Fetterman and Wandersman, 2005.)

e-cigarette

The e-cigarette was discussed as well, highlighting the tobacco industry’s financial backing for the device.  In addition, the FDA and World Health Organization have called for stricter regulation of e-cigarettes.  Additional observations were made during the workshop, including that it is no accident that the e-cigarette looks like a cigarette and that this is part of the tobacco industry’s efforts to “normalize” the use of cigarettes in the public again.


Dialogue

In addition, grantees openly discussed obstacles and generously shared advice and best practices, based on their own experience.
 


Grantees are provided with additional tobacco prevention tools and updates to facilitate their programmatic and evaluative efforts  (See Appendix D for an example of a periodic technical assistance communication.) 

Details concerning evaluation technical assistance are provided on the MISRGO tobacco prevention blog at: http://tobaccoprevention.blogspot.com

Conclusion

MISRGO grantees have established their goals, benchmarks, and baselines.  In addition, they have documented their actual performance.  Between 27% and 53% of the grantees have met[ii] or exceeded their 1st Quarter evaluation benchmarks in each of the CDC intervention areas.  The evaluation dashboards help grantees monitor their own performance.   They are being used to confirm effective performance and alert MISRGO and grantees concerning the need for corrections throughout the year.

The evaluation findings reported in this 1st Quarter evaluation report provide evidence of MISRGO grantee dedication and commitment, concerning tobacco prevention.  They also document progress toward annual programmatic and evaluation goals.





References

Arkansas Department of Health (2009). Tobacco Prevention and Cessation Program
ARKANSAS STRATEGIC PLAN TO PREVENT AND REDUCE TOBACCO USE 2009 – 2014.  Little Rock, Arkansas:  Arkansas Department of Health, p. 3. http://www.healthy.arkansas.gov/programsServices/tobaccoprevent/Documents/TPCPStrategicPlan.pdf


Centers for Disease Control and Prevention National Center for Chronic Disease Prevention and Health Promotion (2014). Tobacco Use.  Targeting the Nation’s Leading Killer
- At A Glance 2011.  Atlanta, Georgia:  Centers for Disease Control and Prevention.

Campaign for Tobacco Free Kids (2013). Annual health care costs in Arkansas directly caused by smoking. http://www.tobaccofreekids.org/facts_issues/toll_us/arkansas

Fetterman, D.M. (2013).  Empowerment Evaluation in the Digital Villages:  Hewlett-Packard’s $15 Million Race Toward Social Justice.  Stanford:  Stanford University Press.

Fetterman, D.M., Delaney, L., Triana-Tremain, B., and Evans-Lee, M. (2014). Empowerment Evaluation and Capacity Building in a 10-Year Tobacco Prevention Initiative.  In Fetterman, D.M., Kaftarian, S., and Wandersman, A. (eds). Empowerment Evaluation:  Knowledge and Tools for Self-assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.

Fetterman, D.M., Delaney, L., and Tremain, B. MISRGO Evaluation Annual Report 2013-2014.  San Jose:  Fetterman & Associates (July 11, 2014).

Fetterman,  D., Delaney, L., Tremain, B. (2013a).  MISRGO Evaluation Annual Report 2012-2013.  San Jose:  Fetterman & Associates (July 18, 2013).

Fetterman, D.M., Delaney, L., and Tremain, B. (2013b).  MISRGO Evaluation 2nd Quarter Progress Report 2013.  San Jose:  Fetterman & Associates.

Fetterman, D.M., Delaney, L., and Tremain, B. (2013c).  MISRGO Evaluation 3rd  Quarter Progress Report 2013.  San Jose:  Fetterman & Associates

Fetterman, D.M., Tremain, B., and Delaney, L. (2012).  MISRGO Evaluation Annual Report 2011-2-12.  July 12, 2012. San Jose: Fetterman & Associates.

Fetterman, D.M., Tremain, B., and Delaney, L. (2011).  MISRGO Empowerment Evaluation:  June 2011 Annual Report.  San Jose: Fetterman & Associates.

Fetterman, D.M. and Wandersman, A. (2005). Empowerment Evaluation Principles in Practice.  New York: Guilford Publication.

Appendices Available Upon Request


[i] Grantees either met or closely approximated their benchmarks.
[ii] Grantees either met or closely approximated their benchmarks.

Monday, November 3, 2014

1st Quarter Empowerment Evaluation Tobacco Prevention Workshop - October 24, 2014

Empowerment Evaluation Tobacco Prevention Workshop
October 24, 2014
1st Quarter 

Minority Sub-recipient Grant Office
University of Arkansas Pine Bluff 
Pine Bluff, Arkansas

This was the 1st workshop of the year.  It included long-term grantees and new folks as well.


We covered 4 main topics: 

1. resources, 
2. e-cigarettes, 
3.  evaluation dashboards, and 
4. pre- and post tests.


1.  RESOURCES.  Tobacco prevention and evaluation resources were discussed, such as:  









40 days to Freedom Tobacco Cessation Program
(faith-based program)


Highlighting Our 10th Year Conducting a Tobacco Prevention Empowerment Evaluation


Fetterman, D.M., Delaney, L., Triana-Tremain, B.,  Evans-Lee, M., (2014).  Empowerment Evaluation and Evaluation Capacity Building in a 10-Year Tobacco Prevention Initiative.  In Fetterman, D.M., Kaftarian, S., and Wandersman, A. (eds.).  Empowerment Evaluation:  Knowledge and Tools for Self-Assessment, Evaluation Capacity Building, and Accountability.  Thousand Oaks, CA: Sage.

2.  E-CIGARETTES. We shared insights and references concerning the use of e-cigarettes. (One of the primary insights included how the tobacco industry is using e-cigarettes to re-normalize cigarettes - "there is a reason they look like cigarettes").



3.  EVALUATION DASHBOARDS.  Evaluation dashboards are used by all grantees to help them monitor their own progress throughout the year

Evaluation Dashboard 
(3rd Quarter 2013 Example)


Grantees took pride in their accomplishments.  One of the most powerful moments in the workshop was when we shared with the group how one grantee's actual performance exceeded their 1st quarter benchmarks.  That grantee shouted out: "that's me."  They appreciated having the group hear how well they were doing.  They also appreciated assistance from the group, when they needed help to improve their efforts.

4.  PRE- AND POST-TESTS

We demonstrated how a pre- and post-test works by using online survey software (SurveyMonkey). 

 We asked the group to rate how hungry they were before lunch.
Then we had lunch (the intervention).
Then we asked the group how hungry they were after lunch.


What a surprise - a big decrease in hunger after the intervention (lunch)!

We took a computer screen snapshot of the survey findings and juxtaposed the pre and post-test results to highlight the change.



We also looked at pre and post-test data concerning other variables and discussed why we would not expect others findings (unrelated to the lunch intervention) to change or move the needle.

What's Working and Not Working 

We also had time to share what's working and not working so that we could help each other out.






Tech Tools

We concluded the workshop with a variety of tech tools to enhance our work, including: word clouds, online survey software, CDC infographics, and data visualization software.

Wordle or Word Clouds

Online Survey Software (SurveyMonkey)








Conclusion

It was a fun filled workshop, packed with information and ideas.



See you next time.