MISRGO Dissemination "Jackpot" - 2009

Dissemination "Jackpot" 2009. One of the charges of the evaluation is dissemination - getting the word out about MISRGO work. An effort is made throughout the year to disseminate the findings and the tools used in the evaluation. However, November and December 2009 represented rare opportunities for maximum exposure and dissemination of MISRGO work.

In addition to highlighting the MISRGO evaluation in a recent professional association training workshop (in the picture above), the three key dissemination outlets were:

1. Professional Association Annual Meeting Presentation
2. Scholarly Book Publication
3. World-wide Recognition Notice About Scholarly Article Publication

Professional Association Presentation

Dr. Fetterman presented many of the technological tools employed in the MISRGO empowerment evaluation at the American Evaluation Association (November 2009).

The presentation was called:

Empowerment Evaluation: Technological Tools of the Trade

The following tools were discussed:

Wordle. Wordle is a powerful qualitative data analysis breakthrough. It is a form of content analysis that produces a visual image of the frequency of key words used in a text. The Wordle image below highlights comments about a MISRGO evaluation workshop. This tool, among others, has been introduced to the MISRGO grantees and evaluators (to help them gain an additional insight into the effectiveness of their programs). It is also used to communicate evaluation findings in a user-friendly and visually appealing manner. In addition, the use of MISRGO evaluation online surveys was discussed.

Digital Photography. The use of digital photography was also explored, using MISRGO case examples. The examples ranged from, a Clearing the Air Conference in Communities of Color to a MISRGO evaluation workshop.

Blogs. Blogs, like this one, were also shared to emphasize the principles of transparency and openness associated with the MISRGO evaluation.

Scholarly Book

Dr. Fetterman published the 3rd edition of Ethnography: Step by Step. This updated version of his popular methods book highlighted some of the MISRGO evaluation work. (The timing of the book revision represented a window of opportunity to highlight MISRGO evaluation activity.)

The examples highlighted in his book include: the Tobacco Prevention (MISRGO) blog, the MISRGO Empowerment Evaluation Collaborative Web Page, and the use of MISRGO online surveys.


World-wide Recognition About Scholarly Article

Sometimes exposure is a byproduct of a publication or presentation, instead of the presentation or publication itself. In this case, the American Evaluation Association recognized the popularity of Drs. Fetterman and Wandersman's 2007 article titled, Empowerment Evaluation: Yesterday, Today, and Tomorrow.


According to the American Evaluation Association, their article was the most accessed American Journal of Evaluation online.


Since the article highlighted MISRGO evaluation work, it not only documented a very effective dissemination strategy, it also served as a "marketing tool" to enhance exposure.

[Reproduced from the American Journal of Evaluation,
Vol. 28, No. 2, June 2007, pp. 179-198]

MISRGO Mapping Project

Grantee Map. We have a shared map called the:

Tobacco Prevention in Arkansas - MISRGO Map.

You can view and edit this map (if you are a MISRGO tobacco prevention grantee or evaluators) at:

http://maps.google.com/maps/ms?ie=UTF8&hl=en&oe=UTF8&msa=0&msid=118091722891591549115.000475d6af7a195c624e2

Inspired by Sarah Hammer's map work - in which she highlights the smoke free places in her area - and Dr. Tarek's workshop on using maps, a map of all of us was created. In addition to letting us know where we are across the State, it provides us with each others' contact information to facilitate sharing and assistance.

Take a look and feel free to edit any of the information on the maps - make any corrections and additions. Right now, however, this information will help us help each other as we work on our baseline evaluation efforts.

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Note 1: For a tutorial on creating your own map of locations of interest go to the following site:
http://maps.google.com/support/bin/answer.py?hl=en&answer=67842

Click here for an abbreviated tutorial with computer screen snapshots

You may want to map out tobacco free areas, tobacco billboards, stores with multiple violations, and so on in your county. This is a very useful tool. You can map out resources in the community including smoke cessation programs, as well as negative influences concerning the use of tobacco by youth.

Note 2: You should already be able to edit this map. However, if not be sure to reply to the posting that was sent to you on October 13, 2009 and accept Dr. Fetterman's invitation to become a collaborator on this map.

If you don't have a Google account, you can create one at
http://www.google.com/accounts/NewAccount?reqemail=.

Baseline, Benchmark, and Goals Workshop

Baseline, Benchmark, and Goals Workshop. Our September 24, 2009 evaluation workshop received rave reviews. A few grantee and evaluator comments included:

"It wet our appetite for more".
"This is precisely what we needed."
"Timely."
"Clear."

MISRGO staff also valued the day: "We've gotten good reviews about the workshop. Thanks so much for all you do!!!"

Dean Johnson

Dean Johnson took time out of his busy schedule to join us and help strategize about next steps in the evaluation.

Dean Calvin Johnson

Three Evaluation Speakers/Facilitators

We had three speakers/facilitators for the day. The day began with Dean Stewart Donaldson from Claremont Graduate University. He facilitated a session entitled:

Practical Program Evaluation: A Program Theory Approach

He highlighted the role of program theory. It is used to make sure everyone agrees with the logic of their program in an evaluation.For example, if you did X (connected them with the Quitline and heard back from them) it would be likely to produce Z (help people stop smoking).
He conducted an exercise where everyone was asked to evaluate the room. It was practical and down-to-earth and emphasized the role of an individual's perspective in evaluation.

Grantee and Evaluator Presenting Insights Based on Dr. Donaldson's Exercise

The second facilitator was Dr. Tarek Azzam, also from Claremont. He presented the tech session. It was titled:

Using Technology to Enhance Applied Research & Evaluation

He highlighted the use of technology to conduct evaluations and to more effectively communicate about evaluation findings.
Tarek covered the use of: GIS, online surveys, and various qualitative data display tools.

Use of GIS to Identify Problem Issues and Areas: Tobacco Billboards & Income

Qualitative Data Display: Frequency of Words Used Visually Represented in a Word Cloud

The third speaker was Dr. David Fetterman, from Fetterman & Associates. He built on the first two speakers ideas and techniques. Dr. Donaldson focused on program theory - does the program make sense as designed. Dr. Tarek highlighted the use of technology to display evaluation findings.

Dr. Fetterman highlighted a visually powerful way of highlighting grantee program baselines, benchmarks, and goals. This will be the focus of next year's reporting - to help us all keep track of our progress throughout the year and for the next 5 years. The title of his session was:

Baseline, Benchmarks, and the Visual Alert System

The MCCTFA Baseline and Quarterly Evaluation Assessment Form was used as a case example. Dr. Fetterman explained how to pull the baseline figure from the narrative. Then he demonstrated how to identify the goal, for example, to increase the number of public buildings with smoke free policies by 15%. From there it was simply a matter of pulling the current year objective, 5%, and establishing benchmarks for the next 5 years. The slides for his presentation are available here online. (Select Baseline.)

Dr. Fetterman presented a translation of the evaluation language we are using at this stage of the empowerment evaluation to help make the process jargon free:

Normal Question (Evaluation Terms)

• Where are you now? (Baseline)
• Where do you want to go? (Objective or Goal)
• How do you plan to get there? (Activities or Strategies)
• How do you know where you are along the way? Are we there yet? (Benchmarks)
• How do you know when you have arrived at your destination? (Comparison of baseline with long-term objective or outcome)

Here is a bar chart that the group used as an example of how to visually report their own work.

Bar Chart: Baseline, Benchmarks, and Goal

Dr. Fetterman providing one-on-one technical assistance

Dr. Fetterman, Ms. Evans, and Ms. Delaney went from table to table, helping grantees and evaluators link activities to their baselines and benchmarks. It was a rewarding and productive day.

House & Senate Health Services Subcommittees Report

House & Senate Health Services Committee report. MISRGO presented the "MISRGO Best Practices & Outcomes" report to the House & Senate Health Services Committee of the House and Senate Public Health, Welfare, and Labor Committees on August 4, 2009. (It is available online here and on the right column.)

The report summarizes the best practices and outcomes of the Minority Initiative Sub-Recipient Grant Office's (MISRGO) tobacco prevention efforts. It begins with:

$94 Million Saved in Excess Medical Costs

MISRGO grantees have saved the State of Arkansas $94,944,281.96 in excess medical costs. This is a significant outcome by any standard. It is also an increase from previous years.

The formula for calculating this savings is straightforward. The figures used in the formula come from the Arkansas Department of Health, Campaign for Tobacco-Free Kids, Centers for Disease Control and Prevention, and the U.S. Census Bureau.

The formula is the number of people who stop smoking, times the per person cost of smoking in the State of Arkansas. Then each grantee’s total is added together to produce the final reported figure. The dollars saved are in terms of excess medical costs saved the State.

A table of the formula is presented below:

A complex formula is used to provide precise data by county. However, a simpler approach can be used to calculate the dollars saved. The overall State average can be used to calculate the excess medical costs saved. The formula is based on the following information:

1. $812 million annual health care costs in Arkansas directly caused by smoking

Source: Campaign for Tobacco-Free Kids (The New York Times Magazine called the Campaign the "the country's leading anti-tobacco lobby.")

2. 4900 people who die each year from their own smoking in Arkansas

Source: Campaign for Tobacco-Free Kids

3. $812,000,000 divided by 4900 = $165,714 approximate per person cost

4. $165,714 times 572 (the number of people who have quit smoking as a result of grantee efforts) = $94,788,408 saved in terms of excess medical cost

STATE-WIDE SUPPORT FOR MISRGO TOBACCO PREVENTION PROGRAMS

A state-wide survey was conducted to solicit community views of MISRGO tobacco prevention programs. The most significant survey finding was the overwhelmingly positive value placed on these tobacco prevention programs in communities throughout the State. When asked if:

“Tobacco prevention & education programs are a good use of public money?”

The response from an overwhelming majority of Arkansans sampled was positive (98% answered yes).

In addition, community members believe grantees’ tobacco and prevention efforts have been effective in PREVENTING people from using tobacco and in getting people to STOP using tobacco.

CDC RECOMMENDED PROGRAM ACTIVITIES

MISRGO grantee programs use CDC recommended tobacco prevention activities. These activities contribute to the outcome data reported earlier, e.g. reduction in the number of smokers in the State and the millions saved in terms of excess medical expenses. A series of bar charts illustrate how MISRGO monitors and supports these efforts across programs and grantees. This tool has also been useful for grantees, placing their activities in comparison with other grantees. It has influenced them to "join in" the collective effort, increasing the "dose" effect with more grantees working on the same tasks and concerns.

The bar charts below provide an insight into the activities held in common across grantees, eg. minority initiatives, coalition building, Quitline promotion, newspaper articles, radio and TV advertisements, and educational materials.

Detailed findings of program activities are presented in the following report: Introducing the Visual Alert System: A Graphical Analysis of the MISRGO Mid-Year Report (Fetterman, Fitch, and Delaney, 2008).

TRAINING

Training is a critical component of MISRGO grantee effectiveness. Grantees and evaluators continually learn more about effective tobacco prevention practices. They also receive evaluation training in order to monitor and assess their effectiveness. (See earlier postings on this site for more information about the evaluation training grantees and evaluators receive to support their efforts.)

The pictures below highlight grantees and evaluators engaged in self-evaluation exercises and Dr. Fetterman providing training in interviewing and observation skills.

POLITICAL SUPPORT FOR THE PROGRAM AND EVALUATION TRAINING

Political support for the program is extensive including:

• University of Arkansas academic officials (Drs. Johnson and Benjamin)
• Senators (Henry "Hank" Wilkins, IV and Tracey Steele)
• Representatives (Stephanie Flowers and Earnest Brown)

MISRGO has received a citation from the State of Arkansas for its evaluation training.

DISSEMINATION OF WORK

One of MISRGO's mandates is to disseminate promising practices, evaluation findings, and program outcomes. MISRGO has made presentations at professional meetings such as the following:

In addition, MISRGO staff and colleagues have published articles and books concerning "lessons learned."
CONCLUSION

The report summarizes many of MISRGO's tobacco prevention best practices and outcomes. A few of these practices and outcomes include:

• 94 million saved in excess medical costs between 2005-present
• 94% engaged in minority initiatives to prevent tobacco consumption
• 94% using educational materials as part of their tobacco prevention campaigns
• 88% engaged in coalition building to combat tobacco use
• 76% promoting the use of Quitline
• 76% using radio and TV ads to combat tobacco use

MISRGO has provided strong leadership and technical assistance and grantees and evaluators remain vibrant and committed. The prognosis is good. MISRGO and its grantees are well positioned to continue to have a positive impact on the minority community in Arkansas in the foreseeable future.

2008-2009 Annual MISRGO Evaluation Report

2008-2009 Annual MISRGO Evaluation Report. The report was issued July 11, 2009. The 2008-2009 MISRGO Evaluation Annual Report highlights many of the accomplishments of the year. They include a more focused view of evaluation objectives and the power of consolidating grantee interests and activities.

The Evaluation Monitoring System and Visual Alert System are tools that have been successfully employed in the field and sparked an interest in common data collection and reporting efforts in the future.

Ironically, the evaluation tools themselves have helped unite a diverse group of grantees and evaluators and initiate a dialogue about the power of speaking with a single united evaluation voice.

This report also discusses the range of evaluation training provided to build evaluation capacity among grantees and evaluators, ranging from formal workshops and exercises to informal one-on-one critiques and consultations. The aim of these training exercises is to enhance the quality of the current tobacco prevention work. It is also designed to enhance the quality of tobacco prevention evaluators in the State, building a stronger and more sustainable evaluation workforce for the future.

April 30, 2009 - EMS Online Collaborative Web Site Workshop

Evaluation Monitoring System Collaborative Web Site Training Workshop.

We focused on how to calculate the excess medical cost saved by each grantee.

Then we demonstrated how to post the bar charts, pictures, text, and related information on the Collaborative Web Site.  This web site allows each of us to create and update our own web page on our shared site.  The workshop was a useful training opportunity for new grantees/evaluators and a useful refresher for those who already generate the Evaluation Monitoring System information and use the collaborative site.  (The first page summarizes our collective data and can be seen in the top left corner.  Collectively we have saved the State of Arkansas over $94 million in excess medical costs (by helping people stop using tobacco).

Below is the Contact Information page of our collaborative web site, highlighting the individual accomplishments of our grantees (as compared with our collective data in the top left corner of this blog).

Dr. Calvin Johnson, Dean of the School of Education at the University of Arkansas at Pine Bluff, welcomed the grantees and evaluators at the workshop (see below).

Evaluators and grantees contemplated the potential of this site and the power of combining our data instead of reporting only by each individual grantee.

We had a guest speaker during the workshop training.  Oluwakemi Talabi (Kemi) is a Tobacco Cessation Program Analyst from the Arkansas Department of Health.  She provided us with an update on the type of information we could expect to receive from the new vendors operating the Quitline.  It was impressive and and excellent opportunity to link up with the Quitline contact people in the Tobacco Prevention and Cessation Program in the Arkansas Department of Health.  (See the contact information cards below.)

This is Kemi speaking to the group (below).  She was gracious and informative.

The Clearing the Air in Communities of Color Conference was held concurrently with the evaluation training workshop.  We were all able to attend parts of the conference including the skits (see below) which were very effective tools to reach minority youth.

After the skits concluded, many of the grantees and evaluators reconvened and discussed next steps with Dr. Fetterman.  Dr. Fetterman explained that he would be providing his comments on each of the mid-year reports (which he has already completed).  In addition, he would continue to provide additional technicals assistance until the Evaluation Monitoring System and Visual Alert System were used consistently by each grantee.  Dr. Fetterman also agreed to continue to provide additional one-on-one assistance concerning posting on the Collaborative Web Site (so that the information would be up-to-date and visible to the public).

Summer Evaluation Workshop

Evaluation Workshop. The free summer evaluation workshop was considered a tremendous success, according to participants, workshop facilitators, and legislative officials. Inquiries came as from as Ethiopia and Liberia. However, most of the interest came from Arkansas and the mid-west. There were approximately 100 participants.

The workshop began with welcome messages from University of Arkansas at Pine Bluff academic officials, including Drs. Johnson and Benjamin. Mayor Redus also welcomed workshop participants.

Dr. Fetterman provided an outline of the workshop, including plenary speakers and workshop facilitators. He also highlighted the local Arkansas evaluation group's meeting during the workshop.

Plenary speakers included: Senator Tracey Steele, Representative Flowers, Kevin Collins (Centers for Disease Control), Charles Gasper, Missouri Foundation for Health, and Charlotte Lewellen-Williams (University of Arkansas Clinton School of Public Service).

The House of Representatives presented a Citation for our work during the workshop:

The topics and speakers were as follows:

• Dr. Stewart Donaldson - Practical Program Evaluation
• Dr. Tarek Azzam - Technological Tools
• Dr. David Fetterman - Empowerment Evaluation
• Dr. Rosalie Torres - Interviewing Skills
• Dr. Abraham Wandersman - Getting to Outcomes

The Arkansas Group of Evaluators held its inaugural meeting during the workshop (at night). The chair, Linda Delaney, orchestrated the evening meeting and plans to continue to build this organization (an official local affiliate of the American Evaluation Association).

Center for Disease Control and Prevention literature was made available to participants free of charge, including:

• Best Practices for Comprehensive Tobacco Control Programs
• Introduction to Process Evaluation in Tobacco Use Prevention and Control
• Introduction to Program Evaluation for Comprehensive Tobacco Control Programs
• Pathways to Freedom: Winning the Fight Against Tobacco

A detailed portrait of the workshop is provided at the Arkansas Evaluation Center blog.

MISRGO Annual Evaluation Report 2008

Report. The Annual Report provides a brief summary of evaluation activities and accomplishments throughout the year. The most significant achievement by the group is presented in their Evaluation Monitoring System. To-date, the group has save the state $94,944,281.96 in excess medical costs. This is an increase of approximately $10,188,113.56 since last year. In addition, the Visual Alert System is being implemented this year. It involves the use of bar charts to graphically illustrate what kinds of activities the grantees are engaged in, as well as the frequency of these activities - across grantees. Additional accomplishments include workshop training, blogs, and publications. Plans for the Summer Evaluation Workshop (which is part of MISRGO training efforts) have been extensive, tailoring national talents to local needs. It has attracted prospective participants from as far away as Liberia and Ethiopia, as well as throughout the United States. It will be held July 10-11, 2008. In addition to these efforts, a certificate and masters program in evaluation is being planned. For a more detailed account of evaluation efforts see the complete report or contact Dr. Fetterman.

Clearing the Air in Communities of Color 2008

Clearing the Air. This year's conference began with Bishop Miles and a talk about "A Vision of a Smokefree Society." The theme was maintained throughout the conference with Kevin Collins (CDC) emphasizing capacity building and Onjewel Smith discussing tobacco industry techniques. Success stories were shared including presentations by Elroy Brown, Larry Owens, and Carolyn Yarbrough. Dr. David Fetterman provided a status update and workshop about MISRGO's empowerment evaluation efforts. He highlighted both the Evaluation Monitoring System and the Visual Alert Systems. Other speakers included: Lloyd Franklin (law enforcement), Ruby Trotter (youth), Misty Paschall (parent), and Bob Mathis (state government). Special presentations were made by: Wilma Houston (Asian Pacific), J.R. "Pinky" Few (Rural Community), Keisha Grigsby (African American Community), and Ruben Arana (Hispanic Community), as well as Advantage Communications. Dr. Benjamin helped to open and close the conference with the AR State PRIDE Youth Performance.

Spring Evaluation Training Workshop - April 24, 2008

Spring Evaluation Training Workshop

Grantees and evaluators came together on April 24, 2008 to participate in evaluation training. Bringing both of them together ensures that they are on the same wave-length about data needs, roles, and relationships. (A copy of the slides used to guide the training day are available on this blog.)

The training was geared toward a new monitoring tool – the Visual Alert System. (This tool is described in the Visual Alert System posting below this one.)

This tool helps monitor grantee progress across programs, providing a readily available update upon demand. For example, bar charts are used to visually document how many grantees are using the Quitline.In addition, grantees and evaluators agreed that the same bar chart format could be used to document individual grantee progress. For example, a grantee would have:

1. The first bar chart would document their baseline of 4 news articles printed so far this year.
2. The second bar chart would document their aspirations – a benchmark number such as 10 new news articles by the end of the year.
3. A third bar chart would document the actual number of articles they had published that year, which might be 8.
4. A fourth chart would compare the first baseline bar chart of 4 with a final end of the year bar chart of 8 – documenting their progress throughout the year.

These bar charts could be forwarded to MISRGO and Dr. Fetterman and Ms. Delaney to document progress across grantees based on activities held in common, such as the Quitline or newspaper articles.


Dr. Johnson, Dean of the School of Education, was present to introduce new points and reinforce existing recommendations, including the need include activities directly related to tobacco prevention efforts even if the activities were not originally listed in the plan. In addition, he reminded grantees of the need to seek external funding.


The Visual Alert System provided an insight into patterns across grantees, alerting grantees and evaluators to successes that should be reinforced and continued, as well as areas meriting attention. The issue of whether we were implementing the “right” or most effective activities was discussed.

The Center for Disease Control document concerning Best Practices helped guide this part of the discussion. A summary of best practices was distributed during the workshop. CDC’s complete document is available on line.
One of the Visual Alert System findings initially suggested that few grantees were applying CDC Best Practices. However, after some discussion it was clear that grantees were in fact applying CDC Best Practices without knowing it, e.g. Quitline, coalition building, and so on. This finding and discussion highlighted the importance of sharing CDC documentation and engaging in a dialogue about what was being implemented in practice.

The Evaluation Monitoring System was also emphasized during the workshop, highlighting the importance of the group continuing to work together as a group. This system has demonstrated how much money they have saved as a unified tobacco prevention coalition - $84 million.











The workshop concluded with a few announcements including the:

1. establishment of the Arkansas Evaluation Center
2. development of an evaluation certificate program (and master’s program) at UAPB
3. upcoming Arkansas Summer Evaluation Institute Workshop – July 10th and 11th

The group responded enthusiastically to the announcements.

Dr. Fetterman pointed to the LCD projection to let them know the details associated with the Summer Evaluation Workshop.

The Summer Evaluation Workshop will include:

• Senator Tracey Steele
• CDC’s Health Education Specialist Kevin Collins
• Dr. Stuart Donaldson, Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University - Introduction to Practical Program Evaluation (Program Theory)
• Dr. Tarek Azzam, Senior Research Associate, School of Behavioral and Organizational Sciences, Claremont Graduate University - Introduction to Technology and Evaluation: Focus on the use of Geographic Information Systems and Online Surveys
• Dr. Rosalie Torres, Torres Consulting Group - Interviewing Skills for Evaluators
• Dr. Fetterman, Director of Evaluation, School of Medicine, Stanford University and Co-Director, Arkansas Evaluation Center, School of Education, UAPB
• Mr. Charles Gasper, Director of Evaluation, Missouri Foundation for Health - A Foundation Perspective about the Significance of Program Evaluation
• Dr. Nacaman Williams, Walton Foundation – A Foundation Perspective about the Significance of Program Evaluation

The Summer Evaluation Workshop Program and Schedule is listed in under Useful Links on the left column of the blog. For your convenience, the program is also available by selecting this Summer Workshop Program 2008 link.

The Spring Evaluation Workshop concluded with an announcement about the new Arkansas Group of Evaluators. AGES is an official affiliate of the American Evaluation Association.

Their first official meeting as an affiliate will be on July 10th in the evening at UAPB. It will be held during the Summer Evaluation Workshop. Ms. Linda Delaney is the chair of the State-wide evaluation group.

Visual Alert System

Visual Alert System. We have developed a system that enables us to monitor our progress quickly. We are using a simple visual tool - bar charts. The MISRGO Mid-Year and Annual Evaluation Reports can be difficult to navigate. There are 18 reports covering varying topics ranging from coalition building to promoting the use of the Quitline. It is a time consuming and laborious task to compare grantee efforts throughout the State by reading the text in each report alone. We have applied this tool to the current Mid-Year Evaluation Reports (see Visual Alert System listed on the right hand column of this blog). The bar charts enable us to quickly scan all of the reports and determine how many of us are building coalitions, using the Evaluation Monitoring System, working with churches to promote tobacco prevention activities, and so on. Grantees can look at the bar charts and decide to shift their efforts to strengthen the groups' efforts as a collective. Policymakers and legislators can also see where we are putting our emphasis to meet our goals. We will provide training on the use of this tool during our upcoming training session (April 24, 2008). This tool among others will be highlighted during our Arkansas Summer Evaluation Institute Workshop, which will be held at the University of Arkansas at Pine Bluff on July 9-10, 2008.

Evaluation Technology Workshop Sept 27, 2007

Evaluation Technology Workshop. MISRGO provided a technology training day at the University of Arkansas at Pine Bluff. The training was designed to familiarize grantees and evaluators with a variety of web-based tools that will help them with their programs and assessments of their programs.

Dr. Fetterman was invited to facilitate the session.

He provided an overview of useful tools, including online surveys, blogs, digital photography, picture file sharing, internet telephony (Skype), online documents and spreadsheets, and videoconferencing on the net.

Grantees and evaluators will be using the online spreadsheet to keep track of the Evaluation Monitoring System data. The Evaluation Monitoring system keeps track of the number of people who have stopped using tobacco as attributed to grantee efforts and translates that figure into dollars saved concerning excess medical costs for the State). This will allow MISRGO, grantees, and evaluators to access the data on demand.



The day was energizing as we interactively created online surveys and blogs and demonstrated the power of the net.


We also demonstrated the value of online calendars to announce event and schedule meetings across grantees and evaluators. Dr. Fetterman's slides are available here. In addition, Dr. Fetterman created a pre-test survey and a post-test survey for participants to help evaluate the session. The results were extremely positive, documenting gains in familiarity and knowledge about online surveys, blogging, online documents and spreadsheets, file sharing (pictures), telephony, and videoconferencing on the net.

In addition, a few emails provide some insight into the participants’ assessment of the workshop: “Thank you, Dr. David, for the session last week. I will put into practice all I have learned.” Another said: “Hi. Thank you for your patience and thoughtfulness with us!!! Thanks also for the blogs!!!” A copy of the Evaluation Technology Workshop Report is available from this blog.
The participants co-created another survey. It was the Arkansas Evaluation Center Needs Assessment survey. The results of the brief needs assessment is being used to inform the Center's curricular development .

MISRGO EVALUATION PLAN - 2007-2008

MISRGO Evaluation Plan. This document is divided into two sections. The first half highlights accomplishments to-date, including creating the Evaluation Monitoring System.

The second half of the report presents the evaluation plan for this coming year. It charts the way for us. We plan to use the report to help us navigate through often mirky and tumultuous waves. It also provides an insight into what's in store for us in the future.

For a copy of the report click here.

MISRGO Evaluation - Grantee Satisfaction with Office Services

MISRGO provides administrative and support services to tobacco prevention programs in Arkansas. Fetterman & Associates are currently facilitating an empowerment evaluation of the program, helping grantees use evaluation to accomplish their objectives (with a continuous cycle of assessment and action).

One part of the evaluation includes assessing MISRGO itself, as a supporting office or set of services. Refinements and improvements in the Office directly benefit and enhance grantee work. MISRGO staff embraced this part of the assessment and helped generate questions for the online survey because they sincerely wanted to know what grantees thought about their services. In addition, they wanted to improve the quality of their services to grantees. They understood the chain of reasoning involved, enhancing services increases the probability of improving grantee success in meeting their tobacco prevention objectives.

The report is available to the public. It was generally favorable, they valued the services they have been receiving. However, they were also critical and suggested areas meriting attention. They also recommended a list of ideas for future workshops and services.

MISRGO is already processing this information and planning to adopt many of the recommendations associated with the report, including soliciting more involvement from grantees in future conferences. This is the first step in evaluating the office, additional activities will include a site visit, interviews with staff members and grantees, and a review of relevant documentation.

4th Annual Conference Clearing the Air in Communities of Color May 23-24, 2007

4th Annual Conference" Clearing the Air in Communities of Color

The conference received letters of support from Governor Mike Beebe, Mayor Carl Redus, Chancellor Lawrence Davis, Jr., Dean Calvin Johnson, and Program Coordinator Marian Evans.

Keynote Speakers included: Senator Henry "Hank" Wilkins, Broadway Joe Booker, and Kenya Masala.

Sessions included: Tobacco Use in the Asian Pacific Community, The Oxygen Project, Tobacco Use Among African American 13 Year Olds in Jefferson County,

Cease Project, Best Practices, Advocacy and You, Fire and Burns Related to Cigarettes, Improve Service Delivery by Embracing Youth Culture, Empowerment Evaluation Principles, Tobacco Control in the Latino Community of Arkansas, and Just Say Y.E.S. on the Move for Better Choices.


Dr. Fetterman presented the Closing Luncheon Presentation.

He provided conference participants (particularly grantees and evaluators), with an update on the status of the evaluation, including the Evaluation Monitoring System, the State-wide Community Survey Results, and the new Arkansas Evaluation Center.

In addition, a special video was presented highlighting the accomplishments of each grantee. (It was both inspiring and humorous as well.)

Number Quit Smoking, MISRGO Office Evaluation & the Arkansas Evaluation Center - April 26, 2007

Evaluation Workshop in Little Rock - April 26, 2007

The April MISRGO Evaluation Workshop in Little Rock was very productive. Dr. Fetterman and Ms. Linda Delaney facilitated the workshop. The participants brought the session alive. They were completely engaged. (Click on picture above to display a brief video of our workshop.)
















There were three key topics covered: the evaluation monitoring system (focusing on the number who quit smoking or quit using tobacco); the evaluation of the MISRGO office; and the Arkansas Evaluation Center. (Click on Fetterman & Associates below to access the slide presentation during the workshop.)

Evaluation Monitoring System

The evaluation team was reminded of the importance of continuing to collect tobacco consumption (or in this case cessation) data. They agreed an ideal data collection effort would collect monthly figures. However, a realistic goals was to assist the grantees as they collect quarterly and bi-annual cumulative data. Evaluators plan on reminding grantees how important it is to continue to collect the data, even if they only record one person who stopped smoking, since that represents $150,000 in excess medical costs saved.

Dr. Christine Sheffer
University of Arkansas for Medical Sciences
Fay W. Boozman College of Public Health
Health Behavior and Health Education Department
SOSQuitline 1.866.NOW.QUIT
SOSWorks Fax 1.888.827.7057

Marian Evans, MISRGO Coordinator, stated she would follow up with Dr. Christine Sheffer, University of Arkansas for Medical Sciences, concerning the SOS data referral plan, to help us track grantee referrals to smoke cessation programs.


MISRGO Office Evaluation

This was one of the most rewarding parts of the workshop. The focus was on plans to evaluate the effectiveness of the MISRGO office, in terms of supporting and facilitating grantee and evaluator activity.

MISRGO office staff members and MISRGO/grantee evaluators worked together to generate a list of useful questions to assess ourselves. Instead of the typical defensiveness associated with endeavor, the MISRGO office staff jumped in, posing difficult questions that they wanted the grantees and evaluators to ask about their performance. They did this because they want to know what they can do better, to better serve the entire community. It was a rewarding and fruitful exercise. An online survey will be developed in the next few weeks and grantees and evaluators will be asked to provide their views concerning the MISRGO office's efforts. This is in the true spirit of empowerment evaluation and self-assessment (to improve and enhance).

Sample Questions

Arkansas Evaluation Center

Dr. Fetterman and Ms. Delaney announced and described the new Arkansas Evaluation Center (and highlighted the new blog page about the Center). The bill (now an act) passed the House and Senate and was approved by the Governor.



Dr. Fetterman and Ms. Delaney took this time to solicit input from the workshop participants concerning the overall design of the Center, possible courses, and the strengths and weaknesses of the online distance learning component (using WebCT).


The participants were generative and productive. They recommended specific courses for a certificate program, such as research design, survey design, focus groups, interviews, and statistics (light). They also recommended a course about ethics, guidelines, and standards.

Based on a straw vote, everyone said they would be interested in participating in a evaluation certificate program. The next steps will include: the development of a needs assessment survey - to determine the level of interest in the program (above and beyond our group), a review of comparable programs (to identify the best features across programs that we might incorporate in our program), and course development (including WebCT distance learning programs).

Arkansas Evaluation Center

The Minority Initiative Subrecipient Grant Office (MISRGO) and the Evaluation Team proposed the creation of an Arkansas Evaluation Center to the Arkansas Legislature on March 19, 2007.

The idea emerged from our tobacco prevention work. The need for more evaluators and more systematic evaluation training throughout the State became increasingly evident. Not only were more evaluators needed for tobacco prevention work but for many other programs as well. Thus our proposal to establish a center to train new evaluators and provide professional development training for working practitioners.

(Click on Picture to Display Full Size Poster of Arkansas Evaluation Center)

The bill describes the mission or purpose of the center. Highlights include:

• enhancing the quality of existing services and programs
• ensuring accountability for funds and services used
• “raising the bar” concerning standards and expectations
• increasing the use of data to inform program operation and decision making.

(Click on picture of Bill to display full PDF of the Bill.)

An empowerment evaluation approach and philosophy has been adopted by the Center to shape practice in communities throughout the State. Empowerment evaluation and other collaborative forms of evaluation are particularly useful in complex collaborative endeavors requiring the participation of diverse groups. It is designed to help groups accomplish their objectives using evaluation concepts and tools.

Special thanks are due to all of the MISRGO staff, grantees, and evaluators for helping to identify this need and transform this need into an action step designed to meet the need.

November 4, 2006 - American Evaluation Association (dissemination)

Dr. Fetterman and Ms. Linda Delaney helped to disseminate the Tobacco Prevention evaluation findings at the American Evaluation Association in November, highlighting the amount of money saved the State as a result of these tobacco prevention programs. They also discussed the politics of tobacco prevention evaluations.

(click on photo above to display video)

September 22, 2006 - Refresher, Focus Group, and Black Caucus

The September 22 workshop was designed to be a refresher and provide additional methodological training. Dr. Fetterman provided an overview of the project and its current status from an evaluation perspective, including the evaluation monitoring system and the community survey. Dr. Fetterman also provided focus group training for the Arkansas MISRGO evaluators. Ms. Delaney faciliated the self-assessment process of the training generating a 4th data point of self-assessment for the empowerment evaluation exercise.



Dr. Fetterman was also invited to meet with the Majority Leader, Senator Tracy Steele, and the Black Caucus during this trip.


He provided them with pertinent data, ranging from the amount of money saved as a result of these programs to the wide-spread community support enjoyed by MISRGO across the State (large constituent base). Dr. Fetterman also met with various news organizations to share the evaluation monitoring system findings and the community survey results.

Evaluation Monitoring Tool Update

We now have an interactive spreadsheet that we share on the web. It is our repository of tobacco consumption data. It provides us with an up-to-date view of: number of materials distributed, number pledge to quit, number who quit, cost saved per person, and cost saved by grantee group. We also have a running total of each of these categories including total cost save the State concerning excess medical costs. As of July 4, 2006, the totals across the State are as follows:


We also maintain a more detailed data sheet, enabling us to view the numbers by grantee (see below).

Community Surveys

We conducted a community survey throughout the State of Arkansas focusing on the views of the grantees' community members. We received 946 responses. The findings were overwhelmingly positive.

A few survey responses are provided below to provide some insight into the level of community support for tobacco prevention programs in the State.


In addition to providing information in the aggregate about community views across the State, we produced a report that provides community member viewpoints about their specific grantee or tobacco prevention program.

Clearing the Air in Communities of Color Conference

We shared the evaluation approaches we have been using to measure change with colleagues at the 3rd Annual Clearing the Air in Communities of Color conference. (See video below.)

(click on picture to display video)

We discussed our training efforts, the empowerment evaluation process, our evaluation monitoring system, and our community surveys. It is a useful summary of our efforts up to May 26, 2006. The slides of our presentation are available to the public.

January 2006

We held a local evaluator workshop in January. It focused on how to use the Evaluation Monitoring System. They will work with the local grantees to help them collect critical baseline data, concerning the number of people smoking in their communities. In addition, it will help programs document the number of people they reach and the number who quit. The system will also translate that into costs saved. [The workshop slides are available on this site.]

A brief video of the training is provided below.

Evaluation Monitoring System and Community Survey
Local Evaluator Workshop Training
(click on picture to display video)

In addition, we are working on community surveys of the grantees so that this information can be fed back into the programs to enable them to enhance their service to their communities.

We will continue to use the Evaluation Monitoring System tool, along with many others, and rate our own performance throughout the evaluation. These tools help to guide us through the journey.

September 2005 - Common Indicators and Qualitative Methods

The September workshop was extremely productive. The grantees agreed upon common indicators to measure program performance, including the size of the target population, the number of smokers, the number reached with educational messages and services, and the number pledged to quit (as well as documented individuals in smoke cessation programs). A sample is provided below (see Figure 1):

Figure 1. Common Indicators

Dr. George Blevins, MISRGO Director,

introducing the September 2005 workshop.

Grantee oriented interviews and surveys were conducted previous to the workshop to solicit grassroots data concerning how grantees collect data, the quality of the data they have, and program implementation. In addition, recommendations were made during the interviews and workshop, including how to collect sign-in and sign-up sheets as well as pledge to quit sheets. Census and At Risk web site were shared to help grantees collect valid and useful data concerning the smoking rates in their communities.

Risk Factor and Census reports shared with grantees

The workshop focused on how to use various qualitative techniques such as ethnographically-oriented observation and interviewing. The workshop also demonstrated how to construct online surveys (with real examples on the web).

Grantees completing an ethnographically oriented observation exercise
and Dr. Fetterman critiquing their work.


The slides from the workshop training are available in pdf format. The Final Report for Year One summarizing work-to-date is online and available on this site. In addition, a brief video is available concerning workshop highlights.

Brief video of September 2005 workshop
focusing on common indicators and
qualitative techniques for data collection.

(Select arrow on bottom left of the screen to display video)

During the meeting we agreed on the need to develop an Evaluation Monitoring System. (See Figure 17 below.) It is described in the September 2005 Annual Report.

We also thought it might be useful to write a Local Evaluator Training Manual. This will help us create greater consistency in practice, ranging from collecting the same type of data to analyzing it in the same format.

We continued to operate as a group or consortium. We have adopted an empowerment evaluation approach to guide our work. This approach enables us to monitor our progress at each stage in our development as a group. We "check in" each time we meet and assess our progress. In this case, we have documented the third data point or assessment of our progress to date (see graph below).