Thursday, May 1, 2008

Spring Evaluation Training Workshop - April 24, 2008

Spring Evaluation Training Workshop

Grantees and evaluators came together on April 24, 2008 to participate in evaluation training. Bringing both of them together ensures that they are on the same wave-length about data needs, roles, and relationships. (A copy of the slides used to guide the training day are available on this blog.)

The training was geared toward a new monitoring tool – the Visual Alert System. (This tool is described in the Visual Alert System posting below this one.)

This tool helps monitor grantee progress across programs, providing a readily available update upon demand. For example, bar charts are used to visually document how many grantees are using the Quitline.In addition, grantees and evaluators agreed that the same bar chart format could be used to document individual grantee progress. For example, a grantee would have:

  1. The first bar chart would document their baseline of 4 news articles printed so far this year.
  2. The second bar chart would document their aspirations – a benchmark number such as 10 new news articles by the end of the year.
  3. A third bar chart would document the actual number of articles they had published that year, which might be 8.
  4. A fourth chart would compare the first baseline bar chart of 4 with a final end of the year bar chart of 8 – documenting their progress throughout the year.

These bar charts could be forwarded to MISRGO and Dr. Fetterman and Ms. Delaney to document progress across grantees based on activities held in common, such as the Quitline or newspaper articles.

Dr. Johnson, Dean of the School of Education, was present to introduce new points and reinforce existing recommendations, including the need include activities directly related to tobacco prevention efforts even if the activities were not originally listed in the plan. In addition, he reminded grantees of the need to seek external funding.

The Visual Alert System provided an insight into patterns across grantees, alerting grantees and evaluators to successes that should be reinforced and continued, as well as areas meriting attention. The issue of whether we were implementing the “right” or most effective activities was discussed.

The Center for Disease Control document concerning Best Practices helped guide this part of the discussion. A summary of best practices was distributed during the workshop. CDC’s complete document is available on line.
One of the Visual Alert System findings initially suggested that few grantees were applying CDC Best Practices. However, after some discussion it was clear that grantees were in fact applying CDC Best Practices without knowing it, e.g. Quitline, coalition building, and so on. This finding and discussion highlighted the importance of sharing CDC documentation and engaging in a dialogue about what was being implemented in practice.

The Evaluation Monitoring System was also emphasized during the workshop, highlighting the importance of the group continuing to work together as a group. This system has demonstrated how much money they have saved as a unified tobacco prevention coalition - $84 million.

The workshop concluded with a few announcements including the:

  1. establishment of the Arkansas Evaluation Center
  2. development of an evaluation certificate program (and master’s program) at UAPB
  3. upcoming Arkansas Summer Evaluation Institute Workshop – July 10th and 11th

The group responded enthusiastically to the announcements.

Dr. Fetterman pointed to the LCD projection to let them know the details associated with the Summer Evaluation Workshop.

The Summer Evaluation Workshop will include:

  • Senator Tracey Steele
  • CDC’s Health Education Specialist Kevin Collins
  • Dr. Stuart Donaldson, Dean of the School of Behavioral and Organizational Sciences, Claremont Graduate University - Introduction to Practical Program Evaluation (Program Theory)
  • Dr. Tarek Azzam, Senior Research Associate, School of Behavioral and Organizational Sciences, Claremont Graduate University - Introduction to Technology and Evaluation: Focus on the use of Geographic Information Systems and Online Surveys
  • Dr. Rosalie Torres, Torres Consulting Group - Interviewing Skills for Evaluators
  • Dr. Fetterman, Director of Evaluation, School of Medicine, Stanford University and Co-Director, Arkansas Evaluation Center, School of Education, UAPB
  • Mr. Charles Gasper, Director of Evaluation, Missouri Foundation for Health - A Foundation Perspective about the Significance of Program Evaluation
  • Dr. Nacaman Williams, Walton Foundation – A Foundation Perspective about the Significance of Program Evaluation
The Summer Evaluation Workshop Program and Schedule is listed in under Useful Links on the left column of the blog. For your convenience, the program is also available by selecting this Summer Workshop Program 2008 link.

The Spring Evaluation Workshop concluded with an announcement about the new Arkansas Group of Evaluators. AGES is an official affiliate of the American Evaluation Association.

Their first official meeting as an affiliate will be on July 10th in the evening at UAPB. It will be held during the Summer Evaluation Workshop. Ms. Linda Delaney is the chair of the State-wide evaluation group.