ACET, Inc. Blog

Recently, John Gargani shared a list of 5 words that evaluators use that can be misleading in reports. John’s blog inspired us to identify “pet peeves” we have found in evaluation work. Here’s our own list:

“Problem”

The use of the word “problem” in itself can elicit negative responses and put people on the defensive. When people are defensive, it can be difficult to engage in open dialogue and creative problem solving. Instead of using the word “problem,” Stella recommends using creative solutions to frame the issue to facilitate discussion. For example, Stella suggests using the word “challenge,” as in, “Participants identified some challenges in receiving quality health care…”

“Larger Population”

Joseph highlighted the overuse of questions about a “larger population,” as in, “how do our results compare to the larger population?” In general, evaluation results are obtained from a specific population within a specific context. Even results from “larger populations” are obtained from specific, albeit larger, populations within a specific context (e.g., Minnesotan 3rd through 8th grade students), and not from some generic “larger population.” In order to make valid comparisons, it’s important that evaluators always classify both their population of interest (e.g., South Minneapolis teens, between the ages of 13 and 17, from households with incomes of $25K or less) and the comparison group (e.g., a sample of Minneapolis teens from households with a variety of income levels). As often as possible, evaluators should not use generic “larger populations,” but should be as specific as possible about their comparison groups to enhance the validity and accuracy of their comparisons.

Describing the “Other” Category

Dan pointed out that, when analyzing qualitative data, there might be 5 common themes, but also an “other” theme for comments from a small number of people. When summarizing the results of the qualitative analysis it is important to talk about all of the themes, including describing what “other” indicates. Without describing what “other” indicates, the reader may have no idea as to what the writer is referring and may have to guess what “other” indicates. It is important that evaluators be as clear as possible when writing to ensure that the reader has a clear understanding of the data.

Pronouns

Kirsten’s pet peeve is overuse or abuse of pronouns. Pronoun overuse can detract from the clarity of writing and the overall readability of the document. From a grammatical standpoint, a pronoun always refers to the last used noun; if the last used noun isn’t what the writer intends to refer to, then a pronoun is inappropriate and the writing will not be as clear as the writer intended.

As a rule of thumb, Kirsten alternates noun-pronoun-noun-pronoun. She acknowledges that her method doesn’t always produce highly engaging prose, but it is clear!

Do you have any writing “pet peeves”? If so, please share in the comments below!

ACET Staff

Over the last few weeks, many of our clients have inquired about the availability of their test data from the Minnesota Department of Education (MDE) to meet reporting requirements. Here’s the most recent update from MDE’s Research and Assessment division:

“MDE is still evaluating the specific timelines for the release of the Reading and Mathematics MCA, MCA-Modified, and MTAS results, and we will provide one week’s notice before the results will be released. While we understand that not having a specific date makes it difficult for your planning purposes, we are holding some flexibility in the event we are able to complete our work and release results earlier than late September. Once a final determination is made, we will provide one week’s notice before results are released. We thank you for your patience as we continue work through the challenges created by the government shutdown.”(http://www.education.state.mn.us/MDE/Accountability_Programs/Assessment_and_Testing/index.html)]

We will keep you updated on projected timeline and new information when they become available.

Joseph

Recently, I detailed the issues with describing evaluation and the work we do here at ACET (http://www.acetinc.com/blog/2010/11/22/part-1-%E2%80%9Cso%E2%80%A6what-is-evaluation%E2%80%9D-dealing-with-blank-stares). As I mentioned, often we are asked to include a basic definition for evaluation and to elaborate on what an evaluation looks like.

If you search online for a basic definition of program evaluation, you would likely find something akin to the Wikipedia classification, which states, “a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency.” http://en.wikipedia.org/wiki/Program_evaluation.

You may also be interested in reviewing the recent expanded definition of program evaluation developed by The joint Committee on Standards for Educational Evaluation recently (The Program Evaluation Standards, 3^rd Ed., 2010). To learn more about the evolution of evaluation definitions, I suggest reading Utilization-Focused Evaluation by Michael Patton, one of the pioneers of evaluation (check out pages 38-39 for his review).

We at ACET have used the following definition for program evaluation: “the process of systematically collecting and analyzing data and using the results to answer questions about a program in order to improve the program’s effectiveness and/or efficiency.” We fully believe that an evaluation should be systematic (objective and specific), results-oriented (just as in education, constructive criticism can be used for good), and should be used to improve the program (we want the program to have optimal results and funding in the future).

If you have additional questions or comments on the variations of program evaluation definitions, I would love to hear from you!

Stella

There’s an uncomfortable moment I often experience: I meet someone new and they, inevitably, inquire about what I do. When I tell them, “I’m a program evaluator,” I get a blank look followed by, “What is a program evaluator?” I go on to explain what we do as evaluators, list common tasks (e.g., design surveys, conduct focus groups, measure program impact), followed by a list of potential client groups (e.g., schools, higher education institutions, nonprofits). All seems fine after this initial conversation. The interesting part is when I talk with that person again, they ask: “What do you do again?” Funnily, this experience is not limited to strangers. Recently, I overheard an immediate family member explaining my occupation to another relative as: “I think she does Internet stuff.”

Over the last decade I have examined ways to best describe program evaluation and keep the description brief and salient. I think this is attributable to the fact that our field of program evaluation is uncommon in the marketplace – there are no Halloween costumes for us or trinkets to display our occupations.

Do you have a brief description that works for you to describe the field of evaluation?

Stella

P.S. Stay tuned for our next blog, which will detail the basic definitions of evaluation.

As you may know, Stella and Kirsten will be making presentations at the American Evaluation Association (AEA) 2010 conference this Saturday. ACET just posted two PowerPoint presentations on AEA’s website in preparation for the conference.

Multipaper Session #798: Evaluating a Child-Welfare Demonstration Program: Evolution, Considerations, and Lessons Learned
Multipaper Session #763: The Impact of Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) on Student Academic Preparedness

The presentations posted in the AEA eLibrary system are open to the public, so please feel free to check it out! To see all presentations posted for the AEA conference, please click here: http://comm.eval.org/EVAL/EVAL/Resources/LibraryDocuments/Default.aspx?LibraryKey=1eff4fd7-afa0-42e1-b275-f65881b7489b

If you have any questions or would like more information, please don’t hesitate to leave a comment or contact us!

This July I participated in a “multi-tier” review of grant applications for the U.S. Department of Education. It was a great experience! I learned a lot about how grants are reviewed at the Federal level and would like to share those processes and some advice with you.

Review Processes

Under a multi-tier review, the required proposal elements are divided into sections and reviewers assess and score specific sections of the proposal. Reviewers are assigned to different tiers based on their level of expertise. At the end of each tier of reviewing, the highest scoring applications move on to the next tier for additional review. Checks-and-balances are built into the entire process to ensure that the entire review process is fair and equitable.

For example, I was a Tier II reviewer and the applications had already been scored on four proposal elements. The Tier II reviewers read and scored the applications for two additional sections: the research foundation of the application and proposed evaluation activities (we did have to read the entire proposal, but scored only those two sections). After reading the assigned applications, I participated in a conference call with another reviewer assigned to the same applications and a panel monitor (an employee of the U.S. Department of Education). During the conference calls, the reviewers discussed the merits and challenges of each application and how well each met the scoring criteria. The reviewers also worked towards aligning the assessments so that we were consistent in our application of the review criteria for all of the applications.

Grant Application Advice

Three key themes emerged from my review:

1 – Make sure the application covers all of the required elements outlined in the RFP.
One of the proposals I read did not include background research or theory in the narrative. As a result, the reviewers scored that section quite low. Although this advice may feel self-evident, it’s a pretty good bet that elements listed as ‘required’ in the RFP will be included as scorable criteria in the review phase. It is very important and very beneficial to have someone read the application and verify that it covers all required elements before submitting it.

2 – Ensure space and point allocations are proportional to application pages.
One of the applications I read included more than five pages describing background research (worth up to 10 points) and but fewer than five pages describing the proposed evaluation (worth up to 15 points). While the research section was very detailed (and was scored highly), the description of the proposed evaluation was very short and the lack of detail resulted in a lower score for the proposed evaluation section. When writing a grant application, it’s important that sections that are worth more points be allocated more space in the proposal than sections worth fewer points. This is especially true when there is a page limit to the proposal!

3 – Review, re-read, and edit the application to ensure an appropriate breadth and depth of the narrative.
Several of the applications I scored included descriptions of proposed evaluation activities that were so broad and sweeping it was difficult to determine what the evaluation would be. With highly competitive grant competitions such as the one I reviewed for, it’s very important to give the reviewers enough breadth and depth in the narrative so they get a sense of proposed program activities, why the program is needed, when the program has been successful in the past, and how the proposed program will be evaluated.

I hope this information is useful to you and your agency when you prepare an application for a grant. Please comment below or email me if you have any questions or would like additional information!

Kirsten

In March, Heather and Kirsten wrote about their experience in the “Grant Writing Boot Camp” and touched briefly on SMART goals. This post will provide a little more insight into developing goals that meet the criterion of being “SMART.”

SMART stands for:  Specific, Measurable, Attainable, Relevant, Time-bound. Let’s review each term one at a time.

Let’s make up an imaginary program and work through creating a SMART goal our fake program: Forever Homes, our hypothetical agency, helps dogs in need of homes find permanent (“forever”) homes.

Specific. To say a goal is specific indicates that the goal itself is not too broadly identified. For Forever Homes, a goal could be stated as “rescue dogs.” While this is a goal of the program, it is not specific. What does it mean to “rescue” a dog? Do we mean rescue all dogs from human ownership and free them to the wild (hopefully not!)?

A more specific goal might be something like “Find forever homes for homeless dogs in Minneapolis.” This goal has been improved in a number of ways: We have identified a specific target (homeless dogs); limited our region for homeless dogs (Minneapolis); and defined “rescue” (find forever homes).

Measurable. A measurable goal is simply one that can be measured. By making a goal specific, we are well on our way to having a measurable goal. It is much easier to measure the specific goal, “find forever homes for homeless dogs in Minneapolis” rather than “rescue dogs.” We can measure specifically how many dogs that have been rescued Minneapolis have been placed in forever homes.

Attainable. An attainable goal is one that is able to be reached. Our current goal, “find forever homes for homeless dogs in Minneapolis,” is specific and measurable, but do we really expect to find forever homes for all homeless dogs in Minneapolis? So, to make our goal more attainable, let’s update it to “find forever homes for 90% of the homeless dogs in Minneapolis identified by Forever Homes.” That’s a goal that might be attainable, as opposed to 100% of dogs.

Realistic. This is very similar to an attainable goal; however, just because our goal is attainable does not mean it is realistic. Finding a forever home for 90% of dogs, although possible, might be far beyond what the organization could expect to realistically achieve. Perhaps a more realistic goal is “find forever homes for 75% of the homeless dogs in Minneapolis identified by Forever Homes.”

Time-bound. The most recent version of our goal may have already sprung the question “wait, what if a homeless dog is found a day before the data is reported for evaluation? Should we really count that dog as ‘not having a forever home’ because the dog has only had one day to find a home?” This is one example of why one must make SMART goals time-bound. Let’s revise our goal again: “find forever homes within 6 months for 75% of the homeless dogs in Minneapolis identified by Forever Homes.” One needs to ask here if the time-bound condition is realistic.

We began with “rescue dogs” and ended with “find forever homes within 6 months for 75% of the homeless dogs in Minneapolis identified by Forever Homes.” There is a drastic difference between the two and it should be obvious through the discussion that the second goal achieves SMART status.

Have any insight into SMART goal development you’d like to share? Any follow-up questions? Please comment below.

Brainstorming creative and useful ways to display data and results can be a challenge. A recent discussion on the LinkedIn page of the American Evaluation Association has turned my attention to a useful tool called the “Periodic Table of Visualization Methods, which can be used to aid and/or inspire this creative process. The table, as suggested by the name, is designed to mirror the periodic table of the elements, but instead of describing the many basic elements and their components, this table is based on the many ways to visually display common evaluation information.

Though the table itself is quite detailed and may look daunting at first glance, you can easily begin to explore its usefulness through its interactive features. Please click on the link and I can further explain this tool: http://www.visual-literacy.org/periodic_table/periodic_table.html.

Once you have activated the link, take a moment and hold your cursor over one of the “elements.” A little graphic should pop-up that will give you an example of that particular visualization method. Each element has its own unique example. Though these are just examples, one can begin to imagine their own information in such displays.

Along with the great examples of each visualization type, the table provides various color coding and symbols to highlight what each type of visualization communicates. For example, yellow indicates a visual display of data while blue-green indicates a visualization of a concept. The two-letter acronyms are described within each box; the symbols used within the boxes are described in the key. Again, hold your cursor over the box for information.

For a more detailed explanation of the development and rationale behind the table, follow this link: http://ow.ly/wk7d

One of the challenges presented by this table is the actual creation of these displays. Though the displays are very visually appealing and often complex, the appeal and complexity demands software that can do such. However, this table does provide some useful guidance and inspiration for how to grab readers’ attention and communicate complex information effectively. What visualization methods stand out the most for you? Which do you think are most appealing? Any personal favorites?

Joseph

We are always in pursuit of new and interesting ways to visually represent all types of data. Qualitative data often presents challenges in doing this – it’s tough making a pie chart out of focus group data! So what are some options? Try Wordle (http://www.wordle.net), an online tool that generates visual representations, or “word clouds” from text. The more frequently a word appears in a selection of text, the larger it appears in the word cloud. The fun part is that Wordle allows you to change the fonts, shapes, and color schemes of the images it creates. We are using this more frequently in our focus group and interview summaries because the images offer an additional dimension of understanding to qualitative data that is difficult to perceive from reading standard text. Each of the ACET staff took a few minutes to create a Wordle example from famous pieces of literature. Which are your favorites?

My Wordle is an excerpt from The Three Musketeers by Alexandre Dumas:

Joseph created a Wordle using an excerpt from Notes from the Underground by Fyodor Dostoyevsky:

Cassie’s Wordle is an excerpt from Shakespeare’s A Midsummer Nights Dream:

Emily created a Wordle with an excerpt from Pride and Prejudice by Jane Austin:

Stella’s Wordle came from Edgar Allen Poe’s The Raven:

And last, but not least, Kirsten crafted a Wordle with an excerpt from A Christmas Carol by Charles Dickens:
Heather

A couple of weeks ago, I posted a blog about essential skills found in highly skilled evaluators. Although having the skills I described is undeniably important to evaluation, one must also have certain personal qualities to be a dynamic evaluator.

1. Evaluators must be good conversationalists. They are active listeners, engage in dialogue, give others time to ask questions before moving on to another topic, and don’t interrupt people. Being a good conversationalist allows evaluators to build rapport with their clients. They understand and appreciate the different perspectives, backgrounds, experiences, languages, and cultural identities that make each person unique. Evaluators also know how to ask relevant questions. When they go into a meeting, evaluators actively participate and understand how to ask the right questions so that they quickly get the information necessary to provide top-notch research and evaluation.
2. Each organization has its own culture, way of thinking, and way of doing business and highly skilled evaluators pick up that information quickly. They also adapt quickly to agency protocols. Doing so helps the evaluator suggest techniques, strategies, and methods most appropriate for an organization, which can aid in the success of any evaluation program.
3. A skilled, dynamic evaluator sees the big picture and the minute details. In doing so, an evaluator can provide the best evaluation and research tools to enhance the effectiveness of a client’s program. “Big picture thinking” is an essential skill because the evaluator can then assure that tasks are completed on time and that deliverables will be ready when the client needs them. Paying attention to minute details is also an essential skill because the evaluator can assure clients that deliverables have been completed accurately and that small, but important, elements of the deliverable have not been overlooked.
4. Last, and this may seem self-evident, a skilled evaluator is a team player. Skilled, dynamic evaluators are very good at working with a diverse group of stakeholders to craft a successful evaluation plan. This is important at all stages of an evaluation: Skilled evaluators participate in a team to design, carry out, and report on the evaluation.

Hopefully these highlights have given you an idea of what makes a successful evaluator. If you have any questions or thoughts, please feel free to share them!

Stella