ACET, Inc. Blog

Check out this video for what not to do in faciliating a dialogue! At just over a minute, this is a short and comical clip that teaches us a few lessons on what to avoid when conducting interviews and/or focus groups.

Notice the kitty’s pointing finger, negative facial expressions, threats of tattling, unnecessary directness and belittling. Who knew such an adorable critter could be so off-putting? That poor puppy didn’t know what to do with himself!

What are some lessons you’ve learned about facilitation?

We often get requests to provide feedback on surveys. As we review the surveys, we tend to see three challenges occur quite frequently. I’ve written this blog to offer suggestions for addressing each of the challenges.

1. Double-Barreled Questions. In a double-barreled question, two concepts or ideas are included in one survey question. For example, the three survey items below are all double-barreled questions:
○        “How much of your monthly income is spent on restaurant dining and entertainment?”
○        “How often do you volunteer your time or make charitable donations?”
○        “How often do you consume beer and smoke cigars?”

But how does a respondent answer a double-barreled question? For example, if a family spent no money on restaurant dining but 10% of their income on entertainment, how should they respond? 0%? 10%? Average the two and respond 5%? Usually survey respondents do provide an answer, but they often utilize an individualized strategy for finding a response, which can be challenging for program staff to interpret.

Double-barreled questions can be addressed by limiting each survey item to one – and only one – concept. Below is a double-barreled question followed by revisions that eliminate the double-barrel:

2. Imbalanced Response Options. Imbalanced response options refer to a set of response choices that do not cover the range of possible choices. An imbalanced response scale is problematic because it limits respondents’ responses to only one end of a scale and may produce inaccurate results. For example, if a respondent wants to answer “strongly disagree” to a question but that option is not available, the survey is not capturing the respondent’s perspective. Using a set of balanced response options ensures that respondents’ answers can be accurately captured. A very good resource for a variety of balanced response scales can be found here: http://www.gifted.uconn.edu/siegle/research/Instrument%20Reliability%20and%20Validity/Likert.html

Below are some examples of imbalanced response options with improvements to the scale:

3. Overlapping Response Options. Overlapping response options simply means that there is some overlap in the response choices available to the respondent. For example, the following survey item has overlapping response options:

How many books did you read in the last month?
0 or 1 book
1 or 2 books
2 or 3 books
3 or more books

Overlapping response options are problematic because they can confuse the survey respondent. If you read two books in the last month, which option would you select: “1 or 2 books” or “2 or 3 books”? In addition, questions with overlapping response options are a challenge to interpret. How do you know how many books were read if the response options overlap? Overlapping response options can be addressed very easily by carefully constructing the response option list. Simply revise and edit the list so that there is no overlap between the response choices.

If you have any questions about the above challenges to designing a survey, please feel free to leave a comment or email me at Kirsten@acetinc.com. Please also contact any of us at ACET if you have questions about survey design in general.

Kirsten

At ACET, we strongly believe in “capacity building” – but what does that mean? Basically, we believe in creating and implementing evaluation plans that involve our clients. Our clients have input in every step of the evaluation process, which enables us to find the plan that best suits the needs of their program or organization. In turn, they learn how to use the evaluation results to improve programming. Since capacity building is so important to us, we work hard to ensure that we maintain open and transparent communication with our clients.

Stella learned of a video while at the American Evaluation Association conference that summed up precisely what NOT to do when communicating with your clients.

http://www.youtube.com/watch?v=AukrpAoAYY0

The video is amusing, but it brings up some essential key points.

“What does this mean?” Use clear language that clients can understand. Those test results may be second nature to you, but like the man in the video, your client may have no idea what those numbers actually mean.

“What does a score of 35 tell me about my patient?” In our experience, clients don’t want random numbers to throw around; they want solid information that they can use to improve their programming. Be sure to clearly delineate what their results mean and answer any questions they may have.

“I used the scale you developed to measure my patient.” Your client trusts that you have expertise about evaluation – after all, they’re using products you likely designed! – but being able to communicate that knowledge is essential in maintaining the relationship.

The end of the video is also telling. Despite the researcher’s supposed credentials, the client doesn’t want to deal with her anymore. Clients, too, may want to stop working with someone they feel isn’t right for their organization, even if that person or company has years of accolades.

Do you have any thoughts to share about communicating with clients? We’d love to hear them!