ACET, Inc. Blog

We are pleased to announce that ACET has an opening for a new Evaluation Specialist! The newly hired Evaluation Specialist will be a highly motivated and enthusiastic individual with demonstrated experience performing evaluation or research studies and writing clear, concise reports for a variety of audiences. Experience with relational databases in Excel and SPSS (or other statistical software) is required.

A qualified candidate will have a master’s degree in the field of evaluation, health science research, policy analysis, or related field. A degree of creativity, independence, and latitude is expected with this position. Job duties for this position include:

1. Working in a team setting to plan an evaluation (e.g., logic model development, evaluation planning, database mapping);

2. Conducting literature and document reviews;

3. Designing qualitative and quantitative measures (e.g., focus group questions, survey items, interview questions, rubrics, site visit protocols);

4. Collecting data and co-facilitating meetings;

5. Compiling results, preparing reports, and documenting progress toward outcomes;

6. Working with clients to build evaluation capacity;

7. Sharing ideas on program evaluation methodologies and frameworks; and

8. Assisting with various aspects of business operations and other relevant duties as assigned.

Applicants should email or send a one-page letter of interest and a resume (not to exceed two pages) to:

ACET, Inc.

9868 Lyndale Avenue South

Minneapolis, MN 55420

Email: info@acetinc.com

To learn more about ACET and the kind of work we do, please visit: http://minneapolis.craigslist.org/hnp/hum/2918713510.html.

I’ve been thinking a little bit about the house building process recently – not thinking of undertaking it, but thinking about how embarking on an evaluation can be surprisingly similar to deciding to build a house. There’s no doubt that designing and building a new home is a gargantuan task and the experience can feel completely overwhelming. But, for those who put the effort in and survive to tell the tale, the results can be superbly rewarding.

So what makes me think that house building is so similar to building an evaluation? For one thing, the first step is basically the same: You call in the experts. For home building, you find a builder and for an evaluation, you find an evaluator. But what really got me thinking about the similarities is this: in either case, once you’ve called in the expert, you have many options of how you want to proceed. In my mind, people fall into three general approaches on either of these projects:

1) “I want to be involved, but I need help figuring things out.”

For this person, the expert becomes a facilitator, who helps them identify their needs and options. For instance, a builder might ask a client questions about how often and what kind  of cooking she/he does, in order to figure out how to layout the kitchen. Similarly, an evaluator might ask a client to list their current needs (e.g., areas of program strengths and challenges) and options (e.g., capacity of the program staff) in order to determine an appropriate evaluation plan.

2) “I’m happy to let the experts handle things.”

This is the person who hires a builder because she/he likes their work and says, “Design me a home.”  This person knows they’ll be happy with the results because they trust the expert, and they don’t have the time or the desire to get involved in the details.  Many organizations hire an evaluator with exactly the same idea of trusting the expert to handle the planning and execution of the entire evaluation process.

3) “I have a really good idea of what I want.”

This person knows exactly how many doors they want in their house—and what they want the doorknobs to look like as well. When working with an evaluator, this type of client has clear expectations about how they want the process to go. For this client the evaluator is contracted to perform specific tasks such as data analysis or on an as-needed basis.

My list above of course is not meant to be exhaustive or inclusive of all scenarios. Given the complexity of evaluations and environments, it is likely that there is a good mix of all three types in a single client. If you have other ways to work with an evaluator, please share!

Stella

A very interesting article came across my desk a couple of weeks ago. In it, the authors describe their investigation of wiki use in U.S. K-12 schools. As you may know, a ‘wiki’ is a website to which users can contribute knowledge and information. Wiki’s are edited in real-time through a web browser. Two popular wikis are Wikipedia (an online encyclopedia) and Wiktionary (an online dictionary). Wikis, in general, are hosted at specific websites and are often free. Anyone can start their own wiki on any topic or can contribute to others’ wikis.

Reich, Murnane & Willet were specifically interested in how wikis were being used in schools and whether or not wikis were being used to promote the development of students’ digital literacy. Reich et al. sampled 1% of the wikis from a site popular for hosting free, educational wikis and 255 of those sites could be linked to K-12 schools in the U.S. Each wiki was assessed to determine how the site was being used. Specifically, the assessment tool included the following questions, and each was answered “Yes” or “No”:

• Do students use the wiki to access classroom materials?
• Are students the primary contributors to the wiki?
• Do students credit the sources of their contributions to the wiki, such as a citation (for books or paper material) or a hyperlink for digital information?
• Do students use text formatting, such as bold or italicized text or bulleted lists?
• Do students respond to each other’s contributions to the wiki?

Each wiki was assessed on 24 unique characteristics and total scores were computed.

Although Reich et al. examined the general characteristics of the wikis, I found the most interesting section of the article to be an examination of wikis used in schools serving larger proportions of lower-income families (Title I-eligible schools) and other schools (non-Title I eligible). Reich et al. found substantial differences in wikis developed at Title I and non-Title I eligible schools:

• More wikis in Title I-eligible schools remained undeveloped (e.g., contained an auto-generated front page only) or were teacher-only sites (50% for Title I-eligible schools and 30% for non-Title I);
• There were also differences in the median time the wiki remained active for Title I and non-Title I eligible schools (6.5 days for Title I-eligible and 32 days for non-Title I eligible schools); and
• Fewer wikis at Title I-eligible schools promoted high levels of student digital literacy compared to non-Title I eligible schools (17% at Title I-eligible and 36% at non-Title I eligible schools).

In other words, although the “space” for educational wikis is free to all teachers and their students, not all students are afforded the opportunity to develop high-level skills in digital literacy that could be learned in a wiki environment. Fewer students in schools with larger proportions of lower-income families have the opportunity to improve their digital literacy through wikis while more students in other communities can develop their digital literacy skills. The authors concluded the article with a description of how students’ online contributions to wikis also have the potential for student assessment.  As Reich et al. point out, online contributions to wikis by students represent long-term learning of how to solve nebulous, ill-structured problems and how to collaborate with others. In addition, using data from students’ online contributions to assess learning is incredibly efficient compared to traditional assessments because online contributions are integrated into the learning process. As a result, there is no loss of instructional time in order to assess what students have learned!

For more information, please see:

Reich, J., Murnane, R., & Willett, J. (2012). The state of Wiki usage in U.S. K-12 schools: Leveraging web 2.0 data warehouses to assess quality and equity in online learning environments. Educational Researcher, 41, 7-15. doi: 10.3102/0013189X11427083

Kirsten

Have you ever been asked to attend a meeting and you were unclear about the meeting purpose? What about a meeting or conference call with more people than the subject or time could accommodate? How about a meeting that was called at the last minute and was not well planned? Or one in which the technology would not function?

Often, it’s easy to spot what makes a ‘bad’ meeting, but what makes a ‘good’ meeting is often more difficult to determine. A substantial part of a good meeting occurs behind the scenes with the facilitator’s pre-meeting preparation. When facilitators’ take the time to carefully prepare, the meeting is usually smoother, more productive, and less stressful for everyone involved. So what should a facilitator focus on during preparation? In my experience, and review of articles on conducting successful meetings, I have found that to conduct the most effective meeting the facilitator should:

• Have a clear objective for the meeting. Meet for a purpose, not just “because you can” or “think you should.”
• Have an agenda for the meeting. Know what goals or objectives you would like to achieve and share that information ahead of time with the meeting participants.
• Be mindful of the number of participants you invite when feedback is required. With too many people, it is difficult for all participants to share their thoughts and for decisions to be made. Although many contextual factors (amount of necessary feedback, interpersonal relationships, time allocation, etc.) will contribute to how many people you should invite, some of the literature has suggested that no more than 12 people make for a good environment for everyone to be heard.
• Send electronic copies of meeting documents to members beforehand. This will allow all participants to adequately prepare themselves for the meeting and to contribute to achieving the meeting’s goals.
• Give everyone enough time to prepare for the meeting. For a meeting that lasts at least one-hour, anticipate 3-5 business days for preparation.
• Ensure technology is working appropriately. Test the technology prior to the meeting to make certain it will work the way you want.

If you are interested in successful meeting facilitation and would like to learn more, try these websites:

• http://www.arma.org/bookstore/files/Francisco.pdf
• http://www.mindtools.com/CommSkll/RunningMeetings.htm
• http://www.dummies.com/how-to/content/conducting-effective-business-meetings.html

Are there any keys or strategies you have found particularly effective? If so, please share!

Dan

Non-Equivalent Dependent Variables (NEDV) may sound complicated but, like a number of other techniques in statistics, the idea is actually pretty straightforward. In essence, NEDV are two (or more) unequal variables that are used together to detect change. Let me illustrate with an example. Say we have a great new method to teach grade school kids how to spell a certain number of words; let’s say 10. We are using a new way of spelling that we think will have a great effect on kids’ ability to spell, so we test kids on the 10 words before we teach them with our new technique (pretest) and then again a few days after we teach them (posttest). We expect to find that kids will get very few if any words right on the first test and most or all right on the second test and this will show that our teaching method was effective. However, you might worry that improvements from pre- to posttest might be due to something other than the instruction we are providing. For example, many children’s television shows teach spelling or parents may work with their kids at home on spelling. If kids received this extra support above and beyond your instruction and showed great results on your 10 word spelling posttest we could not be sure how much of their posttest performance was due to our program or caused by the extra support. Maybe without our program kids would have done as well with just the extra support and our program is wasting resources.

NEDV are used in situations like the one just described because they can be effective at detecting if external factors impacted the results of a study (like our pretest/posttest example). In a typical pretest/posttest study, kids would be asked to spell 10 words on the pretest and spell the exact same 10 words on the posttest; this would be ‘equivalent dependent variables’ because the words are identical. But in an NEDV study kids would spell 10 words on the pretest and 20 words on the posttest: the 10 words from the pretest and 10 new words. We would expect that if kids are receiving extra help they would correctly spell some of the 10 new words we did not teach them, and if we see that kids indeed correctly spelled those 10 new words then we can start to doubt the observed effects of our program on the 10 words we taught them. But, if we see that kids do not correctly spell the 10 new words but do correctly spell the 10 words we taught, then we can be more confident in the observed effects.

Joseph

Hi, I’m Sandy Donovan and I was invited to guest blog on ACET Inc.’s blog this week. I’ve worked with ACET for more than a decade on various evaluation projects. I’ve also worked in the areas of workforce development and career planning for nearly that long, so I thought I’d take this opportunity to write about an important workforce topic: ongoing professional development.

I’m guessing this may be an area that many readers have often told themselves they would begin to prioritize—just as soon as things get less hectic at work. After all, keeping up with one’s day-to-day work tasks can feel like two full-time jobs. It often seems like there aren’t enough hours in the day to work on projects, meet deadlines, head off crises, attend meetings, and reply to emails. . .the list can be endless.

In fact, everything we do at work builds our skills and bolsters our experience. But people with the highest career satisfaction consistently report that they also make an effort to keep one eye on the big picture of their career. This is true whether they already have a career they’re passionate about, or they’re just setting out on a path toward that career.

Why You Should. Dedicating just a few minutes, hours, days—whatever you have available—to your professional development will pay off in multiple ways. Here are just a few benefits of staying on top of your professional development:

• Be more informed about trends and issues in your field. Being aware of what’s happening in your field can keep you engaged and help grow your career.
• Gain new skills and advance in your career. Keeping your skills fresh is more important than ever as technology rapidly evolves.
• Remain passionate about the work you do. If you’re like many of ACET’s clients, chances are your career choice was driven by passion or a desire to make change. Keeping that passion alive will keep you inspired, help you to do better work, and in the end, increase your impact.

How You Can. Most of us agree we’d like to invest in professional development–but we’re not sure where to start, or if we even have the time. Here are a few simple ways to get started:

• Join a professional association. Once you join, you can determine just how involved you want to be. Find an organization in your field in this list of Professional and Trade Associations collected by the U.S. Department of Labor.
• Take a class. Some employers will help you pay for classes, some won’t. You’ll find that anything from taking a community education course to a starting a degree program can boost your career.
• Go online. Social media tools can make professional networking easier than ever. Try finding a LinkedIn group in your field or following colleagues on Twitter.

You may be familiar with policy memos, those one-to-two page documents often delivered by policy analysts to inform decision makers. Policy memos are a remarkably useful tool for all sorts of information exchange. In many ways they are similar to executive summaries. Both policy memos and executive summaries:

• Effectively communicate key messages in a succinct manner;
• Present evidence from both sides of a debate (pros and cons or strengths and weaknesses); and
• Synthesize much information into one  easy to use document.

But there’s one key distinction between executive summaries and policy memos: Policy memos contain a call to action of some sort. Often, this takes the form of a recommendation to follow one course of action over another. Or, it may be a simple set of action steps.

At ACET we often use policy-type memos to communicate with clients, highlighting key findings, strengths, challenges, and next steps. You might want to consider whether this document style could be a similar useful tool in your organization.

Elmer’s Products, Inc. – the company that brings you Elmer’s Glue – is collaborating with the Kids in Need Foundation to provide grants to teachers for classroom projects. Elmer’s and Kids in Need anticipate providing approximately 250 grants to teachers to complete projects in the arts, language, history, math, social studies, science, and other areas (e.g., health and wellness, community service). Although first year teachers may receive special consideration, all K-12 teachers are encouraged to apply. Teachers must select one of the 500 projects included on the Tool Kit Grant website to implement in their classroom. Grants will range in size from $100 to $500 and must be completed by the end of the 2011-2012 school year. Awarding of the grants will be based on financial need, how the selected project meets the educational needs of the students and satisfies state standards, and the number of students who will benefit from participation.

To learn more about the Elmer’s & Kids in Need Foundation Teacher Tool Kit Grants, please visit their website at: http://www.kinf.org/elmers /index.php.

If you know of any other grants for teachers, please share them in the comments. Good luck!

Kirsten

Do you know someone who volunteers tirelessly to promote a better tomorrow? How about someone who volunteers and goes above and beyond what is expected of them? If you know of someone in which either of the previous is true, consider nominating them for a Community TechKnowledge (CTK) Cash Thank You Grant. CTK is a nonprofit that provides other nonprofits with software to manage staff and volunteers, track donors, and manage data for reporting. CTK recently announced they will recognize the efforts and contributions of unpaid volunteers of registered nonprofits or charities through the Cash Thank You Grants program. CTK will award five cash grants totaling $17, 500.

• Three $5,000 grants will be awarded to nonprofit heroes in Animal Rights and Environmental Protection, Health & Human Services, and Arts & Literacy.  The winners will participate in a video public service announcement-produced by a professional-about their nonprofit.
• One $1,000 grant will be awarded to a volunteer and will be selected by the blogger who best promotes the “Heroes with a Heart” campaign.
• And one $1,500 grant will be provided for a volunteer who best provides creative and innovative approaches to HIV/AIDS education or prevention.

Submissions are being accepted from Feb 1– Feb 29. When nominating, you will be asked to provide a 250 word description on why your nominee is an exceptional hero. A judging panel will select a group of finalists and the public will have the opportunity to vote on their favorites through CTK’s Facebook page. Voting will take place between March 15 and April 15. To nominate a hero, or to learn more about the “Heroes with a Heart” campaign, please visit: http://www.facebook.com/CommunityTechKnowledge?sk=app_264413976964065.

Dan

This past Monday, I was invited to participate in a program advisory committee meeting for the Master of Science in Applied Psychology program at the University of Wisconsin-Stout. This committee is made up of current students, professors, and professionals working in evaluation who provide feedback on the program’s goals and future. This meeting was not only important for the program, but also for the field of evaluation. The advisory group’s input was needed for a variety of topics, such as: the current program status, internship requirements, alignment with other similar programs, new applicants, etc. As the Master of Science in Applied Psychology program prepares future evaluators, alignment to future workforce needs and skillsets is key to program success and why input was sought from multiple perspectives. I greatly appreciated being asked to serve as a committee member and having the option provide program input.

To learn more about the Master of Science in Applied Psychology program at the University of Wisconsin-Stout, please visit: http://www.uwstout.edu/programs/msap/.

Dan