Given the complexity of the program evaluation process, planning represents the single most important stage in effective program evaluation. Decisions made in the planning process will guide all other stages by defining the methodology and scope of the inquiry and by creating shared evaluation processes and documents. This framework, created by Hanover Research by examining a variety of industry planning guides as well as program evaluation and management literature, is commonly implemented by us as it enables administrators and their staff to plan project evaluations in a systematic manner.
Step 1: What are you planning to evaluate?
In order to ensure accurate records are maintained, list the program under evaluation on a dedicated planning document. This should include: Program title and brief description of the program, program eligibility selection criteria, and potential confounding variables for the participating students (factors outside of the program under evaluation that may influence the program’s effects). Many times identifying these criteria and variables can be challenging, and partners have consequently depended on Hanover’s program evaluation expertise during this stage of the planning process.
Step 2: What is the purpose of the proposed evaluation?
Clearly stating the purposes enables all individuals responsible for the evaluation process to share an understanding of why the evaluation is critical, and it enhances the focus on a single set of questions that need to be investigated. Make sure to include specifics on why the program is being evaluated and the type of evaluation being planned: formative or summative? Additional tip: Often times, program evaluations can be politically charged; to mitigate the risk of political influences, we recommend you obtain an objective third-party analysis.
Step 3: Who will use the evaluation? How will they use it?
Not only is it important for the analysis to be completed by an organization with a reputable research background and from a neutral standpoint, it is also imperative that it addresses all the questions key stakeholders reviewing it may have. After identifying audiences and stakeholders, an important step is to work collaboratively to shape the questions being asked, determine the way in which the results will be communicated, and decide what the tone, applicability, and actionable advice of the results should look like. Please note: Sometimes this collaboration requires primary research that can be time-consuming and expensive if done on an ad-hoc basis.
Step 4: What key research question(s) will the evaluation seek to answer?
A common challenge Hanover Research hears from partners is how to create a comprehensive report that is actionable and digestible for decision makers and stakeholders who may not be intimately involved with the program. Creating a list of questions can provide stakeholders with a focused scope and the ability to prioritize analysis and recommendations in a summary report. Partners will typically task Hanover to assist with creating such a list of questions and the necessary data points in order to ensure the validity and reliability of the report.
Step 5: When is the evaluation needed?
It is important to consult with administrators and staff tasked with conducting the evaluation and to assess realistic timelines and available bandwidth. Does your district possess the dedicated internal resources to prioritize, schedule, and proceed with the evaluation process in a timely fashion? For several partners, Hanover has served the dedicated source for program evaluations, resulting in a significant increase in the number of discrete program evaluations being completed within each academic year.
Step 6a: What stages are required for this evaluation?
Step 6b: Who will be responsible for each stage? When will each stage be completed?
By creating a map of the necessary stages of an analysis, and by assigning responsibilities and timelines to those stages, everyone can easily be held accountable as the evaluation progresses. It will also be easier to pinpoint breakdowns within the process. This is the stage at which many organizations become overwhelmed. With several different people wearing multiple hats, it can become tricky to course-correct and complete the evaluation on time. Many of Hanover’s partners have relied on having their dedicated project manager keep track of the evaluation so that they can carry out their primary responsibility: student success.
In performing the above planning steps, your organization should be well-positioned to successfully execute an evaluation. However, if you have additional questions or would like expert assistance with program evaluation planning or analysis, e-mail us to speak with a K-12 research director.
