This is the second blog in the series about effective strategies for reviewing and selecting instructional materials. Our first blog discussed planning a successful adoption, which included distributing and aggregating the results of a needs assessment. Now it is time to develop a rubric for the adoption. Developing a rubric from scratch can be daunting. More often, educators look for existing rubrics to customize. This blog provides guidance for creating or customizing a rubric to reflect a district’s priorities and state standards.
Identifying criteria and sub-criteria
In an instructional materials selection rubric, criteria and sub-criteria/guidance statements should describe observable and measurable features of instructional materials. The following resources will help you identify the features you want to include in your rubric:
- Research on features of high-quality materials;
- Best practices for the content area;
- The district’s instructional goals;
- The Needs Assessment feedback;
- The instructional focus of and topics addressed in the state standards.
Keeping the rubric to a manageable length can be one of the biggest challenges when designing a rubric. We have observed, that in order for selection committee members to use a rubric with fidelity, rubrics for selecting core materials should include no more than ten (10) criteria and up to five (5) sub-criteria for each criterion. Thus, the rubric should include the features in instructional materials that your district values most.
Drafting criteria and sub-criteria statements
Each criterion and sub-criterion should be written as a statement using precise, unambiguous language. Terms like consistently, throughout, and repeatedly may be used to describe features that should be reflected throughout the material. For example:
- The material is structured consistently and logically.
- Materials support coherence and connections between and within content at the grade level and across grade levels.
- Materials provide opportunities for students to engage in productive struggle through sensemaking that involves reading, writing, thinking, and acting as scientists and engineers.
It is not imperative to include sub-criteria/guidance statements for each criterion. If used, these statements should clarify or illustrate the criteria. For that reason, sub-criteria statements often include illustrative language, such as includes or does not include.
Judgmental language such as poor, well, exceptional, or excellent should not be included in the criteria or sub-criteria statements; rather, judgments should be addressed in the rubric’s rating scale. The format of the criteria and sub-criteria/guidance statements will affect the type and format of the rating scale and vice versa.
Developing the rating scale
Rubrics should include at least one rating scale. The rating scale allows users to express their level of agreement with each criterion and sub-criterion statement. There are many types of rating scales, including:
As mentioned above, the type of rating scale selected impacts how criteria and sub-criteria statements are formatted. The type and format of the rating scale also affect how the response data can be aggregated and disaggregated. Using a rating scale that assigns a numeric value for each response makes it easy to tally a score for each material from the aggregate response data.
The response choices in the rating scale should be easy to understand and distinguish. The scale should include a sufficient number of response choices to allow selection committee members to communicate their level of agreement with the criteria and/or sub-criteria statements clearly. We have observed that having between three and five response choices in a rating scale works best; too many choices (e.g., 1-10) makes it difficult to discern a meaningful difference in the materials’ aggregate scores.
Testing the rubric
Once a draft of the rubric is complete, it is prudent to run it by a few of the content-area teachers to make sure that it is a manageable length and easy to understand.
Training for effective implementation
Selection committee members must receive training to ensure that they interpret and apply the criteria and sub-criteria/guidance statements consistently. Optimally, district curriculum staff would create an evidence guide that provides text-based examples of the different performance levels for each criterion and key sub-criteria statements. If that is not possible, the selection committee should discuss performance-level examples before beginning the reviews. For example, one of the criteria in any instructional materials rubric should be standards alignment. Selection committee members should receive training on or at least discuss what alignment means and how to evaluate a material’s alignment to state standards.
If available, links to reliable third-party reviews of the materials being considered will facilitate committee members’ completion of the rubric for each material.
Conducting the reviews
While it may be necessary to conduct the reviews virtually, we have observed that having committee members in the same room while they are independently reviewing materials has important benefits. The spontaneous conversations that occur often prompt committee members to explore the materials more deeply and ultimately facilitate consensus building around which materials meet the district’s needs best. When that happens, the selection committee members tend to become strong advocates for the materials selected.
Convene the selection committee to discuss rubric results
Once completed rubrics have been submitted, the person managing the selection process should analyze the committee’s response data to determine for each grade level covered by the adoption, which material scored (1) highest overall, and (2) highest on each criterion. That data analysis should be documented in an easy-to-read format and provided to the selection committee members.
The selection committee should reconvene to discuss the rubric results with the goal of reaching a consensus about the materials to recommend for adoption. However, the district’s voting procedures should address how a material will be selected if consensus is not reached.
Document rationale for making recommendations
The rubric data and the selection committee’s rationale for each recommended material should be documented clearly. If the materials will be adopted by the school board, that documentation should be shared with the board. The documentation should be maintained in the district for several years in case the district’s use of the selected material is ever challenged by parents or the community.
Given the work involved in developing a rubric, a rubric-review process is generally used to select core materials. The next blog in this series will discuss a process for reviewing and selecting supplemental materials.
If you need additional guidance developing or customizing rubrics, check out our 2-hour online rubric development course. This course is designed for anyone who is managing an adoption process – large or small. Additionally, our online Selection Toolkit provides standards-aligned, customizable rubrics for math English Language Arts and science.