Forecasting Risk and Making Predictions about AMOs

If you’re a user of Galileo K-12 Online, or an avid reader of this Blog, you’ll know that one of the primary services Galileo provides is forecasting student risk of not meeting state standards based on their performance on benchmark assessments. The information is designed to help identify students in need of intervention, but it is tempting to use it to get a preview of where the district or school stands with regard to AMOs. The data can be used for this purpose, but I wanted to highlight some points to consider when doing so.

The Aggregate Multitest report can be run in two modes and the choice is made by selecting either the Display risk levels or the Display benchmark performance levels radio button. When the Aggregate Multitest report is run with the benchmark performance levels option, it generates an independent analysis of student performance for each of the selected assessments. Based on their performance on each benchmark assessment, students are categorized into the same classification system that the statewide assessment uses, such as the FAME scale in Arizona or the Advanced, Proficient, Needs Improvement, or Warning categories in Massachusetts. What you’ll see below the bar graph is a display that shows the percent of students in each category, such as this:

It is easy to see how this data would tempt a person to try to project where the schools in the district are likely to stand relative to the AMOs at the end of the year. Let’s say that in this example, the AMO for 4th grade math is that 66% of the students must be classified as either Meets or Exceeds on the statewide assessment. Adding these categories together for benchmark #1 indicates that 61.71% fall into these categories. That’s not quite enough. On benchmark #2 the number rises to 65.66%, which is close, and then finally with benchmark #3 the figure of 72.80% surpasses the AMO.

That’s good news, right? Well, maybe. Probably. Our research has indicted that Galileo benchmark assessments are very good at forecasting likely student performance on statewide assessment. But our research also indicates that considering student performance on multiple benchmark assessments yields even more accurate forecasts of student performance on statewide assessments than considering student performance on individual benchmark assessments in isolation. This is true even when the one, isolated benchmark assessment is the one that is administered most recently when statewide testing begins, as would be the case with benchmark #3 in this example. Details of these investigations can be found in the Galileo K-12 Online Technical Manual.

In order to capture the increased accuracy of data from multiple benchmark assessments, the Galileo K-12 Online Risk Assessment report was developed. The Risk Assessment report is accessed via the second mode for generating the Aggregate Multitest report, by selecting the radio button that says display risk levels. The Risk Assessment report provides the same information Kerridan referred to in her recent post (How Can Galileo Assist in Interventions?), except that the data can be aggregated at the school or district level as well as the classroom level, and it yields a display that looks like this:

Students are classified into the different levels of risk according to their performance on a series of benchmark assessments. This example refers to the same three 4th grade math assessments that were considered earlier. Students are classified as being “On Course” if they scored above the cut score for “meets” (or “proficient” in many other states) on all three benchmark assessments. If they fell below that cut score for one assessment, they are classified as being at “Low Risk”. If they fell below on two of the three assessments, they are at “Moderate Risk”, and students who never scored above the critical cut score are classified as being at “High Risk”. Kerridan’s blog illustrates how to use this information to plan interventions.

This method of projecting student risk of not meeting the standard on the statewide assessment has proven to be very accurate. On average, 96% of students who are classified as being “On Course” after three benchmark assessments go on to demonstrate mastery on the statewide assessment (see the Technical Manual). This is the most accurate approach to projecting student risk for the purposes of identifying students for intervention efforts. It is also the most accurate information a district could have when assessing risk with regard to AMOs. However, because its primary function is to identify groups of students for intervention efforts, its format may not be the most convenient for looking toward AMOs. In this case, only 52% of the students are on course, which is well below the AMO of 66%. But a district can count on a number of students from each of the other risk categories to pass the statewide assessment as well. We have conducted a preliminary investigation, based on 7 large school districts, to see how many students in each risk level category tend to go on to demonstrate mastery on the statewide assessment. The results are presented in the following table.