How do you compare benchmarks when the questions and standards are not the same? Can the results from one benchmark to another be compared?

You can compare the results from one Galileo benchmark test to another when the questions and/or standards are not the same because Galileo benchmark test scores are placed on a common scale. ATI uses standard scaling techniques available through Item Response Theory (IRT) to accomplish the scaling task. The scaling process utilizes information regarding item characteristics and their relationship to student ability.  The relationship between student ability and item difficulty plays a particularly important role in the scaling process.  IRT places the student ability score and item difficulty on the same scale. For example, in IRT a student of average ability will have a fifty-fifty chance of responding correctly to an item of average difficulty. Students who are above average in ability will have a fifty-fifty chance of responding correctly to a corresponding item that is of above average difficulty. Likewise, a student of below average ability will have a fifty-fifty chance of responding correctly to a corresponding item that is below average in difficulty. 

The fact that ability and difficulty are measured on the same scale makes it possible to adjust the estimate of ability based on the difficulty of the items included in the assessment. This adjustment is a key factor in the scaling process making it possible to compare scores from different tests. When scores such as percent correct are used, such adjustment is not possible. For example, if a student received a score of 80 percent correct on one test and 90 percent correct on a second test, the difference could have occurred because the second test was easier than the first, or because of an increase in student performance, or both. 

A district-level user can obtain information about item difficulty and other item parameters  for each item on a benchmark test by generating the Item Parameters Report for the benchmark test in question.

Click here to watch video. 

ATI uses psychometric techniques to guide the construction and validation of measurement instruments such as a benchmark assessment administered by districts. Psychometrics is the field of study that deals with the theory and technique of measurement of knowledge, abilities, and educational measurement.  

ATI begins analyzing data once the benchmark test is given and the testing window is closed. Districts continue to have access to several other reports, such as the Test Monitoring, Item Analysis, and Intervention Alert reports. ATI performs an analysis using psychometric techniques based in Item Response Theory (IRT) to create Galileo Developmental Level (DL) scores. 

View the new “The Psyche of Psychometrics” video to learn more.

Read more about Galileo benchmark assessments.