Comparing Results from Benchmarks When the Questions and Standards Are Different

The administration of benchmark assessments is a common practice with which most educators are familiar. Benchmarks assessments, sometimes referred to as interim assessments, are intended to measure a student’s mastery of skills in a content area against grade-level standards and learning goals. But what happens when the assessments vary? What do you do when the questions and standards are not the same across benchmark assessments? How do you evaluate student progress when the benchmark assessments are not identical to one another?

The team of researchers at ATI make it possible to answer these questions by implementing state-of-the-art measurement techniques. The process begins by using Item Response Theory (IRT) scaling techniques to place student scores on a series of Galileo assessments on a common scale. The relationship between student ability and item difficulty plays a particularly important role in the IRT scaling process. In IRT, student ability estimates and item difficulty estimates inform one another within the context of the same mathematical model. Student ability is estimated in light of the relative difficulty of the items on the test, and the difficulty of the items is estimated in light of the ability level of the students who responded to them. For example, in a common IRT model, a student of average ability will have a fifty-fifty chance of responding correctly to an item of average difficulty. A student who is one standard deviation above the mean ability level will have a fifty-fifty chance of responding correctly to an item that is one standard deviation above the mean in terms of difficulty. Likewise, a student of below average ability will have a fifty-fifty chance of responding correctly to a corresponding item that is below average in difficulty. 

The fact that ability and difficulty are measured on the same scale makes it possible to adjust the student’s scale score, which is an estimate of ability, based on the difficulty of the items included in the assessment. This adjustment is a key factor in the scaling process making it possible to compare scores from different tests. When scores such as percent correct are used, such adjustment is not possible, and scores from different tests cannot be compared. For example, if a student received a score of 70 percent correct on one test and 90 percent correct on a second test, the difference could have occurred because the second test was easier than the first, or because of an increase in student performance, or both.  

In this regard, a powerful report to check out in the Galileo Help files is the Item Parameters Report. The report specifically provides information about item difficulty and other item parameters for each item on a benchmark test. Other helpful reports to check out are the Aggregate Multi-Test and Student Growth and Achievement reports, which present student IRT scale scores, which are called Developmental Level, or DL scores in Galileo, on a series of tests so that student progress may be monitored.

Watch the brief video on Psychometrics 

To learn more about ATI’s research initiatives, visit our website. For a one-on-one demonstration of the reporting features mentioned in this blog, request a personal demo with one of our knowledgeable field services coordinators. 

Other topics of interest:
How to use item parameters to make decisions during test review
How does ATI calculate my district’s psychometrics benchmark test data?

Why A Wide Range of Assessment Options is Crucial for Measuring Student Progress

Academic progress is an important goal in the educational process. The value of finding the right tools to measure academic progress is paramount when planning for next instructional steps and in evaluating the educational process. What are your assessments telling you? Are they giving you the information you need to make decisions? 

All assessments provide direction. However, it is the combination of standards-based assessments crucial to defining a comprehensive assessment system that provides the data needed for student progress, instructional improvement, and goal attainment. Each type of assessment serves a different central purpose. ATI’s Galileo comprehensive assessment system includes a wide range of assessment options including benchmark, formative, screening and placement tests, plus interim and final course examinations, pretests and posttests, and instruments documenting instructional effectiveness. The results from the variety of assessments can be placed on a common scale. The use of a common scale makes it possible to directly compare results from different assessments. The availability of information from multiple measurement sources provides the most complete picture of academic accomplishments which in turn is crucial in sound next-step decision making. 

Each type of assessment serves a different purpose, for instance:

• benchmark and formative assessments are designed to inform instruction
• pretests and posttests are useful for measuring academic progress 
• analyses of instructional effectiveness assessments can identify classes and schools that are highly successful as well as those needing additional assistance 
• screening instruments are useful in identifying students at risk for learning problems 
• placement tests inform grade level placements, and advanced course placements 
• computerized adaptive tests provide efficient measures of academic proficiency
• automated construction of computerized adaptive tests increases testing options and enhances the capability to meet unique local needs 
• observational and rating-scale assessments provide authentic measures of competencies in the environment in which those competencies are used and they provide immediate information that can be used to guide instruction

The Galileo Comprehensive Assessment System

Just as there are multiple types of assessments that can be administered, there are often a variety of testing environments. In order to provide a consistent testing experience and ensure that students are not searching the internet, accessing other applications, or do anything else that might undermine the integrity of your assessment, ATI is introducing the Galileo Student Secure Browser.  The secure browser “locks” students into the browser until the assessment is completed. There are several ways to set up and manage the Galileo Student Secure Browser application, we recommend reviewing the guide “Set-up Assistance for the Galileo Student Secure Browser” to select which is best for your district or school. 

Learn first-hand about the Galileo complete comprehensive assessment system: 

• watch a brief video: Seeing the Big Picture with Galileo's Comprehensive Assessment System
• schedule a personal demo
• visit the ATI website 

Civics Test Update

A number of updates have been made to the Civics Test since Fall 2016.

1. Election results have been incorporated into the questions about state and national offices.
2. Missouri-specific content has been added to accommodate the Missouri Civics Test
3. Text-to-Speech functionality has been incorporated for those students who receive assessment accommodations.

Students have to pass 60% of the questions on the Civics test, which is based on the U.S. Immigration and Naturalization test. They can take it first in 8th grade and can retake it until they pass. There’s no limit to the number of times the test is administered.

Many of you are aware that ATI has been offering the Civics test for a while. The U.S. Immigration and Naturalization test is an oral test but this format was not practical in the school environment so multiple-choice items have been constructed to make testing efficient.

Civic Test Options
ATI offers two versions of the test: 1) only multiple-choice questions (102-110 items); 2) multiple-choice and educator-scored questions (100 items). The difference in length is due to the questions about the current U.S. Representatives. Questions 100-102/110 on the multiple-choice assessment ask about the representative of each of the congressional districts in a state, while there is just one, educator-scored question on the other version of the test. (i.e. "Who is your Congressperson?")

Requesting the Civics Test
It is helpful to specify which version of the Civics test you’d like when requesting the test. Multiple copies of this test can be provided. Please inform us of the number of congressional districts represented in your district when making the version request. Because the test is subject to frequent changes related primarily to changes in office holders, it is not always prudent to make requests too far in advance. ATI can offer the test for review. 

Requesting Post-Election Tests
If you last administered a version of the test before a state-wide/national election some items on the test relating to elected and appointed officials will change. Please note that these will be new items and will not be updated on tests that have already been delivered. Because it is necessary to change the Civics test often, ATI places the date of the revision in its title. This procedure will ensure receipt of the most current version and help avoid confusion.

We hope this is helpful information. If you have any questions, contact us at edmgmtsvsgrp@ati-online.com.

New One-Click Benchmark Performance Levels Reporting Widget by ATI

The new Benchmark Performance Levels reporting widget accessible from the Teacher Dashboard summarizes multiple measures of student performance on district benchmark assessments (e.g., mathematics, English language arts, and other subject areas) for your entire class at once. The colorful pie chart and interactive data displays of assessment results can be used to guide instruction, intervention, and enrichment using the Galileo Digital Curriculum Platform and Galileo online lessons (e.g., Dialogs). Benchmark Performance Levels reporting widget highlights follow.

One-click “Big Picture Class Summary” for each assessment including: 

• color-coded performance levels (e.g., highly proficient, partially proficient) including number and percent of students at each level
• easy-to-interpret Developmental Level (DL) Score range for each performance level
• instant access to number of students assessed, average percent correct, average DL Score, and average percent of standards mastered for the class

One-click “Detailed Class Portrait” for each assessment including:

• total points, percent correct, DL Score, and performance level for each student
• color-coded standards mastery level for each standard assessed (e.g., exceeds standard, meets standard)
• item count for each standard assessed
• one-click access to all items used to assess each standard, including item meta-data (e.g., Depth of Knowledge)
• rapid “Data Search” allowing search by student name, total points, percent correct, or DL score 
• one-click “Data Sorting” in ascending or descending order by student names, total points, percent correct, DL Score, percent of standards mastered, or standards mastery category  
• instant Comparative Analysis Report summarizing assessment results for the class, school, and district
• one-click “Excel Export” for use in conducting further analysis and for creating additional graphical displays such as pivot tables and bar graphs

Register to attend the Thursday, November 2 Learning on Demand session "Exploring the New Benchmark Performance Levels Reporting Widget." Margaret Johnson, ATI Professional Development Specialist, will walk you through accessing and interpreting data from the Benchmark Performance Levels reporting widget. She will illustrate how the summary data as well as the individual student data can be generated which helps to inform instruction and monitor student progress throughout the year. The Learning on Demand session is designed specifically for teachers and specialists responsible for instructional interventions and progress monitoring.

Register on the "Upcoming" tab in the Training Center.

One-click reporting options available in the new Benchmark Performance Levels reporting widget.

Additional Sources of Interest

Fall Learning on Demand Schedule
Dashboard Reporting
Comprehensive Assessment System - Webinar Recording

Critical Tipping Points in Gathering, Understanding, and Using Formative Data

Solutions for the 21^st Century Classroom

Presented by

Hosted by

Districts from across Arizona were represented at the Leadership Seminar to Optimize Implementation held last week at the Phoenix Elementary School District office. The room of packed educators, principals, directors of curriculum and instruction, directors of assessment, instructional coaches, and superintendents interacted and brainstormed with facilitators Susan Jacobs, School Improvement Facilitator Comprehensive School Assistance Program with WestEd and Sarah Callahan Estrada, Ph.D., Senior Research Scientist and Project Manager for Galileo Development with ATI as they guided them through the optimization of district and school initiatives related to gathering, understanding, and using formative data. 

“Tipping points are a reaffirmation of the potential for change and the power of intelligent action. Look at the world around you.It may seem like an immovable implacable place. It’s not. With the slightest push – in just the right place- it can be tipped.”Malcolm Gladwell

The morning began with a brief discussion of real world leadership goals and challenges complemented by a description of available support services from WestEd and ATI. The remainder of the interactive seminar focused on helping the leaders address four critical tipping points to maximize student and staff success in their schools and districts:

• teachers and students benefit from an engaging, relevant, and appropriately rigorous curriculum
• data-driven instruction must be based on trusted and authentic data
• instruction must respond to the implications of student performance data in targeted and authentic ways
• leadership requires the tools to combat initiative fatigue and data overload

Participants walked away with suggested strategies, resources, and action plans to implement immediately in their school or district. Leaders from both WestEd and ATI were available at the end of the seminar and provided needs evaluations to educational leaders. Together they identified next steps for long term solutions supporting a relentless approach to quality implementation of curriculum and instruction promoting student learning.

To find out more about the seminar, access the slide deck. If your school/district would like a needs evaluation, contact us at the following:

Jason Kane Feld                                                                               Bob Rosenfeld

Vice President, Corporate Projects                                                    Senior Manager, Field Services

Assessment Technology Incorporated                                               WestEd

800-367-4762                                                                                    415-717-5450

jason@ati-online.com                                                                        rrosenf@wested.org