Monday, June 25, 2012

How does ATI calculate my district’s psychometrics benchmark test data?

Statewide test data will soon become available to the public. As you begin to review your data from varying sources (e.g., district results from state-wide testing and from district specific assessments) you may be interested to learn how ATI is able to process and compare data. To generate ATI’s scaled scores (ATI refers to these as the Developmental Level Scores or DL scores) ATI uses an analysis based in Item Response Theory (IRT). IRT takes information about the difficulty of the items into account when generating the estimates of student ability (i.e., the DL scores).

What this means for you is that a change in DL scores is a direct measure of growth. For example, if a student obtains a DL score of 1000 on Benchmark #1 and 1100 on Benchmark #2, that would mean that the student’s ability increased by 100 points or one standard deviation. In contrast, by looking at raw scores or percent correct, you cannot be sure what it means if scores increased.

For example, if a student obtained 70 percent correct on Benchmark #1 and 80 percent on Benchmark #2, this increase might be related to the fact that the items on Benchmark #2 were easier or the fact that the student’s ability increased, or both. So DL scores are generally preferable to raw scores as a way to evaluate student progress; however, sometimes the results of this approach are not so intuitive. A teacher may see that his or her class improved in terms of percent correct from Benchmark #1 to Benchmark #2, but that their DL scores have actually decreased. In this case, the items on Benchmark #2 were probably easier than those on Benchmark #1, so essentially the students didn’t get as much “credit” for getting them right.

Once ATI has the DL scores, ATI sets the cut scores that correspond to the achievement levels (e.g., Exceeds, Meets, Approaches). ATI uses two approaches to set cut scores. The first approach is called equipercentile equating. Under the equipercentile equating approach, ATI aligns the distribution of student scores on the benchmark assessment to the distribution of scores on the analogous state assessment for the same district (i.e., we align the third grade math benchmark to the most recent third grade state standardized assessment, such as Arizona’s Instrument to Measure Standards [AIMS] in Arizona). We identify the percentile ranks at which students in the district attained the various cut scores on the state standardized assessment, identify the same percentile ranks in the distribution of scores on the benchmark assessment, and the DL scores at those percentile ranks become the cut scores on the benchmark assessment. This allows ATI to identify not only how many students are likely to pass the state standardized assessment at the end of the year, but which students. Analyses of the accuracy of forecasting indicate that the equipercentile equating approach is highly accurate in forecasting which students are likely to show mastery on the state test and which are not.

Learn more by reviewing the Benchmark Assessment in Standards-Based Education research paper.

Experience Galileo for yourself. There are a number of ways to learn first-hand about Galileo K-12 Online. You can:
  • Visit the Assessment Technology Incorporated website (ati-online.com)
  • Participate in an online demonstration by registering either through the website or by calling 1.877.442.5453 to speak with a Field Services Coordinator
  • Visit us at the
    • Arizona Department of Education Leading Change Conference June 26  through 28 at the Westin La Paloma in Tucson, Arizona.
    • Massachusetts Association of School Superintendents Executive Institute July 11 and 12 at the Mashpee, Massachusetts.
    • Arizona Association of School Business Officials 59th Annual Conference and Exposition July 18 through 21 at the JW Marriott Star Pass Resort and Spa.
    • Colorado Association of School Executives Conference July 23 through 27 at the Beaver Run Resort, Breckenridge, Colorado.
We look forward to communicating with you online or at events.

Monday, June 11, 2012

Introducing Site Bytes


A new feature has been added to the ATI corporate websiteSite Bytes. Site Bytes provide short bits of information on key topics of special interest to those in the education field.

Recent “Bytes” include:
ATI and Arizona school districts launch Community Item Banking and Assessment Development Project

Item banks expand for non-state-tested subjects and grade levels, including foreign language and social studies, in Community Item Banking and Assessment Development Project

Be sure to return to our home page regularly to keep up-to-date with the latest information.

Monday, June 4, 2012

Community Item Banking and Assessment Development Project

Assessment Technology Incorporated is pleased to offer Community Item Banking and Assessment Development Project workshops. This month, ATI is working with the first registered participants in project workshops to develop assessment item specifications, items, and supplemental materials relevant to non-state-tested subjects/grades and for community sharing. These materials will enable all participants to utilize the resulting shared item banks in development of assessments to be used in meeting the requirements of various state educator effectiveness legislation. We invite you to join us in this mission to develop accurate and reliable assessments of student growth for non-state-tested subject areas and grades by registering to participate in the project.

In each of the two-day workshops, participants get hands-on training in the use of ATI's Galileo K-12 Bank Builder application. This is the same tool that is used in the construction of ATI's English language arts, math, and science benchmark and formative item banks. Working collaboratively with each other and with ATI, those attending the workshop are assisted in developing proficiency in constructing item specifications and items that can be used in construction of valid and reliable assessments. Once administered, these assessments can be scored online and the results reported using Galileo's innovative technologies.

Workshop participants develop items in their subject areas to measure student achievement and through an integrated review process enhance and expand the assessment items they have developed. The integrated review process is facilitated both by the expertise of project participants and by use of ATI expertise and proprietary technology. The workshops also focus on assisting participants use these teacher-created items. In the workshop, participants explore Galileo's statistical capabilities and reporting features including the ways in which they can help demonstrate the growth in student comprehension and proficiency.

ATI will be organizing repeated sessions of the workshops later this summer and in the fall of 2012 to assist districts in the continuing expansion of these shared district-developed item banks for assessment of non-state tested subjects and grades.

ATI would like to thank Marana Unified School District’s Mountain View High School and Scottsdale Unified School District’s Chaparral High School for hosting the first rounds of workshops. If you are interested in registering for the project and participating in a future workshop, please contact your Field Services Coordinator at 1.800.367.4762 or at GalileoInfo@ati-online.com.