How to Utilize the Galileo Pre-K Online DL Score to Evaluate and Support Early Learners

The Galileo Pre-K Online Developmental Level (DL) Score is a summarized measure of ability. The DL Score indicates a child’s position on his or her path of developmental progress. When generated for a group of children, it will reveal the group’s average position on the path of development. The path is comprised of a series of empirically ordered capabilities outlining a developmental progression for each scale or developmental area. 

When we know the ability of a child or a group, as summarized by the DL score, we know the capabilities that have been learned and the goals that a child or group is likely ready to learn next and in the future. The DL Score is a valuable data point for planning and monitoring progress towards school readiness.

After the teacher contributes his or her knowledge of a child’s ability by saving observations, the DL Score provides specific information on emerging capabilities by identifying “Ready Now” or “Plan Now” goals. These readiness suggestions provide the basis for planning appropriately challenging instruction.

For DL Score interpretation reference, please see the following links to our research:

•         Estimation of how much the DL Score typically increases based on the length of time enrolled
•        DL Score ranges with corresponding achievement levels of beginning, intermediate, or advanced for children of age 3-5 years and birth to 3 years
•         Predicted individual child DL Score based on the child’s age

 The DL Score can communicate progress to educators at home and in the classroom, identify an intervention opportunity, and inform which skills are ready to be learned for all children at all levels.

Visit ATI Exhibit #527 at the National Head Start Association Conference in Chicago April 7-10 or schedule an online Galileo Overview for more information.

Written by Deborah Kinzer, Field Services Coordinator

One Test, Many Uses

“Help! I’m stuck between my administrators who want to use the December benchmark as a predictive and my teachers who want to use the December benchmark as a summative semester final. What can I do?” 

One single benchmark, if created appropriately, can serve both purposes. 

How? 

Every state is required to have a list of standards students at each grade level need to master. It is the district’s responsibility to make sure that every child masters these standards by the end of the school year. In order to ensure this goal is achieved, many districts provide pacing guides to teachers in order to keep both teachers and students on pace to master all standards by the end of the year. These pacing guides identify which standards are supposed to be taught at what point during the year. The benchmark assessments are then created based on the district pacing guide. 

The ideal benchmark assessment should be between 35-50 items long and have no more than five items on one specific standard. Using the standards taught during the first half of the year can provide both a reliable predictor of students’ progress towards mastery of the state standards and a valid summative assessment for what students were taught in the classroom. The question then becomes how do teachers take the results of these benchmark assessments and translate them into grades?

The answer to this question depends on the district’s method for grading students. The easiest and most straightforward method is a standards-based grading system. This method of grading provides information as to whether a student has mastered a standard or skill or where he or she is at in developing the skill. Information on standards mastery can be obtained using the Galileo® K-12 Online Intervention Report.

Using benchmark assessments for a more traditional method for grading, (e.g., providing a letter grade for students based on percentages) may need more thought. In order to provide accurate ability estimates for students at all ranges of ability, it is important that there is a range of difficulties on the items and that even the student at the highest academic levels needs to be challenged. As a result, sometimes the raw scores do not represent the level of growth and success students have actually demonstrated. One example is an assessment where the average raw score percentage was 46 percent, yet students demonstrated an average growth of 20 points on their Developmental Level scores (DL). Teachers and districts can use Galileo’s benchmark data  when assigning traditional grades. One suggestion to assign grades is to generate a classroom Benchmark Results report.  The report provides information for each student’s risk assessment. By providing a grade (e.g., A for On Course, B for Low Risk, C for Moderate Risk, and D or F for High Risk), valid and reliable student data may be converted into a traditional grading system.

- Karyn White, M.A.
Educational Management Services Director

The Solution to One Administrative Nightmare

The new state laws requiring school districts to rate their teachers have inspired many of the districts I work with to take a hard look at new methods for gathering data and come up with new and creative methods for complying with these laws. Although many of these plans are effective once implemented, the district administrators are faced with the tasks of complex calculations and added paperwork in order to rate each district staff member. At times, this added load becomes a lot for even the most organized administrators to handle.

Here is an example of the kind of challenge faced by a district evaluating the effectiveness of  a 4th grade teacher.
                           
Districts often base teacher evaluations on a point system. In our example, we assume that the maximum teacher evaluation score is 1000 points. We then divide ranges of points into evaluation categories as follows:

• 750-1000              Highly Effective
• 500-750                Effective
• 250-500                Developing
• 000-250                Ineffective

Teachers earn their points based on the following criteria. The teacher rating scale (observation rubric) is worth 100 points. State test growth percentile is worth 100 points. ATI Galileo DL scores or other district determined student growth measures are also factored into the teacher score. In order to get the teacher rating, the administrator multiplies the total points the teacher earned on the rating scale by 5. Then the administrator adds up the state test growth percentiles in math, reading, and science. The administrator then computes the average DL score changes on all benchmarks and adds the DL score changes to the total score. The teacher is then rated based on predetermined categories listed above. 

Ask yourself how long will it take the administrator to be able to evaluate, calculate, and determine the appropriate rating for all of the teachers being evaluated. 
             
The answer to this question is likely to be a long time even in a relatively small district. ATI has a simple, efficient and accurate solution to this challenge. It is a score compiler. The compiler will take data from various sources and calculate a rating for each teacher. ATI enters staff rating scales including  (evaluation rubrics) into Galileo. Administers then can electronically complete scoring observations online. For example, scoring may occur during a scheduled teacher observation. The results of the evaluation are calculated immediately and pulled into the compiler. State test data can be uploaded into Galileo and pulled into the compiler. Finally, ATI pre/posttest data provides information about whether each teacher and/or school has met expected growth, not met expected growth, or exceeded expected growth. This data is also retrieved and placed into the compiler. All of the information as well as any other required information (e.g., other assessment data, surveys, informal observation, school-wide data) is then compiled to provide a rating for each staff member without the administrator needing to complete any of the calculations.

ATI Findings on Predictive Validity and Forecasting Accuracy for the 2011-12 School Year

ATI has released a research brief summarizing current research on the predictive validity of Galileo K-12 Online assessments administered in the 2011-12 school year and the forecasting accuracy of Galileo risk levels based on student performance on these assessments.

The research summarized in the brief was based on data for individual students in grades three through high school in math, reading/English language arts, and science. The sample consisted of the first 26 districts in Arizona, Colorado, and Massachusetts to provide ATI with their statewide assessment data. Collectively, these districts administered 1,105 district-wide assessments.

ATI conducts an Item Response Theory (IRT) analysis for each district-wide assessment which produces a scale score for each student, the Developmental Level (DL) score. Each student is also classified as to their level of risk of failing the statewide assessment based on their performance on all the district-wide assessments they have taken within a given school year. In order of highest to lowest risk of failing the statewide assessment, the possible risk levels comprise “High Risk,” “Moderate Risk,” “Low Risk,” and “On Course.” ATI then evaluates predictive validity by examining the correlation between student DL scores on each district-wide assessment and student scores on the statewide assessment. ATI evaluates forecasting accuracy by examining how students classified at different levels of risk ultimately performed on the statewide assessment.

“Predictive validity analyses examine the strength of the relationship between two measures of student performance, in this case the student DL scores on an assessment in a given grade and content area and the student scores on the statewide assessment in the same grade and content area,” says brief author Sarah Callahan, Ph.D., Research Scientist of Assessment Technology Incorporated. She further states that “The observed correlations in the 26 districts studied suggest that student scores on the 2011-12 Galileo district-wide assessments were strongly related to student scores on the 2012 statewide assessment.”

Key findings include:

• The mean correlations range from 0.69 to 0.78 across grades and content areas with an overall mean of 0.75 which is considered a high correlation.
• As student risk level increased the likelihood of failure on the statewide assessment increased, as illustrated in Figure 1.
• Overall Galileo risk levels accurately forecast statewide test performance for 84 percent of students as shown in Figure 2. 
• Forecasting accuracy was highest in cases where student performance was most consistent.

Dr. Callahan concludes that, based on this research, the 2011-12 Galileo assessments demonstrated adequate levels of predictive validity. The results also suggest that the 2011-12 Galileo risk levels displayed adequate levels of accuracy in forecasting student performance on the statewide assessment. This research is consistent with similar research investigations performed in previous years and suggests that Galileo assessments and risk levels continue to demonstrate adequate levels of predictive validity and forecasting accuracy. Learn more by reading the full three-page brief. 

How does ATI calculate my district’s psychometrics benchmark test data?

Statewide test data will soon become available to the public. As you begin to review your data from varying sources (e.g., district results from state-wide testing and from district specific assessments) you may be interested to learn how ATI is able to process and compare data. To generate ATI’s scaled scores (ATI refers to these as the Developmental Level Scores or DL scores) ATI uses an analysis based in Item Response Theory (IRT). IRT takes information about the difficulty of the items into account when generating the estimates of student ability (i.e., the DL scores).

What this means for you is that a change in DL scores is a direct measure of growth. For example, if a student obtains a DL score of 1000 on Benchmark #1 and 1100 on Benchmark #2, that would mean that the student’s ability increased by 100 points or one standard deviation. In contrast, by looking at raw scores or percent correct, you cannot be sure what it means if scores increased.

For example, if a student obtained 70 percent correct on Benchmark #1 and 80 percent on Benchmark #2, this increase might be related to the fact that the items on Benchmark #2 were easier or the fact that the student’s ability increased, or both. So DL scores are generally preferable to raw scores as a way to evaluate student progress; however, sometimes the results of this approach are not so intuitive. A teacher may see that his or her class improved in terms of percent correct from Benchmark #1 to Benchmark #2, but that their DL scores have actually decreased. In this case, the items on Benchmark #2 were probably easier than those on Benchmark #1, so essentially the students didn’t get as much “credit” for getting them right.

Once ATI has the DL scores, ATI sets the cut scores that correspond to the achievement levels (e.g., Exceeds, Meets, Approaches). ATI uses two approaches to set cut scores. The first approach is called equipercentile equating. Under the equipercentile equating approach, ATI aligns the distribution of student scores on the benchmark assessment to the distribution of scores on the analogous state assessment for the same district (i.e., we align the third grade math benchmark to the most recent third grade state standardized assessment, such as Arizona’s Instrument to Measure Standards [AIMS] in Arizona). We identify the percentile ranks at which students in the district attained the various cut scores on the state standardized assessment, identify the same percentile ranks in the distribution of scores on the benchmark assessment, and the DL scores at those percentile ranks become the cut scores on the benchmark assessment. This allows ATI to identify not only how many students are likely to pass the state standardized assessment at the end of the year, but which students. Analyses of the accuracy of forecasting indicate that the equipercentile equating approach is highly accurate in forecasting which students are likely to show mastery on the state test and which are not.

Learn more by reviewing the Benchmark Assessment in Standards-Based Education research paper.

Experience Galileo for yourself. There are a number of ways to learn first-hand about Galileo K-12 Online. You can:

• Visit the Assessment Technology Incorporated website (ati-online.com)
• Participate in an online demonstration by registering either through the website or by calling 1.877.442.5453 to speak with a Field Services Coordinator
• Visit us at the
• Arizona Department of Education Leading Change Conference June 26  through 28 at the Westin La Paloma in Tucson, Arizona.
• Massachusetts Association of School Superintendents Executive Institute July 11 and 12 at the Mashpee, Massachusetts.
• Arizona Association of School Business Officials 59th Annual Conference and Exposition July 18 through 21 at the JW Marriott Star Pass Resort and Spa.
• Colorado Association of School Executives Conference July 23 through 27 at the Beaver Run Resort, Breckenridge, Colorado.

We look forward to communicating with you online or at events.

What is the DL Score… and, how can it be my Pre-K tool?

Firstly, what is it?

The Developmental Level (DL) Score is a path-referenced score and a summarized measure of ability. This score indicates a child’s position on the path of developmental progress. When generated for an aggregate group of children it will reveal the group’s average position on the path of development. The path is comprised of a series of empirically ordered capabilities outlining a developmental progression for each scale or developmental area.

Secondly, how is it helpful?

When we know a child’s or group’s ability, as summarized by the DL score, we know the kinds of things that have been learned and the things that the child or group is ready to learn now and in the future. The DL score is a data point for goal planning and monitoring progress towards achieving program-wide goals. For individuals, the DL score drives the identification of emerging skills.

Thirdly, how can I make use of it?

After the teacher contributes knowledge of a child’s ability by saving observations, Galileo will identify emerging capabilities as “Ready Now”/“Plan Now” goals. These readiness suggestions provide the basis for planning appropriately challenging learning opportunities. Because developmental ability progress is measured in terms of change in position on a scale, it will be helpful to know how the DL score correlates to achievement levels. Check this link to see the chart detailing each assessment scale’s DL score range and its corresponding achievement level of Beginning, Intermediate, or Advanced.

-Deborah Kinzer, Field Services Coordinator