Race to the Top: Preparing to Choose, Implement and Manage Your Local Instructional Improvement System

Among the four core educational areas of innovation and reform within the Race to the Top initiative (Race to the Top Application for Initial Funding, CFDA 84.395A), is that of building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction and elevate student learning (p. 3). Specifically, the initiative calls for states in collaboration with local education agencies to “increase the acquisition, adoption, and use of local instructional improvement systems that provide teachers, principals, and administrators with the information and resources they need to inform and improve their instructional practices, decision-making, and overall effectiveness; and supporting participating LEAs and schools that are using instructional improvement systems…” (p. 31). As further articulated in the funding application, instructional improvement systems are technology-based tools and other strategies that provide teachers, principals, and administrators with meaningful support and actionable data to systemically manage continuous instructional improvement, including such activities as:

•instructional planning;
•gathering information (e.g., through formative, interim, and summative assessments;
•looking at student work and other student data);
•analyzing information with the support of rapid-time reporting;
•using this information to inform decisions on appropriate next instructional steps;
•evaluating the effectiveness of the actions taken

Such systems promote collaborative problem-solving and action planning; they may also integrate instructional data with student-level data such as attendance, discipline, grades, credit accumulation, and student survey results to provide early warning indicators of a student’s risk of educational failure (p.9).

The Race to the Top initiative provides a unique, exciting, and timely opportunity for school districts to achieve the goal of elevating student achievement beyond current levels, and to do so in ways that enhance our competitiveness in the global society. Among the many elements that will contribute to success in achieving this goal is the well-informed selection, implementation and management of a high quality local instructional improvement system. In a nutshell, the opportunity to change achievement by the end of each school year is within our reach when we are empowered with actionable information on student learning that helps us to adjust instruction throughout the year. Learning occurs on a daily basis, thus, it is reasonable to assume that the goal of increasing learning can and should be supported one day at a time. A well-designed, research-based, standards aligned instructional improvement system can help to address this need.

As part of our ongoing outreach initiatives to help school districts and states address Race to the Top goals, ATI recently hosted in Colorado a statewide collaborative seminar “Elevating Student Achievement: Exploring What Works”. The goals of the seminar were to:

•Share evidence regarding what is working in districts to elevate student achievement;
•Identify the critical components of effective initiatives built upon the use of an instructional improvement system and aimed at elevating student achievement;
•Discuss challenges faced in initiatives designed to elevate achievement, and the solutions being implemented by school districts; and
•Consider the various management procedures and technology for addressing challenges that may be encountered in implementing programs designed to elevate achievement.

The seminar was attended by school district leaders, educators, and researchers and included presentations by WestEd’s Local Accountability Professional Development Series Project Director (website), the Director of Student Assessment and the Principal Consultant for the Colorado Department of Education Office of Standards and Assessments (website), and Assessment Technology, Inc. (website). Among the successes of this seminar was the formation of grassroots discussion panels comprised of school district leaders in the areas of curriculum, assessment, planning, technology, and evaluation. These individuals often have the responsibility of coordinating the development of criteria for selecting their local instructional improvement system and for implementation and management of the system. Below is a sampling of contextual highlights and questions addressed by panel members:

•An effective instructional improvement system begins with the specification of goals to be achieved within a specified time period. What challenges make it difficult to specify essential standards to be targeted for instruction within a given time period? How can those challenges be addressed effectively?
•An effective instructional improvement system uses formative assessments and a common set of interim assessments within each grade and subject. What challenges make it difficult to adopt common interim assessments and how can those challenges be addressed?
•In an effective instructional improvement system, instruction is adjusted following each interim assessment by implementing re-teaching and enrichment interventions. What challenges make it difficult to implement re-teaching and enrichment? How can those challenges be effectively addressed?
•Implementing an instructional improvement system requires groups of administrators and teachers to work together to provide an assessment system that can be used to inform instruction for all students. What are some of the management challenges associated with designing, scheduling, and implementing customized formative and interim benchmark assessments? What are the challenges associated with designing and implementing curricular interventions based on information about student learning? How can these challenges be met?
•Re-teaching and enrichment interventions require unanticipated allocations of time and resources to the instructional process. How do you meet the challenges of allocating time and resources to re-teaching and enrichment?

If you would like information regarding how panel members responded to these and other questions that emerged during the seminar, please contact us at ATI and/or watch the seminar video to be posted soon on the ATI website. If you would like to share your comments on these challenges within the forum community, please do so here. And, if you have questions that you would like to pose related to instructional improvement systems and the Race to the Top initiative, please post them here for comment from ATI and the forum community.

Jason K. Feld, Ph.D.

Elevating Student Achievement Seminar - Submit a Question

ATI is hosting the Elevating Student Achievement Seminar: Exploring What Works on February 3 at the Hyatt Tech Center in Denver. If you’re interested in registering, please visit the seminar website for specific details about the event.

The seminar will be a dialog on instructional interventions and the elevation of student achievement. If you know of any others in your district interested in registering, please feel free to pass the following link on to them - www.ati-online.com/WhatWorks

We’re also encouraging those interested in submitting a question pre-event for seminar panel discussions on the following topics:

• The benefits of and challenges encountered when implementing an instructional improvement initiative.
• The benefits of and challenges encountered when managing an instructional improvement initiative.

E-mail your questions to GalileoInfo@ati-online.com and they will be considered for inclusion at the seminar. Questions related to the seminar that we are unable to address at the event will be answered here on the Town Hall Blog.

We look forward to hearing your questions!

So what are item parameters, anyway?

Item Response Theory (IRT) item parameter estimates play an important role in many of the reports generated in Galileo K-12 Online. In addition to describing item characteristics in the Item Parameter report, they help to determine student Development Level (DL) scores and to identify the standards that are recommended for re-teaching efforts in the drill-downs from the Risk Assessment report. But what are they?

The best way to understand what item parameters refer to is to look at an Item Characteristic Curve. On an item characteristic curve, which presents the data for one, specific item, student ability (based on their performance on the assessment as a whole) is plotted on the horizontal axis, with a mean of 0 and a standard deviation of 1. The probability of answering the item correctly is plotted on the vertical axis. Typically the probability of answering correctly is relatively low for low ability students and relatively high for students of higher ability.

The first example (Item #4) is an example of a great item as far as the parameters go. The b-value (difficulty) for that item was 0.689, which is a bit on the difficult side, but not too bad. The important point here is that b-values (item difficulty) are on the same scale as student ability. So what this example is telling us is that students at or above 0.689 standard deviations above the mean are likely to get the answer correct. Students below that point on the ability scale are more likely to answer incorrectly. The b-parameter is also known as the location parameter, because it locates the point on the ability scale where students start demonstrating mastery of the concept.



The a-value (discrimination) refers to how well the item discriminates between different ability levels. It’s how steep the rise is in the curve that shows the probability of answering correctly. Ideally, there is a nice, steep rise in the probability of answering correctly like the one for question 4. That indicates that there is a dramatic change in how likely it is that a student has mastered the concept that’s pin-pointed within a very narrow range of the ability scale. You can be pretty confident that students above 0.689 standard deviations above the mean “get it” and that students below that point generally don’t. The discrimination parameter for question 4 is 1.459.

The next example, Item 5, shows an item that doesn’t discriminate quite as well as Item 4. The a-value on that one is 0.53. It’s also a pretty easy item, with a b-value of -1.07. So, on this one, most students are likely to get it correct, unless they’re more than one standard deviation below the mean of the ability scale.

It’s not necessarily the case that an easy item automatically has poor discrimination. The final example, Item 8, is an easy item that discriminates very well. Although students at most ability levels are likely to select the correct answer, there is still a dramatic increase in likelihood of answering correctly within a relatively narrow range of the ability scale.

The guessing parameter (the c-value) is the probability of getting the item correct by just guessing. It defines the lower limit of the item characteristic curve. For a multiple choice item with four answer choices, the guessing parameter should be around 0.25 or, preferably, a bit lower.

A Closer Look at the Benchmark Results Page

When looking at the Benchmark Results page, which is the page that teachers generally go to when analyzing assessment results, you are encouraged to focus on the student’s Developmental Level (DL) or Scale Score rather than the student’s raw score.

This is because the DL score, and the student’s associated mastery level, provides a much better picture of a student’s ability. A raw score will simply tell a user what a student got right and what a student got wrong. The DL score factors in, not only what items a student got right and wrong, but also the difficulty and discrimination value (how well does this item discriminate between students of different ability levels) of the items.

When a student takes an ATI Benchmark Assessment, they will earn a particular DL score. A DL score is a score that takes the relative difficulty of the assessment into consideration. DL scores on two assessments can be compared in a meaningful way whereas raw scores cannot. For example, 70% correct on a very easy assessment does not mean the same thing as 70% correct on a very difficult assessment. However, a DL (scale) score of 954 on one assessment means the same thing as a DL score of 954 on another assessment, as long as the two assessments have been placed on the same scale. The particular DL score a student earns places them in a particular mastery category. Each state has its own mastery categories (Below the Standard/Unsatisfactory, Approaches the Standard/Partially Proficient, Meets the Standard/Proficient, Exceeds the Standard/Advanced) but they are similar in nature. For example:

Cut scores are then established to determine which Mastery category a student will be placed based on his or her performance on the assessment. The cut scores that define the mastery categories are established for Bencmark 1 using equipercentile equating to align students’ scores to their scores on last year’s state assessment results. The cut scores on all other assessments administered in a school year are established based on the amount of growth in terms of scale scores that is expected from one benchmark to another.

What does this mean for the user? Users can rely on DL scores and their associated mastery catagories, to help identify students to target for interventions, even after one benchmark assessment is given. A teacher’s goal should be to see an increse in DL scores (and mastery catagories) as the year progresses and students learn more of the standards. To acheive this goal, teachers will want to analyze the Class Development Profile Grid, Item Analysis and Risk-Level Report to identified standards on which to focus their re-teaching instruction. Click here to learn more about how these reports can assist with interventions.

Lesson Plan Documentation: A Great Use of Instructional Dialogs

Galileo Instructional Dialogs can serve as a unique recordkeeping tool for teacher documentation of which standards are covered during each teaching day as well as very detailed notes about the actual lessons or activities used in the classroom.

Start with a template Instructional Dialog with just the title on each slide. This Instructional Dialog may be created at the beginning of the year using whatever lesson plan format the teacher or the district advocates. Copy the template dialog.

Once the template is copied, fill in the blanks.

The lesson plan is preserved electronically and attached to a standard. The teacher only needs to view the resource in order to see the lesson plan.

A completed lesson plan in a different electronic format may also be attached as a resource.

Generate a short quiz at the end of the Instructional Dialog. This can check the effectiveness of the lesson.

Finally, schedule the dialog which will allow the Instructional Dialog/lesson plan to show up on Galileo’s class calendar for an effortless view of what has been accomplished in the classroom.

Intervention Alert Report

The Intervention Alert Report is quickly becoming one of the more popular reports in Galileo. I have recently visited a number of districts and received positive feedback from many teachers. This report lists all of the learning standards on a given assessment and displays the percentage of students who have demonstrated mastery of the learning standards. Teachers can quickly identify which standards are not mastered since they are highlighted in red. This allows the users to easily identify standards to address during interventions. ATI has recently enhanced this report by listing the performance band (i.e., meets standard, approaches standard…) within each cell. It also gives the teacher information on how students performed on each standard at the school and the district levels. In addition to teachers, principals and district administrators can run this report at a school or district level. This is an actionable report that allows the user to schedule Assignments or use Quiz Builder, or drill-down through the data to view individual Student Results. It is available in the Reports area for district-, center-, and class-level users. It may also be accessed from the class dashboard page.

Thoughts on Race to the Top: Collaboration and local control

By now, grant applications for the federal Race to the Top (RTT) program are in preparation across the country. There are a lot of state department of education employees who are likely to have a hardworking holiday season as applications for the first wave of funding are due in January. With the 4.3 billion dollars that has been allocated, states will have significant resources that can be brought to bear to make the kind of sweeping and dramatic updates and changes to school systems that are called for as part of this federal initiative.

The guidelines presented to the states to prepare their RTT applications contained two clear themes. On the one hand, state education initiatives are supposed to preserve the “flexibility and autonomy” of LEAs. There is clear recognition of the need for districts to be supported in their efforts to make decisions about curriculum assessments and other issues that are in the best interests of their staff and students. In addition to the call for local control there is also a clear mandate for collaboration. States are encouraged to adopt common standards and collaborate to produce common assessments. One of the questions that state governments face in their preparation of their proposals is how best to balance these two, at times seemingly contradictory, objectives.

One of the ways that collaboration could be facilitated, while at the same time preserving the decision making power of the districts is for the state to make available to districts an item bank in which all of the items are on the same scale. These items could be used on both district interim assessments and the state test. What would this mean for the sake of districts you ask? Such an item bank would afford the opportunity to make sure that the assessments composed of these items, either entirely or in part, can be placed on the same scale. This means that the scores are directly comparable. The 500 on the math test given in the middle of the year could be compared directly to the 550 on the state test at the end of the year. Put another way, in this case, the statement could be made that the ability level required to achieve 550 on the benchmark test is higher than the ability level required to achieve 500 on the state test. Without tests that are on a common scale, such comparisons are not possible. The 550 might represent higher ability than the 500 and then again it might not. Having a common measuring stick could go a long way towards facilitating collaborative work.

A common item bank could also greatly assist smaller districts in their efforts to implement valid and reliable interim assessments for the purpose of informing instruction. The utility of assessment results is greatly aided to the extent that they reflect the kids that actually attend the district schools and the instructional priorities of the district. Research has consistently shown that the items behave differently when students change or when instruction changes. Ongoing analysis of test behavior is critical to making sure results are reliable and valid for the kids with whom it will be used. Such analysis is difficult with districts that have only small numbers of kids. Having a common item bank from which to draw could make it much easier to do the necessary analyses to back up an assessment initiative with a small district.

Achieving these beneficial results does not require that both the state and district tests are comprised entirely of items from this bank. The only requirement is that they both contain a sample of items from the bank. This would leave the district free to include local items reflecting content which may not be of interest to other districts in the state. Easy communication and collaboration need not be sacrificed in order to continue to achieve the flexibility and autonomy that allows districts to make sure that instructional improvement systems meet their priorities.

Anyhow, I had best sign off at this point. This post is already rather lengthy. We would, as always, be interested in hearing the feedback of others about these ideas or about other topics.