Access One-Click Reports with the Galileo Performance Benchmark Levels Widget

The Benchmark Performance Levels reporting widget summarizes multiple measures of student performance on district benchmark assessments (e.g., English language arts, math, and other subject areas). Accessible from the Galileo Teacher Dashboard, educators can view critical assessment results from an entire class via a colorful pie-chart and interactive displays of data. For a quick demonstration, check out the Galileo Tours Video.

The colorful pie chart provides a “Big Picture Class Summary” for each assessment including color-coded performance levels (e.g., highly proficient, partially proficient) illustrating the number and percent of students at each level. This reporting widget also provides instant access to number of students assessed, average percent correct, average DL Score, and average percent of standards mastered for the class.

At the click of a button, drill down to an interactive data display of a “Detailed Class Portrait” for each assessment including data for each student as well as comparative data for the class, school, and district. The chart also provides one-click access to the items used to assess each standard including item metadata such as DOK information. Rapid Data Search and Data Sorting capabilities make it easy to analyze the data for individual students as well as trends.

The Benchmark Performance Levels reporting widget allow users to quickly view assessment results

Need to export a report?  No problem.  Using the “Excel Export” button, educators can export Galileo reports to an Excel file making it easy to conduct further analyses or create additional graphical displays such as pivot charts and bar graphs.

With its colorful pie chart and interactive displays of data, easy-to-interpret results, and convenient exporting ability, the Benchmark Performance Levels reporting widget is a powerful tool helping to inform instruction and monitor student progress throughout the year.  

Schedule your personal demonstration today by contacting an ATI Field Services Coordinator or look out for the upcoming video on the Benchmark Performance Levels Widget video in the new Galileo Tours video series on the ATI YouTube page. 

Tips for Effective Test Reviews

ATI provides districts the opportunity to review district created assessments. This process can be an easy and a  valuable tool to ensure that assessments suit specific district and student needs.  Listed below are tips for effective test reviews.

1)      Keep the purpose of the assessment in mind.

There are numerous purposes for assessments.  The purpose of the assessment should dictate the review.  Look at an example from the kindergarten Common Core math standards.

 ·         K.OA.2 Solve addition and subtraction word problems.   

 If the purpose of the benchmark is to test instruction, the items on the assessment should match the district approved pacing guide.  If the pacing guide states that only addition will be taught during the first quarter then only items for addition should be included in the first quarter assessment. 

 On the other hand, if the purpose of the assessment is summative, the entire range of the standard should be assessed.

 

2)      Trust the data.

 ATI provides parameters estimates on items for test reviews.  The parameters provide information related to the discrimination, difficulty, and guessing  for each item. Since ATI analyzes item performance based on thousands of students,  the data can provide an objective view of an item and how it is likely to function on the district assessment.

3)      Consider the bigger picture.

Every teacher teaches concepts using favorite vocabulary and language. For this reason, reviewers approach item review expecting to see specific target words on items or wanting to see questions asked in specific formats.  Keep in mind that the target words may change as the tests and teaching methodologies vary over time.  The bigger picture for students is that they are able to perform the skill and demonstrate knowledge no matter how the item is presented on the high stakes assessments.  This is going to be especially important as the traditional state tests transition into the new common assessments created by the consortiums SMARTTER and PARCC. As vocabulary and item formats change, it will be beneficial for students to have been exposed to multiple formats for testing specific standards. Consider keeping an item on an assessment even if it contains new or different vocabulary and allowing students to gain valuable experience in test taking and at the same time expand their knowledge. Reviewers may be surprised at how adaptable students really are.

4)      Allow the students the opportunity to excel.

One trap districts fall into is creating assessments that are too easy.  If assessments are too easy, there is no way for the data to show growth from one test to the next.  In addition, if all students receive 100 percent on the assessment, the data  will not provide information about how to help students get to the next level. Avoiding this pitfall is relatively easy.  Reviewers should make sure that items having a full range of difficulty are included on the assessment. 

 

District Suggestions for New Items

ATI welcomes suggestions for new items. Such suggestions help to ensure that assessments will meet district needs. ATI currently has one of the largest item banks in the nation containing over seventy-thousand certified items. Moreover, as mentioned above, the banks are continuously growing.

ATI has stringent guidelines for creating items. When making suggestions for new items, there are a number of considerations that a district may find useful. Suggestions for new items are likely to benefit the district most when they focus on the measurement of a specific skill or capability. If the district curriculum includes a particular capability that is not currently well represented in ATI item banks, suggesting the inclusion of additional items assessing the skill in question will be beneficial as long as the skill suggested aligns to the standard. At times, textbooks contain concepts which do not align to any state standards. In order to maintain the validity of ATI assessments, only items which align to a specific state standard may be added to ATI’s assessment bank. The more specific a request provided by a district, the more likely that ATI’s writers will be able to produce the exact items a district desires.

Suggesting stylistic changes is generally less useful. ATI does develop items specifically aligned to the various styles of statewide assessments. ATI, as well as states testing companies, have specific guidelines for measuring reading level. Many items use specific wording in order to maintain appropriate grade-level readability. For example, the phrase “past tense” will not be used on a third grade test since the word “tense” has a reading level ranking of sixth grade. It is important to ensure that items on local benchmark and formative assessments not be limited to a particular style. Only practicing items using specific wording when learning a new concept may cause challenges for students later. Even state released items change their wordings from year to year. One year the first grade word problem may include the phrase “in all” and the following year it may include the phrase “altogether”. It is important for students to practice items using both phrases. In addition, as items become closer in style to a statewide test, the danger that test outcomes will be contaminated by the phenomenon of “teaching to the test” increases. Benchmark and formative assessments are intended to measure the mastery of standards, not merely the ability to respond to items written in one particular style.

Pretests and Posttests

Pretests and posttests are among the most familiar forms of assessment in education. Moreover, interest in their use is rising as the nation becomes increasingly aware of the importance of assessing student academic growth over time. Although pretests and posttests are familiar forms of assessment, the ways in which they can be used and misused are sufficiently unfamiliar to deserve discussion. Determining how best to design and implement pretests and posttests is complex because these forms of assessment can be used effectively in many ways. Each way introduces the possibility of misuse. When misuse occurs, the potential value of the information that these assessments can provide is compromised. The purpose of this blog is to outline the uses of pretests and posttests and to factors issues related to use and misuse that can assist in preserving the considerable information value that can be realized when these forms of assessment are implemented effectively.

Uses of Pretests and Posttests

As its name implies, in education a pretest is an examination given prior to the onset of instruction. By contrast, a posttest measures proficiency following instruction. Pretests and posttests may serve a number of useful purposes. These include determining student proficiency before and after instruction, measuring student progress during a specified period of instruction, and comparing the performance of different groups of students before and after instruction.

Determining Proficiency Before or After Instruction

A pretest may be administered without a posttest to determine the initial level of proficiency attained by students prior to the beginning of instruction. Information on initial proficiency may be used to guide early instructional planning. For example, initial proficiency may indicate the capabilities that need special emphasis to promote learning during the early part of the school year. A posttest may be administered without a pretest to determine proficiency following instruction. For example, statewide assessments are typically administered toward the end of the school year to determine student proficiency for the year.

The design of pretests and posttests should be informed by the purposes that the assessments are intended to serve. For example, if the pretest is intended to identify enabling skills that the student possesses that are likely to assist in the mastery of instructional content to be covered during the current year, then the pretest should include skills taught previously that are likely to be helpful in promoting future learning during the year. Similarly, if a posttest is intended to provide a broad overview of the capabilities taught during the school year, then the assessment should cover the full range of objectives covered during that period. For instance, the posttest might be designed to cover the full range of objectives addressed in the state blueprint.

Measuring Progress

A pretest accompanied by a posttest can support the measurement of progress from the beginning of an instructional period to the end of the period. For example, teachers may use information on progress during the school year to determine whether or not proficiency is advancing rapidly enough to support the assumption that students will meet the standard on the upcoming statewide assessment. If the pretest and posttest are to be used to measure progress, then it is often useful to place pretest scores on a common scale with the posttest scores. When assessments are on a common scale, progress can be assessed with posttest items that differ from the items on the pretest. The problem of teaching to the test is effectively addressed because the item sets for the two tests are different. More specifically, it cannot be claimed that students improved because they memorized the answers to the specific questions on the pretest. Item Response Theory (IRT) provides one of a number of possible approaches that may be used to place pretests and posttests on a common scale. When IRT is used, the scaling process can be integrated into the task of estimating item parameters. Integration reduces computing time and complexity. For these reasons, ATI uses IRT to place scores from pretest, posttests, and other forms of assessment on a common scale.

Comparing Groups

A pretest may be given to support adjustments needed to make comparisons among groups with respect to subsequent instructional outcomes measured by performance on a posttest. Group comparisons may be implemented for a number of reasons. For example, group comparisons are generally required in experimental studies. In the prototypical experiment, students are assigned at random to different experimental conditions. Learning outcomes for each of the conditions are then compared. Group comparisons may also occur in instances in which there is an interest in identifying highly successful groups or groups needing additional resources. For example, group comparisons may be initiated to identify highly successful classes or schools. Group comparisons may be made to determine the extent to which instruction is effective in meeting the needs of NCLB subgroups. Finally, group comparisons involving students assigned to different teachers or administrators may be made if a district is implementing a performance-based pay initiative in which student outcomes play a role in determining staff compensation.

If a pretest and posttest are used to support comparisons among groups, a number of factors related to test design, test scheduling, and test security must be considered. Central concerns related to design involve content coverage and test difficulty. Both the pretest and the posttest should cover the content areas targeted for instruction. For example, if a particular set of objectives is covered on the pretest, then those objectives should also be addressed on the posttest. Targeted content increases the likelihood that the assessments will be sensitive to the effects of instruction. Both the pretest and the posttest should include a broad range of items varying in difficulty. Moreover, when the posttest follows the pretest by several months, the overall difficulty of the posttest generally should be higher than the difficulty of the pretest. Variation in difficulty increases the likelihood that instructional effects will be detected. For example, if both the pretest and the posttest are very easy, the likelihood of detecting effects will be reduced. In the extreme case in which all students receive a perfect score on each test, there will be no difference among the groups being compared.

When group comparisons are of interest, care should be taken to ensure that the pretest is administered at approximately the same time in all groups. Likewise the posttest should be administered at approximately the same time in all groups. Time on task affects the amount learned. When the time between assessments varies among groups, group comparisons may be spuriously affected by temporal factors.

Test security assumes special importance when group comparisons are made. Security is particularly important when comparisons involve high-stakes decisions. Security requires controlled access to tests and test items. Secure tests generally should not be accessible either before or after the time during which the assessment is scheduled. Galileo K-12 Online includes security features that restrict access to items on tests requiring high levels of security. Security imposes a number of requirements related to the handling of tests. When an assessment is administered online, the testing window should be as brief as possible. After the window is closed, students who have completed the test should not have the opportunity to log back into the testing environment and change their answers. Special provisions must be made for students who have missed the initial testing window and are taking the test during a subsequent period. When a test is administered offline, testing materials should be printed as close to the scheduled period for taking the assessment as possible. Materials available prior to the time scheduled for administration should be stored in a secure location. After testing, materials should either be stored in a secure location or destroyed.

Misuse of Pretests and Posttests

The misuse of tests generally occurs when tests are used for purposes other than those for which they are intended. This is true for pretests and posttests as well as for other kinds of assessments. As the previous discussion has shown, pretests and posttests are designed to serve a limited number of specific purposes. When these assessments are used for other purposes, there is a risk that the value of the information that they provide will be compromised. For example, if a pretest or posttest were to be used as a customized benchmark test, assessment results could be misleading. Conversely, if a customized benchmark assessment were used as a pretest or posttest, the credibility of assessment results could be compromised.

The central purpose of customized benchmark tests is to inform instruction. This purpose carries with it implications for test design and implementation that are generally not compatible with the purposes served by pretests and posttests. Benchmark assessments are interim assessments occurring during the school year following specified periods of instruction. Benchmark assessments provide a measure of what has been taught and an indication of what needs to be taught to promote further learning. Pretests and posttests are generally not well suited to serve as benchmarks because they often include constraints that limit their use in informing instruction. For instance, it is useful for teachers to analyze performance on benchmark assessment items to determine the kinds of mistakes made by students. This information is subsequently used to guide instruction. Pretests and posttests often call for high levels of security that curtail the analysis of specific items for purposes of informing instruction.

The temptation to use pretest and posttests as benchmarks may stem from the laudable motive of reducing the amount of time and resources devoted to testing students. Reductions in testing time increase the time available for instruction and reduce the costs associated with assessment. These are desirable outcomes. Unfortunately there often is a heavy cost associated with using assessments for purposes for which there are not intended. Often the cost is to compromise the validity of the assessments. When validity is compromised, results can be misleading. Pretests and posttests are valuable assessment tools. When they are appropriately designed, they can make a highly significant contribution to the success of an assessment program.

Elevating Student Achievement: What Works

There is widespread agreement that in order to continue to be competitive in the global society in which we live, we must elevate student achievement significantly beyond current levels. The current national commitment to elevating student achievement is apparent in the Federal Race to the Top initiative which commits over four billion dollars to the national effort to increase student achievement.

We have certainly not reached the point where we can say that we have solved the problem of how to significantly increase student achievement. We will not be able to make that claim until the goal of increasing achievement has been realized. Yet, there is evidence regarding what works to elevate achievement. Moreover, the evidence suggests that the solution to the problem of elevating achievement can be rather straight forward.

There is evidence suggesting that one effective way to elevate student achievement is to administer a series of interim benchmark assessments aligned to academic standards and to the district curriculum and to use the results of each benchmark to re-teach the skills that students have not yet mastered. The Massachusetts Department of Education commissioned MAGI Services to conduct a study evaluating the use of ATI’s Galileo K-12 Online to elevate achievement. The study revealed that when teachers used Galileo benchmark assessments to guide instruction achievement was enhanced. This finding was not surprising to us. Nearly twenty years ago we conducted a study published in 1991 in the American Educational Research Journal that revealed findings consistent with those found in the MAGI Services study. We also note that for many years, WestEd has emphasized the importance of using interim assessments to guide instruction in its Local Accountability Professional Development Series. A number of “success stories” have also emerged that support re-teaching based on benchmark results. The remarkable success of the Vail Unified School District in Arizona is an example. Many other examples are beginning to emerge.

We believe that interim benchmark assessment can play an important role in elevating achievement in the years ahead. In this connection, we were delighted to find support for re-teaching utilizing interim benchmark assessment results and results from other assessments in the information provided in the Federal Register regarding the Race to the Top initiative. The Federal Register information on the Race to the Top program defines interim assessments, discusses the importance of establishing the reliability and validity of these assessments, and describes their use in guiding instructional decisions. A policy brief on the role of interim assessments in a comprehensive assessment system prepared jointly by the National Center for the Improvement of Educational Assessment and the Aspen Institute attests further to the growing recognition of the importance of interim benchmark assessments in guiding instructional decisions.

Given the straightforward nature of the approach and the growing body of evidence supporting its use, it would seem that jumping on the interim assessment bandwagon would not be a bad idea. We at ATI certainly agree. We have been developing interim benchmark assessments for use in guiding instruction for several years. However, deciding to use interim assessments to guide instruction is a bit like deciding to lose weight. It’s simple. Eat less, or if you want a complicated version, eat less and exercise more. Yet, losing weight is easier said than done. That is also the case with respect to using benchmark assessment to guide instruction.

Recently in a presentation to the Arizona Educational Research Organization (AERO), we initiated a discussion of the critical characteristics of interim benchmark assessments, the management technology needed to support an interim assessment initiative, the challenges associated with implementing such an initiative, and ways of addressing those challenges. You can find the presentation on our website. In the weeks and months ahead, we will be presenting success stories from a number of districts that will inform the discussion. The first of these recounts the remarkable achievements of the Laveen Elementary School District located in Laveen Arizona. This district achieved increases in the percentage of students meeting standards on the Arizona statewide test AIMS at every grade.

Stay tuned and give us your two cents. I know it’s not four billion dollars, but we still value it, and I am sure that the school districts meeting the challenge to elevate student achievement will value it too.

Assessment Based on Instruction or the Benchmark Assessment Series?

Should students be assessed based only on instruction provided during the benchmark period or should all grade-level standards be assessed on each benchmark? This question is asked frequently by ATI clients.

ATI recommends that school districts assess students on standards targeted for instruction if:

· There is an established pacing guide. The school district has a relatively large number of students who will be taking each test. The school district has a pacing guide that includes instruction on all state-tested standards.

There are two particularly important benefits associated with the approach of aligning assessments with a pacing calendar: First, students are assessed only on material that they have been recently taught. Second, it is typically possible to include more items on each standard being assessed than would otherwise be the case. This increases the amount of information on student performance available for each standard that is assessed.

In contrast, when a school district has many different pacing guides, there are challenges associated with the attempt to align assessment with what is currently targeted for instruction. Disagreements about which standards should be assessed on which benchmarks tend to surface when teachers are not all on the same page. It is difficult to make decisions about which classrooms and pacing guides should take precedence over the others.

In addition, analyzing data and making predictions on how students will do on a high stakes tests is most effective when large numbers of students take an assessment. There is an adverse effect on the stability of item parameter estimates used in determining item difficulty, discrimination, and guessing. With many different pacing guides, the numbers of students taking each assessment decreases.

ATI offers an alternative for school districts that don’t have a widely used pacing guide in place or who would only have small number of students taking each assessment. The Benchmark Assessment Series assesses all grade level standards thereby providing Development Level information for students no matter what pacing guide their teachers are following, allows a greater number of students to take the same assessment, and will pinpoint areas where large groups of students have not mastered concepts which will be assessed on the state tests.

ATI Custom Benchmark Assessments in Two Easy Steps

Time Line

It takes six to eight weeks to create a benchmark assessment. Leaving plenty of time will result in a great product. The first step is to submit assessment plans to ATI. Two weeks later, drafts will be delivered to the district for review. The reviews should then be completed and returned to ATI within two weeks. The final step is for ATI to finalize, publish, and place the assessments into the district secure library for printing, scheduling, and administration.

Step 1: Planning the Assessments

In this step, districts decide what standards will be included on each assessment. The following are a couple of points to keep in mind:

- A total of three to four assessments per year is suggested.

- All teachers need to know which standards will be assessed on each assessment.

- It is suggested to limit assessments to 35 to 50 items (seven to 20 standards).

- Only five items maximum are needed for each standard.

- Limit the number of reading texts on an assessment by putting all standards for similar genres on the same assessment. If five items are requested for one narrative, one expository, and one persuasive standard, up to 15 texts may be needed to cover the requested items. On the other hand if three standards which all relate to expository content are picked, the number of texts required to be read to answer all the questions on the assessment will be greatly reduced.

Another type of standard to watch for is the compare and contrast standards. Requesting five items, when each item needs to compare two stories, leads to at least 10 texts. Keeping standards for the same genre together on one assessment will help on these items because the texts will be able to be used to assess multiple standards.

Step 2: Reviewing the Drafts

In this step, districts review the assessments to pick the items located in the item banks which best serve district needs. The following are a couple of points to keep in mind:

- Items should range in difficulty in order to provide information concerning all the students in the group.

- Limit the number of reviewers. More reviewers make it more difficult to make a final decision. It is best to have a couple of key individual reviewers who are very familiar with the districts curriculum, teachers, and goals.

- If possible, use the items which are provided on the assessment. These items most likely have proven parameters which will help provide you with important information concerning student performance on the assessment.

Care must be taken when administering benchmark assessments to subsets of students or to students from multiple grade levels

Galileo K-12 Online benchmark assessments serve two functions simultaneously. One is to provide teachers with timely feedback regarding which standards their students have and have not mastered. The other is to forecast the students’ likely performance on the high stakes statewide assessment such as AIMS in Arizona or MCAS in Massachusetts. Both of these functions are equally important, and in most cases both goals are achieved in harmony by the single benchmark assessment. However, there are some cases where the two goals are in conflict. In today’s post, I want to alert district administrators to a potential problem and to give them a way to avoid it when planning benchmark assessments.

In the typical scenario, a benchmark assessment is given to all students in the district in a given grade level. For example, all fifth-graders in the district might take a fifth-grade math benchmark assessment. It is expected that all of these students will also take the fifth-grade math high-stakes statewide assessment. This is important because the benchmark assessment must be aligned to the statewide assessment in order to generate cut scores for performance levels and to forecast student performance on the statewide assessment. If the same set of students is expected to take both the benchmark assessment and the statewide assessment, then the comparison between the two assessments is essentially a comparison of apples to apples, and all is well. The cut scores that are calculated for the benchmark assessment should provide accurate forecasts of student performance on the statewide assessment and, in fact, the accuracy rate for Galileo K-12 Online benchmark assessments is quite high (see the Galileo K-12 Online Technical Manual.)

There are cases, however, where the set of students taking a benchmark assessment is not the same as the set that will be taking the statewide assessment. In these cases, the calculation of accurate cut scores for benchmark assessments becomes more complicated. A common scenario is one in which advanced 8th-graders are taking a high school algebra course and, quite reasonably, they take the high school math benchmark assessments instead of the 8th grade math benchmark assessments. This makes perfect sense for the first goal of benchmark assessments: providing feedback to teachers regarding student mastery of state standards. It does, however, create problems for the goal of forecasting student performance on the statewide assessment. In most cases these students will be taking the 8th grade statewide assessment, and not the high school statewide assessment, and so the comparison when calculating cut scores becomes one of apples to oranges.

In order to calculate accurate cut scores for the high school math benchmark assessment in the above scenario, the scores from the 8th grade students must be removed from the data set, so that the set of students on the benchmark assessment will be the same as the set of students who will be taking the high school statewide assessment. Additionally, care must be taken when calculating the cut scores for the 8th grade math benchmark assessment. This is because a specific region of the student distribution, the advanced students, will not be present in the distribution of scores for the 8th grade benchmark. If no adjustment is made to account for the absence of the advanced students, then the cut scores that are calculated will be too low, and too many students will be classified as being likely to pass the statewide assessment. This, of course, will result in rude surprises when the statewide assessment results come in.

The take-home message, then, is to be sure to be clear about who will be taking benchmark assessments when you are planning them. Steps can be taken in cases such as the one described here to make sure that the cut scores on benchmark assessments are accurate, but only if ATI knows about the unusual circumstances in advance. If you are designing benchmark assessments in Galileo K-12 Online and there will be any out-of-grade testing, or if the set of students on the benchmark assessment will not be the same as the set that is taking a particular statewide assessment, please let your Field Services or Educational Management Services representative know right away. Forearmed with as much information as possible, ATI can work with your district to make sure that the benchmark assessments provide accurate forecasts of student performance on statewide assessments as well as providing timely feedback regarding the mastery of standards to classroom teachers.

How Can Galileo Assist in Interventions?

As I interacted with school districts during the Intervention Forum, a number of people wanted to know exactly how Galileo could assist educators with interventions. You already know Galileo provides assessment data which is broken down by standard. It allows educators to see which students have made progress and which students still need additional instruction. While this information helps to identify which students may need additional help, the more challenging task is finding content to use in an intervention. Galileo not only helps identify and group students for you, but Galileo suggests Instructional Dialogs that can be used in your re-teaching initiatives.


The Risk Level report on the Benchmark Results page will group a class of students based on how at-risk the student is for not passing the state assessment. You will want to identify the group of students that you would like to expose to an intervention: High Risk, Moderate Risk, Low Risk or On Course students.


Once you have identified a group of students to work with, you will be presented with an intervention strategy for that group of students. Galileo will organize all of the standards tested into steps for re-teaching. Determine which instructional step you’d like to focus on. Then click on the Assignments button to see recommended Instructional Dialogs for each of the state standards that make up that step of the intervention strategy.





The Dialogs listed are links, so you can preview the lesson and see if it is something you’d like to use with the group of students. Teachers have implemented Dialogs by having students do them online, they have given students hand-held responders to use as they presented the Dialog, and teachers have simply presented the Dialog and have asked students to verbally respond. Notice as you preview a Dialog, each one has a quiz or a formative assessment attached. This quiz is meant to help teachers determine if students learned the standard during the intervention.

If you see a Dialog you’d like to use with students, just continue scrolling down the page and complete the online form to schedule the Dialog.

You are now ready to proceed with your intervention. As you can see, Galileo automatically links assessment data to instruction. Your benchmark data assesses the instruction that has occurred. You can run and analyze reports broken down by individual standards and individual students to determine what students need help with. You can then group students and assign already-made instructional Dialogs to aid in re-teaching students. And finally, to ensure students have learned the content of a re-teaching intervention, there is a follow-up quiz or formative assessment that can be administered automatically to students.

Have you had a chance to use or implement Dialogs? Tell us about the experience.