New Science Offerings in Massachusetts

ATI has developed sets of science assessments aligned to the new Massachusetts Science and Technology/Engineering (STE) standards for grades 5 through 8. Each set consists of a pretest, three comprehensive assessments, and a posttest. They are all based on the Massachusetts Comprehensive Assessment System (MCAS) blueprints and have been constructed according to the guidelines released by the MA Department of Education (DOE).

If you are interested in high school science assessments, ATI offers sets of science assessments in biology, chemistry, and physics. Review the science and technology/engineering blueprints by grade.  

It is important to note that ATI pretests and the posttests can be used for instructional effectiveness purposes. ATI’s state-of-the-art statistical analyses and actionable Galileo reports provide data from the pretests and posttests to inform professional development. Additionally, the data can be used in support of both enhanced teaching and leadership skills and the elevation of student performance.

The Educational Management Services team will gladly help you get started with the MA Galileo K-12 science assessments. Contact them today. 

For more information on the 2016 MA STE Standards, the DOE has compiled a list of frequently asked questions which can be found at http://www.doe.mass.edu/stem/standards/faq.html.

Keeping it Simple with Galileo Assessments

It’s summer again, and that means many districts are in the thick of planning for next year’s assessments and maybe looking closer at all that Galileo has to offer, too. And since Galileo is constantly being refreshed, revitalized, and rejuvenated with updates, upgrades, and even new features, there’s a lot to explore, including technology enhanced (TE) items, digital curricula, and of course all the usual enhancements to learning that have made Galileo the stalwart resource that it has been for many years. Summer is a great time for teachers and administrators to get better acquainted with the various reports and test creation platforms that are so helpful to teachers who are guiding student learning. As an ATI Educational Management Services Coordinator, I often hear from school districts who are clearly not getting the most out of what Galileo can do. A great way to peruse the untapped possibilities is to meander through the help files in Galileo, where there is a wealth of useful and easy-to-find information about a myriad of relevant topics—reports, digital curriculum, scheduling, scoring, printing, the Dashboard, student records, data setup, user accounts, etc., etc.—virtually everything except how to tie your shoes, and yes, including assessment planning.  

As one who works exclusively with districts that design their own assessments using the Galileo tools, benchmark assessment planning is the focus of my time at this point in the year. Fortunately, assessment planning doesn’t require participation of a rocket scientist. My advice is to keep a couple basic principles in mind:  keep it simple, and let the system work for you. If there is not a strong reason to get fancy with your testing, then don’t! You’ll be spinning your wheels just to end up at the same place you would be if you let the ship fly on autopilot. What do I mean? Well, ATI has a series of ready-to use comprehensive assessments and a whole system built around them to provide reliable growth and achievement data that teachers can use to quickly see and target areas of student weakness. A good standard approach is a pretest, 1 to 3 mid-year benchmarks, and a posttest. And while certainly it is true that we don’t want to over-burden and discourage students with comprehensive tests that repeatedly test them on areas where they have not yet received instruction, ATI’s comprehensive pre- and posttests can be combined with quarterly or other periodic curriculum-aligned assessments designed by the district to effectively glean both the information on annual performance that administrators want to see, and the information on student progress that teachers need to support their vital and central role in the educational process. 

In Galileo’s Assessment Planner, the districts that choose to design their own tests can select the standards desired for each curriculum-aligned test and the number of items needed for each standard. We do the rest! Whenever questions come up, ATI’s trained and experienced staff is here to offer guidance and support. Keep in mind, the Test Review phase of test construction is your opportunity to tweak the particular items on a test, not a license to turn it inside out. Why? Because the drafts we deliver have been carefully balanced and designed to provide the best, most reliable data that can be provided within the confines of the particular blueprint; too many changes to the test carry the possibility of unknowingly upsetting that balance and causing that reliability to drift. So, keep it simple, enjoy your summer, and let Galileo work for you.   

Contributed by
Ben Tucker,  Educational Management System Coordinator

New Design for ATI Comprehensive Benchmark Assessments Series

ATI is offering two different assessment series for Common Core State Standards for 2014-15. One version will assess the same set of standards outlined in the Partnership for Assessment of Readiness for College and Careers (PARCC) end-of-year blueprints.  The other version will assess the same set of standards outlined in the Smarter Balanced end-of-year blueprints. Each set of assessments will consist of five tests including a pretest, three benchmark assessments, and a posttest. These tests will all assess the same set of standards. Districts may choose to use all of or any combination of these five tests to chart growth and Common Core State Standards mastery throughout 2014-15.

Instructional effectiveness (IE) versions of the pretests and posttests are also available. ATI recommends that districts and charters implementing instructional effectiveness initiatives with the goal of assessing student growth over the entire year use the instructional effectiveness pretests and posttests. If desired, school districts and charters may automatically pull the results of ATI’s Categorical Growth Analyses evaluating student growth from an IE Pretest to an IE Posttest directly into ATI’s score compiler. This enables districts and charters to easily combine student growth data, teacher observation or rating scale data, and other data required for teacher performance classification into a staff score compiler.

All ATI assessments are designed to maximize reliability and provide the most precise estimates of student ability and growth. In support of this goal, the majority of items included on ATI assessments have a successful history of performance and established Item Response Theory (IRT) item parameters (i.e., discrimination, difficulty, and guessing). In order to accurately assess students of all abilities, ATI assessments typically include items with a range of difficulties. Pretests and IE pretests represent a special case since these assessments are typically administered prior to students receiving instruction related to the assessed standards. For pretests and IE pretests, ATI intentionally selects easier items that are more appropriate for the students’ current level of performance and will provide the most accurate estimates of their current ability. In the past, easier items were sometimes drawn from prior-grade-level content; however, in 2014-15, all items, including easier items, will be drawn from current-grade-level content.

Karyn White, M.A., Educational Management Services Director

Pretests and Posttests

Pretests and posttests are among the most familiar forms of assessment in education. Moreover, interest in their use is rising as the nation becomes increasingly aware of the importance of assessing student academic growth over time. Although pretests and posttests are familiar forms of assessment, the ways in which they can be used and misused are sufficiently unfamiliar to deserve discussion. Determining how best to design and implement pretests and posttests is complex because these forms of assessment can be used effectively in many ways. Each way introduces the possibility of misuse. When misuse occurs, the potential value of the information that these assessments can provide is compromised. The purpose of this blog is to outline the uses of pretests and posttests and to factors issues related to use and misuse that can assist in preserving the considerable information value that can be realized when these forms of assessment are implemented effectively.

Uses of Pretests and Posttests

As its name implies, in education a pretest is an examination given prior to the onset of instruction. By contrast, a posttest measures proficiency following instruction. Pretests and posttests may serve a number of useful purposes. These include determining student proficiency before and after instruction, measuring student progress during a specified period of instruction, and comparing the performance of different groups of students before and after instruction.

Determining Proficiency Before or After Instruction

A pretest may be administered without a posttest to determine the initial level of proficiency attained by students prior to the beginning of instruction. Information on initial proficiency may be used to guide early instructional planning. For example, initial proficiency may indicate the capabilities that need special emphasis to promote learning during the early part of the school year. A posttest may be administered without a pretest to determine proficiency following instruction. For example, statewide assessments are typically administered toward the end of the school year to determine student proficiency for the year.

The design of pretests and posttests should be informed by the purposes that the assessments are intended to serve. For example, if the pretest is intended to identify enabling skills that the student possesses that are likely to assist in the mastery of instructional content to be covered during the current year, then the pretest should include skills taught previously that are likely to be helpful in promoting future learning during the year. Similarly, if a posttest is intended to provide a broad overview of the capabilities taught during the school year, then the assessment should cover the full range of objectives covered during that period. For instance, the posttest might be designed to cover the full range of objectives addressed in the state blueprint.

Measuring Progress

A pretest accompanied by a posttest can support the measurement of progress from the beginning of an instructional period to the end of the period. For example, teachers may use information on progress during the school year to determine whether or not proficiency is advancing rapidly enough to support the assumption that students will meet the standard on the upcoming statewide assessment. If the pretest and posttest are to be used to measure progress, then it is often useful to place pretest scores on a common scale with the posttest scores. When assessments are on a common scale, progress can be assessed with posttest items that differ from the items on the pretest. The problem of teaching to the test is effectively addressed because the item sets for the two tests are different. More specifically, it cannot be claimed that students improved because they memorized the answers to the specific questions on the pretest. Item Response Theory (IRT) provides one of a number of possible approaches that may be used to place pretests and posttests on a common scale. When IRT is used, the scaling process can be integrated into the task of estimating item parameters. Integration reduces computing time and complexity. For these reasons, ATI uses IRT to place scores from pretest, posttests, and other forms of assessment on a common scale.

Comparing Groups

A pretest may be given to support adjustments needed to make comparisons among groups with respect to subsequent instructional outcomes measured by performance on a posttest. Group comparisons may be implemented for a number of reasons. For example, group comparisons are generally required in experimental studies. In the prototypical experiment, students are assigned at random to different experimental conditions. Learning outcomes for each of the conditions are then compared. Group comparisons may also occur in instances in which there is an interest in identifying highly successful groups or groups needing additional resources. For example, group comparisons may be initiated to identify highly successful classes or schools. Group comparisons may be made to determine the extent to which instruction is effective in meeting the needs of NCLB subgroups. Finally, group comparisons involving students assigned to different teachers or administrators may be made if a district is implementing a performance-based pay initiative in which student outcomes play a role in determining staff compensation.

If a pretest and posttest are used to support comparisons among groups, a number of factors related to test design, test scheduling, and test security must be considered. Central concerns related to design involve content coverage and test difficulty. Both the pretest and the posttest should cover the content areas targeted for instruction. For example, if a particular set of objectives is covered on the pretest, then those objectives should also be addressed on the posttest. Targeted content increases the likelihood that the assessments will be sensitive to the effects of instruction. Both the pretest and the posttest should include a broad range of items varying in difficulty. Moreover, when the posttest follows the pretest by several months, the overall difficulty of the posttest generally should be higher than the difficulty of the pretest. Variation in difficulty increases the likelihood that instructional effects will be detected. For example, if both the pretest and the posttest are very easy, the likelihood of detecting effects will be reduced. In the extreme case in which all students receive a perfect score on each test, there will be no difference among the groups being compared.

When group comparisons are of interest, care should be taken to ensure that the pretest is administered at approximately the same time in all groups. Likewise the posttest should be administered at approximately the same time in all groups. Time on task affects the amount learned. When the time between assessments varies among groups, group comparisons may be spuriously affected by temporal factors.

Test security assumes special importance when group comparisons are made. Security is particularly important when comparisons involve high-stakes decisions. Security requires controlled access to tests and test items. Secure tests generally should not be accessible either before or after the time during which the assessment is scheduled. Galileo K-12 Online includes security features that restrict access to items on tests requiring high levels of security. Security imposes a number of requirements related to the handling of tests. When an assessment is administered online, the testing window should be as brief as possible. After the window is closed, students who have completed the test should not have the opportunity to log back into the testing environment and change their answers. Special provisions must be made for students who have missed the initial testing window and are taking the test during a subsequent period. When a test is administered offline, testing materials should be printed as close to the scheduled period for taking the assessment as possible. Materials available prior to the time scheduled for administration should be stored in a secure location. After testing, materials should either be stored in a secure location or destroyed.

Misuse of Pretests and Posttests

The misuse of tests generally occurs when tests are used for purposes other than those for which they are intended. This is true for pretests and posttests as well as for other kinds of assessments. As the previous discussion has shown, pretests and posttests are designed to serve a limited number of specific purposes. When these assessments are used for other purposes, there is a risk that the value of the information that they provide will be compromised. For example, if a pretest or posttest were to be used as a customized benchmark test, assessment results could be misleading. Conversely, if a customized benchmark assessment were used as a pretest or posttest, the credibility of assessment results could be compromised.

The central purpose of customized benchmark tests is to inform instruction. This purpose carries with it implications for test design and implementation that are generally not compatible with the purposes served by pretests and posttests. Benchmark assessments are interim assessments occurring during the school year following specified periods of instruction. Benchmark assessments provide a measure of what has been taught and an indication of what needs to be taught to promote further learning. Pretests and posttests are generally not well suited to serve as benchmarks because they often include constraints that limit their use in informing instruction. For instance, it is useful for teachers to analyze performance on benchmark assessment items to determine the kinds of mistakes made by students. This information is subsequently used to guide instruction. Pretests and posttests often call for high levels of security that curtail the analysis of specific items for purposes of informing instruction.

The temptation to use pretest and posttests as benchmarks may stem from the laudable motive of reducing the amount of time and resources devoted to testing students. Reductions in testing time increase the time available for instruction and reduce the costs associated with assessment. These are desirable outcomes. Unfortunately there often is a heavy cost associated with using assessments for purposes for which there are not intended. Often the cost is to compromise the validity of the assessments. When validity is compromised, results can be misleading. Pretests and posttests are valuable assessment tools. When they are appropriately designed, they can make a highly significant contribution to the success of an assessment program.