ATI item banks, currently containing more than 80,000 items, are among the largest item banks aligned to standards in the nation. The items within these banks have been developed by experts in the field of assessment and instructional design – our content specialists are highly trained and often former educators themselves. We also value the input of those actively in the education field, thus, during the course of a single school year, ATI items are reviewed by hundreds of educators. The combination of internal processes and continuous review by educators means ATI produces assessments that meet local needs and are highly effective in forecasting student mastery of performance standards and in guiding instruction to enhance student achievement.
The following is a glimpse into ATI’s item and assessment construction process.
Item Specifications
The first step in writing a new item to be included in the ATI item banks is to review the standard to be assessed with the item. The standard is broken down into the component skills that make up the standard. These skills are the starting point for developing a list of item specifications that define the characteristics of the particular class of item to be written.
Item Construction
After specifications have been written, items are constructed corresponding to the specifications using the online Bank Builder tool. Bank Builder includes features that promote item quality, such as allowing for the use of item families, or groups of items, that refer to the same contextual material.
To ensure reliable and efficient measurement of standards, passage length is controlled to ensure that assessments are an appropriate length to fit within the time limits typically found in school settings. Reliability is a direct function of assessment length.
The item construction process also includes attention to important principles of item writing. For example, when writing selected-response items, incorrect alternatives must be plausible answers that represent common mistakes or misunderstandings. By developing constructed-response, interactive, and performance items an opportunity exists for more authentic assessment of student proficiencies. The nature of the standard being assessed is evaluated to find the best alignment for the task (e.g., using constructed-response to measure summarizing; using inclusive performance assessment items to measure a broader standard; using interactive items to practice skills like measuring and observing.)
Careful attention is paid to all text and images related to items to ensure that the material is at the appropriate reading level. One measure of readability used routinely is the Flesch-Kincaid Index, a formula based on the number of words per sentence and the number of syllables per word. Item text and images are also analyzed to ensure appropriate use of technical language and for sensitivity to diversity in backgrounds.
Item Review and Certification
The internal review process for item specifications, items, and item families occurs in three steps. Step one is an independent review using the online bank review tool in Galileo K-12 Online. Reviewers may accept the material being reviewed, reject it, or accept it with modifications. Following the initial review, the material is subjected to a final review. When consensus is reached between the two reviewers and the writer, the material under review is accepted for use. In the case of item review, the item is certified. Only certified items are provided by ATI for use by school districts.
The item review process continues once a certified item is made available for use in an assessment. Items placed in district-designed assessments are subjected to review by each member of the district’s review team during the assessment review process. The development of new items based on district suggestions is one source of the continuing expansion of ATI item banks.
Wednesday, May 16, 2012
Monday, April 30, 2012
Emerging Technology for Common Core State Standards
Implementing Common Core State Standards in your district can be made easier by utilizing technology tools for testing and reporting. As the trend moves toward tailoring content to individual students, with assessment data providing educators a gauge of where students’ knowledge growth is at any given time, there is an increased need for a comprehensive assessment system.
To support Common Core State Standards implementation effectively and efficiently, a comprehensive assessment system should include a variety of standards-based assessments, each serving different but complimentary purposes. Within a Common Core State Standards comprehensive assessment system, results from a variety of standards-based assessments should be placed on a common scale – making it possible to compare results over time and create a more complete picture of standards mastery.
The next generation comprehensive assessment system within Galileo K-12 Online is capable of providing an array of Common Core State Standards-aligned assessments. These include:
To support Common Core State Standards implementation effectively and efficiently, a comprehensive assessment system should include a variety of standards-based assessments, each serving different but complimentary purposes. Within a Common Core State Standards comprehensive assessment system, results from a variety of standards-based assessments should be placed on a common scale – making it possible to compare results over time and create a more complete picture of standards mastery.
The next generation comprehensive assessment system within Galileo K-12 Online is capable of providing an array of Common Core State Standards-aligned assessments. These include:
- Pre-built and district customized interim benchmark assessments: Inform instruction and predict student risk of not meeting standards measured by statewide tests.
- Formative assessments: Provide information on student learning and the impact of intervention as a part of daily teaching and learning activities.
- Standards-aligned pretests and posttests: Measure academic progress over an extended period of time and support state legislative instructional effectiveness initiatives.
- Screening instruments: Help identify students at risk for learning problems.
- Placement tests: Inform grade-level placements and advanced course placements.
- Multi-Stage Computer Adaptive Testing (CAT): Provide efficient measures of academic proficiency and help support targeted instruction and response to intervention initiatives.
- Observational assessments: Provide measures of student capabilities as they are demonstrated in the environment in which those capabilities are used.
- Final course and summative assessments: Provide measures of student mastery of specific course content.
- Test arrays: Small item sets administered on separate occasions and combined for reporting. Ideal when it is not convenient to administer a long test on a single occasion.
- Dialogic assessments: Instruction-embedded online assessments making it possible to use assessment information to rapidly adjust instruction and to track instructional outcomes.
- ASK Technology: Allows for automated importation of assessments initially built outside Galileo. Once assessment is imported, ASK offers the capability of aligning to standards, automated scoring (selected-response items), reporting, and data aggregation.
- Colorado League of Charter Schools and Assessment Technology Incorporated free seminar, “Galileo K-12 Instructional Improvement System: Supporting Colorado Educational Initiatives with Comprehensive Assessment” May 16 at the Colorado League of Charter Schools, Conference Room, Denver, Colorado. Click here to register.
- 43rd Annual CASE Conference July 23 through 27 at the Beaver Run Resort, Breckenridge, Colorado.
- Massachusetts Association of School Superintendents Annual Executive Institute July 11 and 12 at the Mashpee High School, Mashpee, Massachusetts.
Monday, April 23, 2012
ATI Content Specialist Recognized in Illustrative Mathematics Task Writing Contest
ATI wishes to congratulate Aaron Stidham, an ATI Content Specialist in Mathematics, for having his work selected for the Illustrative Mathematics Task Writing Contest. Illustrative Mathematics is a project of Dr. Bill McCallum, the Math Team Coordinator for Common Core State Standards. The purpose of the Illustrative Math web site is described as:
“Illustrative Mathematics will provide guidance to states, assessment consortia, testing companies, and curriculum developers by illustrating the range and types of mathematical work that students will experience in a faithful implementation of the Common Core State Standards, and by publishing other tools that support implementation of the standards.”
Aaron submitted a task of comparing volumes of aquariums using cross-sections to determine if the volumes were equal. His task has been judged to be an effective illustration of the high school math standard G-GMD 2: Give an informal argument using Cavalieri's principle for the formulas for the volume of a sphere and other solid figures. Aaron's example is scheduled to be placed on the site soon, and a link will be posted here once it is published.
Aaron also has submitted an additional example illustrating F-BF3: Build new functions from existing functions, still under consideration for inclusion.
We are proud of Aaron's commitment to promoting educational quality through clear and illustrative examples that assist educators in implementing Common Core State Standards-aligned materials for the benefit of their students. Aaron and the other members of our math content team at ATI are committed to continuing ATI's outstanding service to our educational partners as we all transition to Common Core State Standards.
“Illustrative Mathematics will provide guidance to states, assessment consortia, testing companies, and curriculum developers by illustrating the range and types of mathematical work that students will experience in a faithful implementation of the Common Core State Standards, and by publishing other tools that support implementation of the standards.”
Aaron submitted a task of comparing volumes of aquariums using cross-sections to determine if the volumes were equal. His task has been judged to be an effective illustration of the high school math standard G-GMD 2: Give an informal argument using Cavalieri's principle for the formulas for the volume of a sphere and other solid figures. Aaron's example is scheduled to be placed on the site soon, and a link will be posted here once it is published.
Aaron also has submitted an additional example illustrating F-BF3: Build new functions from existing functions, still under consideration for inclusion.
We are proud of Aaron's commitment to promoting educational quality through clear and illustrative examples that assist educators in implementing Common Core State Standards-aligned materials for the benefit of their students. Aaron and the other members of our math content team at ATI are committed to continuing ATI's outstanding service to our educational partners as we all transition to Common Core State Standards.
Monday, April 16, 2012
Galileo K-12 Online Categorical Growth Summary Report Overview
How are your schools preparing to show data on expected student growth per teacher? You can now review expected growth rate data within Galileo K-12 Online by running the new Categorical Growth Summary Report. An educator can utilize this data to monitor their students’ growth between two assessments and an administrator can utilize this data for an educator’s evaluation.
How does Categorical Growth Analysis work?
The Categorical Growth Analysis (CGA) model provides an objective evaluation of the magnitude of student growth. ATI uses the CGA approach to evaluate student growth from the pretest to the posttest that can be applied even to individual classrooms, regardless of district/charter school size. This is how the approach works:
How does Categorical Growth Analysis work?
The Categorical Growth Analysis (CGA) model provides an objective evaluation of the magnitude of student growth. ATI uses the CGA approach to evaluate student growth from the pretest to the posttest that can be applied even to individual classrooms, regardless of district/charter school size. This is how the approach works:
- Cut scores for the pretest and posttest are established. The cut scores on the posttest are raised relative to those on the pretest by the amount of progress that is expected over the course of the year. The posttest cut score is based on the average expected growth across a large number of districts.
- Classify students at Time 1 (the first test/pretest) with regard to whether they demonstrated mastery of the state standards.
- At Time 2 (the second test/posttest), track whether each student's mastery classification stays the same as it was at Time 1, or changes.
- Focus on the students who changed mastery status from Time 1 to Time 2.
- The hope is that more students will have moved from non-mastery to mastery than from mastery to non-mastery.
- Conduct a chi-square analysis to determine if the number of students moving in one direction is significantly greater than the number of students moving in the other direction.
- Expected growth is not maintained. This outcome is supported in those instances in which there is a significant decrease in the relative proportion of students achieving standards mastery.
- Expected growth is maintained. This outcome is supported in those instances in which there is no significant change.
- Expected growth is exceeded. This outcome is supported in those instances in which there is a significant increase in the relative proportion of students achieving standards mastery.
Wednesday, April 11, 2012
Benchmark Assessment Options
Should students be assessed based on instruction provided during the benchmark period or should the majority of standards and/or skills assessed by the state-wide assessment be assessed on each benchmark? This question is asked frequently by ATI clients.
ATI recommends that school districts assess based on instruction if:
• all teachers in the school district agree to follow a pacing guide.
• the school district has a group of at least 350 students all able to follow the same pacing guide.
• the school district has a comprehensive pacing guide that includes the majority of standards and/or skills assessed by the state-wide assessment
Students seeing only material with which they are familiar and having the ability to include more items on each standard, thereby helping with differentiation of standard mastery levels, are benefits to assessing students based on instruction.
In contrast, when a school district has many different pacing guides, challenges with assessing based on instruction arise. Differing views about which standards should be assessed on which benchmarks tend to surface when teachers are not all on the same pacing guide. It is difficult to make decisions about which classrooms and pacing guides should take precedence over the others.
In addition, analyzing data and making predictions on how students will do on a test is most effective when 350 or more students take an assessment. Having fewer than 350 students taking an assessment may cause assessment data to be less stable
A further consideration when deciding between benchmark assessments based only on instruction and benchmarks based on all grade-level standards is the extent to which the district’s pacing guide covers the majority of state-tested standards/skills. If the district pacing guide does not cover the majority of standards/skills included in the state-wide assessment, the district may want to consider a broader assessment than one based solely on instruction. Two ways to accomplish this are given in the paragraph below. An example of the reason to consider an assessment broader than instruction-only items would be the situation in which a district does not include probability in sixth grade pacing guide. In this case, sixth grade students who do not already know probability are at an automatic disadvantage when taking the state-wide test if it includes probability items. Therefore the district may want to test mastery of probability even though it is not on the pacing guide.
ATI offers alternatives for school districts who struggle with one or more of these challenges. For example, a benchmark assessment that assesses all grade level standards provides relevant development level information for students no matter what pacing guide their teachers are following, allows a greater number of students to take the same assessment, and will pinpoint areas where groups of students have not mastered concepts which will be assessed on the state tests. It is also the case that when pacing guides do not cover the majority of the standards/skills measured on the state-wide assessment, customized instruction-based assessments can be built that include targeted standards/skills not part of the pacing guide but included in state-wide testing.
-Karyn White, M.A.
Educational Management Services Director
ATI recommends that school districts assess based on instruction if:
• all teachers in the school district agree to follow a pacing guide.
• the school district has a group of at least 350 students all able to follow the same pacing guide.
• the school district has a comprehensive pacing guide that includes the majority of standards and/or skills assessed by the state-wide assessment
Students seeing only material with which they are familiar and having the ability to include more items on each standard, thereby helping with differentiation of standard mastery levels, are benefits to assessing students based on instruction.
In contrast, when a school district has many different pacing guides, challenges with assessing based on instruction arise. Differing views about which standards should be assessed on which benchmarks tend to surface when teachers are not all on the same pacing guide. It is difficult to make decisions about which classrooms and pacing guides should take precedence over the others.
In addition, analyzing data and making predictions on how students will do on a test is most effective when 350 or more students take an assessment. Having fewer than 350 students taking an assessment may cause assessment data to be less stable
A further consideration when deciding between benchmark assessments based only on instruction and benchmarks based on all grade-level standards is the extent to which the district’s pacing guide covers the majority of state-tested standards/skills. If the district pacing guide does not cover the majority of standards/skills included in the state-wide assessment, the district may want to consider a broader assessment than one based solely on instruction. Two ways to accomplish this are given in the paragraph below. An example of the reason to consider an assessment broader than instruction-only items would be the situation in which a district does not include probability in sixth grade pacing guide. In this case, sixth grade students who do not already know probability are at an automatic disadvantage when taking the state-wide test if it includes probability items. Therefore the district may want to test mastery of probability even though it is not on the pacing guide.
ATI offers alternatives for school districts who struggle with one or more of these challenges. For example, a benchmark assessment that assesses all grade level standards provides relevant development level information for students no matter what pacing guide their teachers are following, allows a greater number of students to take the same assessment, and will pinpoint areas where groups of students have not mastered concepts which will be assessed on the state tests. It is also the case that when pacing guides do not cover the majority of the standards/skills measured on the state-wide assessment, customized instruction-based assessments can be built that include targeted standards/skills not part of the pacing guide but included in state-wide testing.
-Karyn White, M.A.
Educational Management Services Director
Monday, April 2, 2012
School Readiness Mini-Trainings at the NHSA Annual Training Conference
With national focus on school-readiness goals, Head Start programs more reasons than ever to demonstrate program positive child outcomes. Assessment Technology Incorporated will be offering mini-trainings at the National Head Start Conference Annual Training on Wednesday, April 18 and Thursday, April 19. The trainings will discuss the path to school readiness through Galileo Progress Reports with school readiness scale and Milestone Reports with program-defined readiness skills/goals.
The mini-trainings will cover a variety of topics including how to demonstrate program-level child progress using ability scores and mastery of school readiness skills with the Progress Report, and how to focus data aggregation and data analysis on program-defined subsets of school readiness skills and goals using the Milestone Reports.
The Path to School Readiness: Galileo Progress Reports with School Readiness Scale
Wednesday, April 18, and Thursday, April 19 at 10:00 a.m.-10:20 a.m.
Learn how the Galileo Pre-K Online Progress Reports with school readiness scale data provide actionable information to help pave the way in preparing children for kindergarten. See a live demonstration of Galileo Pre-K Online showing educators how they can aggregate and analyze child assessment data at multiple points during the year. The demonstration will offer examples of Child Progress Reports that provide data, including school readiness scale data, for decision-making. These reports show program-level child progress using ability scores and achievement levels, a baseline from which growth can be measured, patterns of progress that help to establish different goals for different groups of children, and progress data to create achievement goals for the next observation period. Find out more by attending this mini-training at exhibit #323.
The Path to School Readiness: Galileo Milestone Reports with Program-Defined Readiness Skills/Goals
Wednesday, April 18 and Thursday, April 19, 2012 at 2:00 p.m.-2:20 p.m.
Attend this session to learn how Head Start programs are focusing data aggregation and analysis on essential school readiness goals that support children as they transition into kindergarten. Galileo Pre-K Online Milestone Reports provide critical child assessment data on subsets of school readiness skills that are of importance to the individual learning needs of children. The Galileo Milestone Reports can be run at multiple points during the year and help educators plan, assess, and demonstrate achievement of their school readiness goals for children. Gain insights on how to monitor plans toward attaining successful outcomes for specific groups of children and how to apply filters, such as age, when running the Galileo Milestone Reports. ATI offers planning suggestions to help educators select readiness goals to include in program action plans. Find out more by attending this mini-training at exhibit #323.
Experience Galileo Online during an online overview and see how it provides a better way to address your goal of raising student achievement. To register, visit the Assessment Technology Incorporated website (ati-online.com), call 1.877.442.5453, email GalileoInfo@ati-online.com, or visit us at the following events:
The mini-trainings will cover a variety of topics including how to demonstrate program-level child progress using ability scores and mastery of school readiness skills with the Progress Report, and how to focus data aggregation and data analysis on program-defined subsets of school readiness skills and goals using the Milestone Reports.
The Path to School Readiness: Galileo Progress Reports with School Readiness Scale
Wednesday, April 18, and Thursday, April 19 at 10:00 a.m.-10:20 a.m.
Learn how the Galileo Pre-K Online Progress Reports with school readiness scale data provide actionable information to help pave the way in preparing children for kindergarten. See a live demonstration of Galileo Pre-K Online showing educators how they can aggregate and analyze child assessment data at multiple points during the year. The demonstration will offer examples of Child Progress Reports that provide data, including school readiness scale data, for decision-making. These reports show program-level child progress using ability scores and achievement levels, a baseline from which growth can be measured, patterns of progress that help to establish different goals for different groups of children, and progress data to create achievement goals for the next observation period. Find out more by attending this mini-training at exhibit #323.
The Path to School Readiness: Galileo Milestone Reports with Program-Defined Readiness Skills/Goals
Wednesday, April 18 and Thursday, April 19, 2012 at 2:00 p.m.-2:20 p.m.
Attend this session to learn how Head Start programs are focusing data aggregation and analysis on essential school readiness goals that support children as they transition into kindergarten. Galileo Pre-K Online Milestone Reports provide critical child assessment data on subsets of school readiness skills that are of importance to the individual learning needs of children. The Galileo Milestone Reports can be run at multiple points during the year and help educators plan, assess, and demonstrate achievement of their school readiness goals for children. Gain insights on how to monitor plans toward attaining successful outcomes for specific groups of children and how to apply filters, such as age, when running the Galileo Milestone Reports. ATI offers planning suggestions to help educators select readiness goals to include in program action plans. Find out more by attending this mini-training at exhibit #323.
Experience Galileo Online during an online overview and see how it provides a better way to address your goal of raising student achievement. To register, visit the Assessment Technology Incorporated website (ati-online.com), call 1.877.442.5453, email GalileoInfo@ati-online.com, or visit us at the following events:
- National Head Start Association (NHSA) 39th Annual National Training Conference with Galileo Pre-K Mini-Trainings “The Path to School Readiness: Galileo Progress Reports with School Readiness Scale” and “The Path to School Readiness: Galileo Milestone Reports with Program-Defined Readiness Skills/Goals" April 17 through 20 at the Gaylord Opryland, Nashville, Tennessee, exhibit #323.
- Illinois Association of School Administrators (IASA) Annual Conference April 25-27 at the Crowne Plaza Hotel, Springfield, Illinois exhibit #100.
Monday, March 26, 2012
The Skinny on Instructional Effectiveness System Development and Roll-Out Activities
Have you ever hoped that a new service would be available as soon as it is has become someone’s dream? Have you ever wondered what occurs behind the scenes in preparation for rolling-out a new service? The material below is a very short explanation of the steps involved in bringing ATI’s most recent new service component to Galileo users.
Getting a new service to end users is a very complex process. ATI is currently rolling out the Instructional Effectiveness Assessment System – a new service component in the Galileo K-12 Instructional Improvement System. Would it surprise you to know that the planning for this new service component began over two years ago? As you would imagine, the first step in the process was the recognition that districts would benefit by having a reliable and valid method for measuring the effectiveness of instruction integrated within the context of student education. Once the initial recognition had occurred, ATI management began to flush out defining characteristics that would be important in such a service component. A few of the questions that had to be answered in this phase included: What data should the system contain? How should that data be obtained? How and to whom should the data be reported? What types of data analysis would be most effective in understanding and using the data? How do you add yet another evaluation process without interfering with the educational process? Periodically potential end users were asked to look at the plans as they were being developed and to provide feedback on the emerging plan’s characteristics.
Once the overall conceptualization of the service had begun to take shape, the ATI computer design staff was presented with the defining characteristics of the project. Their tasks included designing the interfaces that would be needed in the gathering and displaying of data in a user-friendly way and which were associated with each of the goals of the project. The computer designers not only had to figure effective ways to display the data gathering portion of the system but they also had to design the ways in which the data could be most effectively presented to those would be using it in data-driven decision making and communication.
As the computer designers moved toward basic design completion, they worked interactively with the ATI software engineers whose tasks included developing the technological solutions required to accomplish the goals of the plan. The IT department was brought into the process as it is their responsibility to ensure that the technology required for all services is available at all times. They evaluated the then current infrastructure to ensure its ability to handle each aspect of addition the new service component. Additionally at this point, the ATI communication staff was introduced to the project and charged first with developing a plan to inform Galileo users of the instructional effectiveness benefits and functionality of the instructional effectiveness system and then to develop supporting materials to communicate these benefits and functionality to Galileo users. The ATI professional development team was also involved at this point as it is their task to provide professional development sessions to assist the ATI field services staff to fully understand the new service and to assist users maximize all tools/services within the Galileo instructional improvement system including new ones.
Toward the later stages of this developmental process, 38 Arizona school districts participated in a pilot of the instructional effectiveness project. These districts are in the process of providing feedback part of which has already been incorporated into the system. ATI is grateful to these districts for piloting the system and for providing excellent feedback to make the system even more exciting.
And so the process goes each time a new service is prepared for the Galileo instructional improvement system. It is a complex set of operations and we love doing it. Hopefully, you enjoy having new service components as they are developed.
Getting a new service to end users is a very complex process. ATI is currently rolling out the Instructional Effectiveness Assessment System – a new service component in the Galileo K-12 Instructional Improvement System. Would it surprise you to know that the planning for this new service component began over two years ago? As you would imagine, the first step in the process was the recognition that districts would benefit by having a reliable and valid method for measuring the effectiveness of instruction integrated within the context of student education. Once the initial recognition had occurred, ATI management began to flush out defining characteristics that would be important in such a service component. A few of the questions that had to be answered in this phase included: What data should the system contain? How should that data be obtained? How and to whom should the data be reported? What types of data analysis would be most effective in understanding and using the data? How do you add yet another evaluation process without interfering with the educational process? Periodically potential end users were asked to look at the plans as they were being developed and to provide feedback on the emerging plan’s characteristics.
Once the overall conceptualization of the service had begun to take shape, the ATI computer design staff was presented with the defining characteristics of the project. Their tasks included designing the interfaces that would be needed in the gathering and displaying of data in a user-friendly way and which were associated with each of the goals of the project. The computer designers not only had to figure effective ways to display the data gathering portion of the system but they also had to design the ways in which the data could be most effectively presented to those would be using it in data-driven decision making and communication.
As the computer designers moved toward basic design completion, they worked interactively with the ATI software engineers whose tasks included developing the technological solutions required to accomplish the goals of the plan. The IT department was brought into the process as it is their responsibility to ensure that the technology required for all services is available at all times. They evaluated the then current infrastructure to ensure its ability to handle each aspect of addition the new service component. Additionally at this point, the ATI communication staff was introduced to the project and charged first with developing a plan to inform Galileo users of the instructional effectiveness benefits and functionality of the instructional effectiveness system and then to develop supporting materials to communicate these benefits and functionality to Galileo users. The ATI professional development team was also involved at this point as it is their task to provide professional development sessions to assist the ATI field services staff to fully understand the new service and to assist users maximize all tools/services within the Galileo instructional improvement system including new ones.
Toward the later stages of this developmental process, 38 Arizona school districts participated in a pilot of the instructional effectiveness project. These districts are in the process of providing feedback part of which has already been incorporated into the system. ATI is grateful to these districts for piloting the system and for providing excellent feedback to make the system even more exciting.
And so the process goes each time a new service is prepared for the Galileo instructional improvement system. It is a complex set of operations and we love doing it. Hopefully, you enjoy having new service components as they are developed.
Subscribe to:
Posts (Atom)
