New Science Offerings in Massachusetts

ATI has developed sets of science assessments aligned to the new Massachusetts Science and Technology/Engineering (STE) standards for grades 5 through 8. Each set consists of a pretest, three comprehensive assessments, and a posttest. They are all based on the Massachusetts Comprehensive Assessment System (MCAS) blueprints and have been constructed according to the guidelines released by the MA Department of Education (DOE).

If you are interested in high school science assessments, ATI offers sets of science assessments in biology, chemistry, and physics. Review the science and technology/engineering blueprints by grade.  

It is important to note that ATI pretests and the posttests can be used for instructional effectiveness purposes. ATI’s state-of-the-art statistical analyses and actionable Galileo reports provide data from the pretests and posttests to inform professional development. Additionally, the data can be used in support of both enhanced teaching and leadership skills and the elevation of student performance.

The Educational Management Services team will gladly help you get started with the MA Galileo K-12 science assessments. Contact them today. 

For more information on the 2016 MA STE Standards, the DOE has compiled a list of frequently asked questions which can be found at http://www.doe.mass.edu/stem/standards/faq.html.

The Clear Choice: The Galileo K-12 Online Instructional Improvement and Effectiveness System

Districts and charters across the country have identified Galileo K-12 Online as their clear choice for a high-quality, standards-based, and research supported instructional improvement and effectiveness system. The system provides an array of curriculum, assessment, instructional effectiveness, and reporting tools. Use of Galileo facilitates advancements in teaching strategies, assessments, and implementation of instructional effectiveness initiatives. ATI offers educators key technology and services including:

• Standards-aligned item banks for teachers including a large array of technology enhanced (TE) item types
• Multiple types of classroom and district assessments to both support and measure student learning
• IRT-based Developmental Level (DL) scale scores that go beyond raw scores.
• Dashboards for all:
•  Teacher Dashboard linking actionable data to instruction
•  Admin Dashboard supporting effective leadership
•  Student/Parent Dashboard portal increasing engagement and learning
• Support for intervention/enrichment activities for all students
• Staff experienced in education, content development, measurement, evaluation research, software programming, and web design

Experience Galileo for yourself. Learn first-hand about Galileo K-12 Online: 

• Visit the Assessment Technology Incorporated website.
• Participate in an online demonstration.
• Check out the variety of resources available including videos, webisodes, webinars, and  Dialogs.
• Contact a friendly ATI Field Services Coordinator.

Clarifying the Role of Text Readability Scores in Benchmark and Formative Assessment

I want to take this opportunity to address some questions that have come up regarding readability metrics related to the texts appearing in Galileo K-12 Online assessments. One of the interesting aspects of readability metrics is the attempt to quantify, mathematically, the literary and informational expression found in text. These measures provide a handy guideline for determining if the structural complexity of a text is relevant to typical readers at a grade level. However, there are some limitations to the numerical categorization of text, which will be covered in this blog post.

First, it’s important to address the purpose of testing students using text-based items.

The ATI Assessment and Instructional Design team’s goal in developing the texts and items is to provide a basis for clear measurement of students’ abilities to comprehend and analyze text in a way that informs educators on each individual student’s progress toward standards mastery. To make sure that all students being assessed are challenged, the most effective strategy is to include texts and items of variable difficulty. This means that some items are achievable by all students, some by most students, and some only by students who are proficient in the expectations required to complete those tasks. Note that Common Core State Standards information recommends a broad text readability range in measuring performance on Common Core State Standards-aligned assessments.

All texts used in Galileo Online since 2002 have been analyzed using the Flesch-Kincaid Readability measure. 

The Flesch-Kincaid formula:

A Flesch-Kincaid score can be calculated manually or by utilizing the Microsoft Word Spelling and Grammar tool with the Readability option enabled to get the score. As can be seen in the formula, the number of syllables/words and words/sentences in an analyzed text are the key determiners of the grade level rating of a text.

Below is an example of the paragraph above, analyzed using Flesch-Kincaid, and then revised using an understanding of the nature of the formula.

A Flesch-Kincaid score can be calculated manually or by utilizing the Microsoft Word Spelling and Grammar tool with the Readability option enabled to get the score. As can be seen in the formula, the number of syllables/words and words/sentences in an analyzed text are the key determiners of the grade level rating of a text. Flesch-Kincaid Grade Level as written: 14.3 (post high school)

A Flesch-Kincaid score can be calculated manually. It can also be found utilizing the Microsoft Word Spelling and Grammar tool. This is done with the Readability option enabled to get the score. The formula calculates the number of syllables/words and words/sentences in an analyzed text. These are the key determiners of the grade level rating of a text. Flesch-Kincaid Grade Level with sentences simplified: 8.1 (middle school)

A Flesch-Kincaid score can be found by hand. It can also be found using the MS Word Spelling and Grammar tool. This is done with the review option set to get the score. F.K. uses the number of syllables/words and words/sentences in a text. These are the key parts of the grade level rating of a text. Flesch-Kincaid Grade Level with sentences simplified and polysyllabic words reduced: 4.0 (elementary school)

As can be seen from the three examples above, the readability number, while useful, is simply an analysis of the structure of the passage, not the appropriateness of content. Edits made the text simpler for less-accomplished readers because they are more familiar with simpler sentences and words, but did not simplify the topic. Sometimes compound sentences and polysyllabic words are unavoidable, even at earlier grades. The presence of these complex sentences and more difficult words will affect the readability statistics of the text, but may or may not affect the students’ ability to comprehend the text. For example, longer words that the students are familiar with will not make the text more difficult to read even though the readability formula will interpret them that way. Awareness of these limitations makes this quantitative measure simply a guideline to avoid overly complex sentence structures and longer, unfamiliar words.

One question that is often asked is the meaning of the decimal points. A text with a score of 4.4 is not intended to be aligned to the fourth month of fourth grade, but simply to be  structurally more complex than a text with a score of 4.2.

In addition to Flesch-Kincaid readability measures, Galileo K-12 Online incorporates The Lexile® Framework for Reading developed by MetaMetrics® into our benchmark texts. The model used in Lexile measures has some similarity to Flesch-Kincaid. More information on the formula used to compute Lexile measures can be found at the MetaMetrics Web site.  While some grade-level equivalent information is identified, MetaMetrics has made the following statement on grade equivalence:

There is no direct correspondence between a specific Lexile measure and a specific grade level. Within any classroom or grade, there will be a range of readers and a range of reading materials. For example, in a fifth-grade classroom there will be some readers who are ahead of the typical reader (about 250L above) and some readers who are behind the typical reader (about 250L below). To say that some books are "just right" for fifth graders assumes that all fifth graders are reading at the same level. The Lexile® Framework for Reading is intended to match readers with texts at whatever level the reader is reading.

We all recognize that the quantitative structural scores of text are just one aspect of readability. The other aspect is the qualitative element of the topic and ideas presented therein. ATI’s content specialists and each district’s teachers choose topics and themes that are meaningful to the students. We encourage all of our district partners to engage in the assessment review process, which allows the replacement of items and texts to better serve the needs of each district’s students. ATI develops items aligned to texts to serve a broad range of student abilities, and is glad to work with our partners to make sure that the texts presented in the final version of custom assessments have the support of the educators who will use them to inform instruction.

Further questions on readability and text appropriateness? Please comment below.

One Test, Many Uses

“Help! I’m stuck between my administrators who want to use the December benchmark as a predictive and my teachers who want to use the December benchmark as a summative semester final. What can I do?” 

One single benchmark, if created appropriately, can serve both purposes. 

How? 

Every state is required to have a list of standards students at each grade level need to master. It is the district’s responsibility to make sure that every child masters these standards by the end of the school year. In order to ensure this goal is achieved, many districts provide pacing guides to teachers in order to keep both teachers and students on pace to master all standards by the end of the year. These pacing guides identify which standards are supposed to be taught at what point during the year. The benchmark assessments are then created based on the district pacing guide. 

The ideal benchmark assessment should be between 35-50 items long and have no more than five items on one specific standard. Using the standards taught during the first half of the year can provide both a reliable predictor of students’ progress towards mastery of the state standards and a valid summative assessment for what students were taught in the classroom. The question then becomes how do teachers take the results of these benchmark assessments and translate them into grades?

The answer to this question depends on the district’s method for grading students. The easiest and most straightforward method is a standards-based grading system. This method of grading provides information as to whether a student has mastered a standard or skill or where he or she is at in developing the skill. Information on standards mastery can be obtained using the Galileo® K-12 Online Intervention Report.

Using benchmark assessments for a more traditional method for grading, (e.g., providing a letter grade for students based on percentages) may need more thought. In order to provide accurate ability estimates for students at all ranges of ability, it is important that there is a range of difficulties on the items and that even the student at the highest academic levels needs to be challenged. As a result, sometimes the raw scores do not represent the level of growth and success students have actually demonstrated. One example is an assessment where the average raw score percentage was 46 percent, yet students demonstrated an average growth of 20 points on their Developmental Level scores (DL). Teachers and districts can use Galileo’s benchmark data  when assigning traditional grades. One suggestion to assign grades is to generate a classroom Benchmark Results report.  The report provides information for each student’s risk assessment. By providing a grade (e.g., A for On Course, B for Low Risk, C for Moderate Risk, and D or F for High Risk), valid and reliable student data may be converted into a traditional grading system.

- Karyn White, M.A.
Educational Management Services Director

Text in the Common Core State Standards

A primary expectation of the Common Core State Standards (CCSS) for English Language Arts is for students to be comfortable analyzing complex texts. CCSS defines text complexity in both quantitative (e.g., Flesch-Kincaid and Lexile® measures) and qualitative (e.g., sophistication of ideas, assumption of prior knowledge) terms. The standards documents offer a guide for evaluating text rigor and complexity as well as lists of sample texts that meet the requirements (http://www.corestandards.org/ELA-Literacy).  Embedded in the CCSS are requirements for types of texts as well, including “multiple accounts of the same event,” “modern works of fiction,” “seminal U.S. texts,” and “foundational works of American literature.” ATI’s ELA writers have expanded our text offerings to support assessment of the new standards.

In elementary school, we have added challenging nonfiction texts with higher Flesch-Kincaid and Lexile® measures, such as “The Ice Woman: Louise Boyd’s Arctic Adventures,” and “Up, Up, and Away: The Story of Helium.” Our new texts include newspaper articles, letters, and first-person accounts of festivals, neighborhood gardens, and carnivals. We’ve written contrasting biographies of famous and important writers. To our rich set of folktales and cultural myths, we’ve added contemporary fiction featuring modern-day children with contemporary concerns, as well as excerpts from classic children’s books like The Secret Garden and Little Women.

High school CCSS emphasize classic American literature and historical documents. We’ve recently added poems by Emily Dickinson, as well as several short stories by Kate Chopin. Our banks now include presidential speeches, including inaugural addresses by FDR, Lincoln, and Washington, as well as excerpts from The Federalist Papers and Common Sense.

Eleanor Gallagher
English Language Arts Content Coordinator

Next Generation Science Standards

In the next few months, a consortium of state and local governments, the National Research Council (NRC), the National Science Teachers Association, and other stakeholders will finalize a set of national science standards, the Next Generation Science Standards (NGSS), which they anticipate most states will adopt. The 26 lead state partners in the development of the standards, including Arizona, California, and Massachusetts, have promised to seriously consider adoption of the common standards.

The NGSS come at a time when students in the United States ranked 17^th in science among 31 nations administering the most recent (2009) Program for International Student Assessment (PISA), and when only 18 percent of high school seniors perform at a proficient level in science. They come at a time when science education is not only essential to remaining competitive in the global economy, but vital in people’s everyday lives in which they must interpret scientific information to make important decisions about technology, the environment, and many other aspects of their lives.

The standards are being based on the NRC’s Framework for K-12 Science Education. The Framework reflects advances in science in the past 15 years, as well as the most current research regarding how students learn science. It is also internationally benchmarked, meaning it is informed by a thorough examination of the science standards of top performing countries. The Framework establishes in detail three interconnected dimensions: scientific and engineering practices (e.g., developing and using models, planning and carrying out investigations, analyzing data), cross-cutting concepts (e.g., recognizing patterns, cause and effect, matter and energy), and core content.

While the NGSS aren’t finalized yet, key facets of the standards are clear:

·         Scientific and engineering processes and cross-cutting concepts will be taught in conjunction with all core content throughout the year. Evidence suggests that active engagement in scientific and engineering processes and cross-cutting concepts helps deepen students’ understanding of core content and better reflects how science is done in the real world.

·         Emphasis will be on depth rather than breadth. In other words, instruction will focus on the essential core ideas and develop them progressively and coherently across K-12. This will help students acquire a deeper understanding that allows them to apply these core ideas to gain new knowledge.

·         Science and engineering will be integrated so that students understand the relationship between the two and how they are applicable to their everyday lives and to solving current world challenges such as developing clean energy and preventing and treating diseases.

·         Science standards will coordinate with Common Core State Standards for English language arts (ELA) and math so that science is an integrated part of students’ overall education.

ATI is prepared to help states that adopt the common standards implement them. Currently, our science libraries contain over 17,000 items, and these items assess a vast range of science and engineering practices, cross-cutting concepts, and core content. Many ATI science assessment items are constructed to engage students using textual and graphical resources. These Item Families reflect the NGSS emphasis on integration of processes, cross-cutting concepts, and core content by allowing these three dimensions to be assessed in concert. Item Families range in style from reports of scientific experiments to expository texts to diagrams. For example, an assessment may present a student with background information and results of an experiment and then ask the student a series of questions, some which assess the student’s mastery of scientific practices and cross-cutting concepts, others that assess the student’s comprehension of core content related to the experiment.  

ATI has over 200 science Instructional Dialogs that can be used for instruction before or interventions after assessment. They address a variety of content and skills, from the scientific method to natural resources to chemical bonding. A K-2 Dialog titled Animals and Their Parents exemplifies the NGSS emphasis on integration of the three dimensions, as well as the integration between science and other core subjects. The Dialog actively engages students in the processes of making observations and predictions and analyzing data (e.g., “Based on this picture of an adult cheetah, what do you think a baby cheetah looks like?”), as well as the cross-cutting concepts of recognizing patterns and identifying stability and change, as they learn about some of the common similarities and differences between young animals and their parents. The Dialog coordinates with ELA curriculum by drawing upon the electronic picture book The Mouse, the Rooster, and the Cat. At the beginning of the Dialog students are asked to identify some of the ways the mouse protagonist physically resembles its mother. As the Dialog progresses, they observe physical differences and compare the mouse and its mother in terms of behavior too (i.e. the mother helps protect the little mouse).  

Once the NGSS are finalized, ATI science content specialists will promptly map existing assessment and instructional content to the new standards and begin work developing new content to supplement and enhance it. The most recent draft of the Next Generation Science Standards are available at http://www.nextgenscience.org/next-generation-science-standards

Michelle Ross
Science Content Coordinator
Assessment Technology Incorporated

District Suggestions for New Items

ATI welcomes suggestions for new items. Such suggestions help to ensure that assessments will meet district needs. ATI currently has one of the largest item banks in the nation containing over seventy-thousand certified items. Moreover, as mentioned above, the banks are continuously growing.

ATI has stringent guidelines for creating items. When making suggestions for new items, there are a number of considerations that a district may find useful. Suggestions for new items are likely to benefit the district most when they focus on the measurement of a specific skill or capability. If the district curriculum includes a particular capability that is not currently well represented in ATI item banks, suggesting the inclusion of additional items assessing the skill in question will be beneficial as long as the skill suggested aligns to the standard. At times, textbooks contain concepts which do not align to any state standards. In order to maintain the validity of ATI assessments, only items which align to a specific state standard may be added to ATI’s assessment bank. The more specific a request provided by a district, the more likely that ATI’s writers will be able to produce the exact items a district desires.

Suggesting stylistic changes is generally less useful. ATI does develop items specifically aligned to the various styles of statewide assessments. ATI, as well as states testing companies, have specific guidelines for measuring reading level. Many items use specific wording in order to maintain appropriate grade-level readability. For example, the phrase “past tense” will not be used on a third grade test since the word “tense” has a reading level ranking of sixth grade. It is important to ensure that items on local benchmark and formative assessments not be limited to a particular style. Only practicing items using specific wording when learning a new concept may cause challenges for students later. Even state released items change their wordings from year to year. One year the first grade word problem may include the phrase “in all” and the following year it may include the phrase “altogether”. It is important for students to practice items using both phrases. In addition, as items become closer in style to a statewide test, the danger that test outcomes will be contaminated by the phenomenon of “teaching to the test” increases. Benchmark and formative assessments are intended to measure the mastery of standards, not merely the ability to respond to items written in one particular style.

Assessment Based on Instruction or the Benchmark Assessment Series?

Should students be assessed based only on instruction provided during the benchmark period or should all grade-level standards be assessed on each benchmark? This question is asked frequently by ATI clients.

ATI recommends that school districts assess students on standards targeted for instruction if:

· There is an established pacing guide. The school district has a relatively large number of students who will be taking each test. The school district has a pacing guide that includes instruction on all state-tested standards.

There are two particularly important benefits associated with the approach of aligning assessments with a pacing calendar: First, students are assessed only on material that they have been recently taught. Second, it is typically possible to include more items on each standard being assessed than would otherwise be the case. This increases the amount of information on student performance available for each standard that is assessed.

In contrast, when a school district has many different pacing guides, there are challenges associated with the attempt to align assessment with what is currently targeted for instruction. Disagreements about which standards should be assessed on which benchmarks tend to surface when teachers are not all on the same page. It is difficult to make decisions about which classrooms and pacing guides should take precedence over the others.

In addition, analyzing data and making predictions on how students will do on a high stakes tests is most effective when large numbers of students take an assessment. There is an adverse effect on the stability of item parameter estimates used in determining item difficulty, discrimination, and guessing. With many different pacing guides, the numbers of students taking each assessment decreases.

ATI offers an alternative for school districts that don’t have a widely used pacing guide in place or who would only have small number of students taking each assessment. The Benchmark Assessment Series assesses all grade level standards thereby providing Development Level information for students no matter what pacing guide their teachers are following, allows a greater number of students to take the same assessment, and will pinpoint areas where large groups of students have not mastered concepts which will be assessed on the state tests.