Clarifying the Role of Text Readability Scores in Benchmark and Formative Assessment

I want to take this opportunity to address some questions that have come up regarding readability metrics related to the texts appearing in Galileo K-12 Online assessments. One of the interesting aspects of readability metrics is the attempt to quantify, mathematically, the literary and informational expression found in text. These measures provide a handy guideline for determining if the structural complexity of a text is relevant to typical readers at a grade level. However, there are some limitations to the numerical categorization of text, which will be covered in this blog post.

First, it’s important to address the purpose of testing students using text-based items.

The ATI Assessment and Instructional Design team’s goal in developing the texts and items is to provide a basis for clear measurement of students’ abilities to comprehend and analyze text in a way that informs educators on each individual student’s progress toward standards mastery. To make sure that all students being assessed are challenged, the most effective strategy is to include texts and items of variable difficulty. This means that some items are achievable by all students, some by most students, and some only by students who are proficient in the expectations required to complete those tasks. Note that Common Core State Standards information recommends a broad text readability range in measuring performance on Common Core State Standards-aligned assessments.

All texts used in Galileo Online since 2002 have been analyzed using the Flesch-Kincaid Readability measure. 

The Flesch-Kincaid formula:

A Flesch-Kincaid score can be calculated manually or by utilizing the Microsoft Word Spelling and Grammar tool with the Readability option enabled to get the score. As can be seen in the formula, the number of syllables/words and words/sentences in an analyzed text are the key determiners of the grade level rating of a text.

Below is an example of the paragraph above, analyzed using Flesch-Kincaid, and then revised using an understanding of the nature of the formula.

A Flesch-Kincaid score can be calculated manually or by utilizing the Microsoft Word Spelling and Grammar tool with the Readability option enabled to get the score. As can be seen in the formula, the number of syllables/words and words/sentences in an analyzed text are the key determiners of the grade level rating of a text. Flesch-Kincaid Grade Level as written: 14.3 (post high school)

A Flesch-Kincaid score can be calculated manually. It can also be found utilizing the Microsoft Word Spelling and Grammar tool. This is done with the Readability option enabled to get the score. The formula calculates the number of syllables/words and words/sentences in an analyzed text. These are the key determiners of the grade level rating of a text. Flesch-Kincaid Grade Level with sentences simplified: 8.1 (middle school)

A Flesch-Kincaid score can be found by hand. It can also be found using the MS Word Spelling and Grammar tool. This is done with the review option set to get the score. F.K. uses the number of syllables/words and words/sentences in a text. These are the key parts of the grade level rating of a text. Flesch-Kincaid Grade Level with sentences simplified and polysyllabic words reduced: 4.0 (elementary school)

As can be seen from the three examples above, the readability number, while useful, is simply an analysis of the structure of the passage, not the appropriateness of content. Edits made the text simpler for less-accomplished readers because they are more familiar with simpler sentences and words, but did not simplify the topic. Sometimes compound sentences and polysyllabic words are unavoidable, even at earlier grades. The presence of these complex sentences and more difficult words will affect the readability statistics of the text, but may or may not affect the students’ ability to comprehend the text. For example, longer words that the students are familiar with will not make the text more difficult to read even though the readability formula will interpret them that way. Awareness of these limitations makes this quantitative measure simply a guideline to avoid overly complex sentence structures and longer, unfamiliar words.

One question that is often asked is the meaning of the decimal points. A text with a score of 4.4 is not intended to be aligned to the fourth month of fourth grade, but simply to be  structurally more complex than a text with a score of 4.2.

In addition to Flesch-Kincaid readability measures, Galileo K-12 Online incorporates The Lexile® Framework for Reading developed by MetaMetrics® into our benchmark texts. The model used in Lexile measures has some similarity to Flesch-Kincaid. More information on the formula used to compute Lexile measures can be found at the MetaMetrics Web site.  While some grade-level equivalent information is identified, MetaMetrics has made the following statement on grade equivalence:

There is no direct correspondence between a specific Lexile measure and a specific grade level. Within any classroom or grade, there will be a range of readers and a range of reading materials. For example, in a fifth-grade classroom there will be some readers who are ahead of the typical reader (about 250L above) and some readers who are behind the typical reader (about 250L below). To say that some books are "just right" for fifth graders assumes that all fifth graders are reading at the same level. The Lexile® Framework for Reading is intended to match readers with texts at whatever level the reader is reading.

We all recognize that the quantitative structural scores of text are just one aspect of readability. The other aspect is the qualitative element of the topic and ideas presented therein. ATI’s content specialists and each district’s teachers choose topics and themes that are meaningful to the students. We encourage all of our district partners to engage in the assessment review process, which allows the replacement of items and texts to better serve the needs of each district’s students. ATI develops items aligned to texts to serve a broad range of student abilities, and is glad to work with our partners to make sure that the texts presented in the final version of custom assessments have the support of the educators who will use them to inform instruction.

Further questions on readability and text appropriateness? Please comment below.

Depth of Knowledge

Depth of knowledge, or DOK, evaluates the alignment between the depth or complexity of cognitive processing required for a student to complete an assessment task and how well that assessment task aligns with the cognitive complexity expressed in the standards being assessed.

The concept of DOK, developed by Norman Webb, breaks down into four levels - explained in the table that follows.

ATI offers a number of ways to measure the achievement of complex skills aligned to Common Core State Standards using Galileo K-12 Online assessment solutions. Items involving analysis and reasoning (available now in the ATI item banks) measure student capabilities and comprehension beyond factual recall. Assessments can be integrated into the instructional process and thereby expanded to cover student performance occurring over time periods that extend beyond the limited time periods allotted for selected-response tests.

Frequently Asked Questions: ATI and Common Core State Standards

Text reference: Are state-level standards and assessments aligned? Wisconsin Center for Education Research Highlights Fall 1999: 1-3.

New Galileo Item Types

Galileo® K-12 Online item banks currently contain over 110,000 items in math, reading, science, and writing aligned to Common Core State Standards (CCSS) and Next Generation Science Standards* (NGSS) as well as to state standards used to transition to CCSS. Item types include traditional selected-response, constructed-response, and technology-enhanced items supporting transition to CCSS and NGSS and reflecting released plans for Partnership for Assessment of Readiness of College and Careers (PARCC) and Smarter Balanced Assessment Consortium (SBAC) assessments.

CCSS and NGSS differ from earlier state standards in a variety of ways that directly affect the development of items and assessments reflecting the new standards. First, the new standards promote the integration of assessment and instruction. Second, the new standards emphasize the development of higher order thinking skills reflecting high Depth of Knowledge (DOK) levels. Third, the new standards replace the heavy reliance on selected-response items characteristic of earlier assessments with a more balanced approach involving selected-response items, technology-enhanced items, and constructed-response items. Finally, the new standards have introduced changes in the assessment process (e.g., the introduction of computerized adaptive tests as the format for SBAC assessments).

ATI has implemented a plan to respond to these new requirements including the following elements: 1) developing online Instructional Dialogs supporting the integration of assessment and instruction, 2) developing new content in the form of selected-response, constructed-response, and technology-enhanced items assessing the higher order thinking skills emphasized in the new standards, 3) introducing a multi-stage Computerized Adaptive Testing option, and 4) designing professional development offerings to assist districts/charter schools to make the transition to CCSS and NGSS.

These innovations will assist districts and charter schools in transitioning instruction and assessment to align with the new standards. These innovations will also ultimately assist districts and charter schools in elevating student achievement by providing students with the opportunity to master the content addressed in the new standards as well as to practice the online test-taking skills required for success in the next generation of statewide assessments created by PARCC and SBAC.

Read more here

*Next Generation Science Standards (NGSS) is a registered trademark of Achieve. Neither Achieve nor the lead states and partners that developed the NGSS was involved in the production of, and does not endorse, this product.

Community Assessment and Item Banking Initiative Professional Development Offerings

The Community Assessment and Item Banking (CAIB) Initiative provides a way for users to develop and to share both assessment items and assessments for use in evaluating student capabilities in areas not tested on statewide assessments. As part of the initiative, ATI is offering professional development to prepare educators for developing high-quality items and is providing services related to assessment development and data analysis. Participating districts are provided access to a continually growing repository of shared, district-written, high-quality items and customized assessments in areas not currently addressed on statewide tests.

CAIB Professional Development Modules:
Professional development supporting the CAIB initiative consists of a series of three modules that guide districts through the process of developing high-quality assessment items and assessments. The first module introduces participants to the Galileo item writing and review tools and guides districts through the process of writing items for their own use. The second module takes participants through a guided review of the items they have written. Participants in this training refine their review skills and help contribute vetted items to the shared Community Item Bank. Module Three is offered online and on an ongoing basis to district staff members who have completed the first two modules and as the district wishes to refine and expand on the items available for district and community use.

To schedule a professional development session:
On-site (Modules One and Two) and webinar instruction (Module Three) is arranged through the ATI Professional Development Department. Up to 40 participants may take part in on-site sessions held at a district-provided computer lab. Sessions are booked on a first-come, first-served basis. Contact a Professional Development staff member at 1.877.442.5453 or at ProfessionalDevelopment@ati-online.com for additional information, pricing, and to reserve a date.

For more information:
View Professional Development Community Assessment and Item Baking Initiative

ATI Item Development: Expertise and Experience

ATI item banks, currently containing more than 80,000 items, are among the largest item banks aligned to standards in the nation. The items within these banks have been developed by experts in the field of assessment and instructional design – our content specialists are highly trained and often former educators themselves. We also value the input of those actively in the education field, thus, during the course of a single school year, ATI items are reviewed by hundreds of educators. The combination of internal processes and continuous review by educators means ATI produces assessments that meet local needs and are highly effective in forecasting student mastery of performance standards and in guiding instruction to enhance student achievement.

The following is a glimpse into ATI’s item and assessment construction process.

Item Specifications

The first step in writing a new item to be included in the ATI item banks is to review the standard to be assessed with the item. The standard is broken down into the component skills that make up the standard. These skills are the starting point for developing a list of item specifications that define the characteristics of the particular class of item to be written.

Item Construction

After specifications have been written, items are constructed corresponding to the specifications using the online Bank Builder tool. Bank Builder includes features that promote item quality, such as allowing for the use of item families, or groups of items, that refer to the same contextual material.

To ensure reliable and efficient measurement of standards, passage length is controlled to ensure that assessments are an appropriate length to fit within the time limits typically found in school settings. Reliability is a direct function of assessment length.

The item construction process also includes attention to important principles of item writing. For example, when writing selected-response items, incorrect alternatives must be plausible answers that represent common mistakes or misunderstandings. By developing constructed-response, interactive, and performance items an opportunity exists for more authentic assessment of student proficiencies. The nature of the standard being assessed is evaluated to find the best alignment for the task (e.g., using constructed-response to measure summarizing; using inclusive performance assessment items to measure a broader standard; using interactive items to practice skills like measuring and observing.)

Careful attention is paid to all text and images related to items to ensure that the material is at the appropriate reading level. One measure of readability used routinely is the Flesch-Kincaid Index, a formula based on the number of words per sentence and the number of syllables per word. Item text and images are also analyzed to ensure appropriate use of technical language and for sensitivity to diversity in backgrounds.

Item Review and Certification

The internal review process for item specifications, items, and item families occurs in three steps. Step one is an independent review using the online bank review tool in Galileo K-12 Online. Reviewers may accept the material being reviewed, reject it, or accept it with modifications. Following the initial review, the material is subjected to a final review. When consensus is reached between the two reviewers and the writer, the material under review is accepted for use. In the case of item review, the item is certified. Only certified items are provided by ATI for use by school districts.

The item review process continues once a certified item is made available for use in an assessment. Items placed in district-designed assessments are subjected to review by each member of the district’s review team during the assessment review process. The development of new items based on district suggestions is one source of the continuing expansion of ATI item banks.

District Suggestions for New Items

ATI welcomes suggestions for new items. Such suggestions help to ensure that assessments will meet district needs. ATI currently has one of the largest item banks in the nation containing over seventy-thousand certified items. Moreover, as mentioned above, the banks are continuously growing.

ATI has stringent guidelines for creating items. When making suggestions for new items, there are a number of considerations that a district may find useful. Suggestions for new items are likely to benefit the district most when they focus on the measurement of a specific skill or capability. If the district curriculum includes a particular capability that is not currently well represented in ATI item banks, suggesting the inclusion of additional items assessing the skill in question will be beneficial as long as the skill suggested aligns to the standard. At times, textbooks contain concepts which do not align to any state standards. In order to maintain the validity of ATI assessments, only items which align to a specific state standard may be added to ATI’s assessment bank. The more specific a request provided by a district, the more likely that ATI’s writers will be able to produce the exact items a district desires.

Suggesting stylistic changes is generally less useful. ATI does develop items specifically aligned to the various styles of statewide assessments. ATI, as well as states testing companies, have specific guidelines for measuring reading level. Many items use specific wording in order to maintain appropriate grade-level readability. For example, the phrase “past tense” will not be used on a third grade test since the word “tense” has a reading level ranking of sixth grade. It is important to ensure that items on local benchmark and formative assessments not be limited to a particular style. Only practicing items using specific wording when learning a new concept may cause challenges for students later. Even state released items change their wordings from year to year. One year the first grade word problem may include the phrase “in all” and the following year it may include the phrase “altogether”. It is important for students to practice items using both phrases. In addition, as items become closer in style to a statewide test, the danger that test outcomes will be contaminated by the phenomenon of “teaching to the test” increases. Benchmark and formative assessments are intended to measure the mastery of standards, not merely the ability to respond to items written in one particular style.