Attending Conferences

Attending conferences can be overwhelming. When you visit a booth, be sure to take home any pertinent literature or samples, etc. that vendor offers. When you get home, all the amazing products you’ve seen over the last day or so might blur together. You may want to jot down a note or two while visiting the booth/vendor so you know what each product offers. This can be a great way to remember later on.

Ah, “Those Lazy, Crazy, Hazy, Days of Summer” ... No data needed for now, but do you want it and will you have it in the fall?

The summer of 2009 is now in full swing with students, parents, teachers, and administrators enjoying a well-earned vacation from a very exciting, busy, and oftentimes challenging school year.

During this past year, and as a result of the American Recovery and Reinvestment Act (ARRA) of 2009, we have seen a remarkable array of new policies and reforms occurring in K-12 education, backed by an unprecedented re-investment in education by the federal government. While this might not be among the “hot topics” of discussion around the summer lemonade stand or by the poolside, it has certainly been on most people’s radar and in the news almost daily these past few months.

Suffice to say, the immediate impact of the Act is akin to creating a new story-line and a new debate among educators, researchers, policy-makers, and the public about the future of our nation’s educational system. Certainly not the stuff of summertime fun but undoubtedly, a topic that will pick up momentum again as the 2009-2010 school year approaches. If however, you find yourself yearning for a summer thirst-quencher on this topic then consider the following.

One of the major goals of the ARRA is to improve our nation’s education system and enhance student learning through the increased use of technology innovations and actionable data to help inform educational decision-making.

In order to accomplish this goal, two key types of data are needed. The first is data on student mastery of state standards obtained not only through end-of-year statewide tests, but more importantly, continuous data on student learning and mastery of standards that can be used in “real-time” to inform instruction and intervention decision-making. Consequently, technology innovations represented in the new generation of online educational management systems must have the capacity to provide local school districts with an integrated array of locally customized assessment tools aligned with the district’s overall educational plan (e.g., pacing guide) for the year. To the extent that student learning and progress can be captured in this fashion, the second type of data – data that documents the impact of interventions on student learning and standards mastery – becomes a reality.

The paramount and practical importance of local school district empowerment in implementing an online educational management system that provides data in this way should not be underestimated. Rapid access to reliable data on student learning - where the student is and what needs to be planned for next for progress to continue - is a key element for planning effective learning opportunities and helping students meet the educational challenges of the 21st century.

It is perhaps stating the obvious to say that the importance of the data lies not in the need to gather and report it, or to simply answer a question, but rather so that positive action in the best interests of students can occur in a timely and purposeful fashion.

As stated by Pennsylvania Gov. Ed Rendell, chairman of the National Governors Association, at the March 2009 forum, Leveraging the Power of Data to Improve Education, “…even the best data collection system is worthless if it does not change what goes on in the classroom."

A few of my friends have wondered about this issue. Why collect data, or for that matter use all this sophisticated technology if it does not really change what is going on? Then there are my other friends who point out, that access to the technology and to the data is not supposed to change things, but rather, make change possible. It’s an interesting debate and I can see a valid argument on both sides.

What do you think? Let us know and in the meantime, enjoy those “Lazy, Crazy, Hazy, Days of Summer.”

Counting the Mountains and the Lakes: Quantile Regression and NCLB

I am sure that the title of this post sounds a bit odd. Let me explain.....

A statistics book that I was recently reading starts out in the preface with a quote by Francis Galton in which he teased some of his colleagues for always falling back on averages to the exclusion of other analytic approaches thereby missing much of what could be discovered. Galton chided that they were much the same as a resident of “flat English counties, whose retrospect of Switzerland was that, if its mountains could be thrown into its lakes, two nuisances would be got rid of at once (Natural Inheritance).” The author of this statistics book (which can be seen here) then proceeds to describe the use of a statistical technique called quantile regression which provides a means to examine some of the “mountains and lakes” that might be found in data by those willing to look beyond averages. I’ll get back to this procedure in a bit. Don’t worry… I won’t bore you with its inner workings. One of my colleagues here is rather fond of pointing out that statistics isn’t a topic for polite conversation. Rather, I will try and talk a bit about what sort of real life questions quantile regression is being used to answer. Some of these real life questions concern new ways to look at student growth within the context of NCLB.

We are all very aware of the data that are gathered as part of NCLB and the types of questions that these data are used to address. The fundamental question has been: Are children meeting the standard? If they aren’t, then schools and districts are subject to penalties. Over the course of the years since NCLB was implemented, there have been a growing number of educators and members of the research community arguing that this approach isn’t adequately attentive to issues that are essential to the ultimate success of efforts to raise student achievement. The fundamental issue that has not yet been adequately addressed is student growth. Looking only at whether students have met the standard doesn’t make a distinction between a school in which students started at a low level and are making rapid progress towards ultimately mastering state standards from one in which the students started at a similar level but weren’t progressing. To paraphrase Galton, failing to recognize this particular mountain range could mean that opportunities are missed to support educational intervention efforts that are proving successful. Lack of sensitivity to student growth also has potential implications for high achieving students . Without being attentive to student growth, there is no way to highlight the differences between high achieving students who are growing and those that are not.

In order to get a more complete view, several states have implemented growth models for determining accountability under NCLB. Thus far 15 states make use of such a model for determining AYP. The growth model that is used in Colorado is particularly intriguing because of the fashion in which it applies quantile regression to the question of growth. The Colorado approach allows for a student’s growth to be compared against his or her academic peers. Students can be evaluated to determine if they are making more or less progress than students who are essentially starting from the same place. High achieving students aren’t lumped together with students who are behind. This approach focuses attention both on each student’s current level of skill and on the progress that they are making. Because of this more complete view, the Colorado Department of Education (CDE) is able to give schools credit for moving students forward, even if they haven’t yet got to the point where they will ultimately pass the test at the end of the year. This approach also more clearly identifies student progress at the upper end.

The information that looking at accountability in this fashion can provide is obviously more nuanced and complete than the more basic approaches that have been employed. The question that must be addressed is whether the approach is shining the light on all the mountains and lakes that should ultimately be considered. We believe that tracking growth using quantile regression can provide information that is useful for guiding instruction and that cannot be easily obtained in other ways. For example, quantile regression analysis can be of assistance in determining growth rates for students starting at different ability levels. Information of this kind can be very useful for guiding instruction in ways that elevate student achievement and that are maximally beneficial for all students. This fall, ATI will be developing new reporting tools providing growth information derived from quantile regression. We would be interested in hearing from you regarding this initiative. We are particularly interested in hearing from those of you working in states where such an approach has been put in place. How has it worked in practice? What sort of issues have arisen?

On the Assessment of Writing

One of the topics being considered in many states is how to best assess students' writing skills. The implementation of multiple-choice items to assess the writing ability of students has become more popular in recent years. Among states where Galileo K-12 Online is currently used, California and Massachusetts both use multiple-choice items to assess some aspects of writing. Arizona is reportedly adding multiple-choice writing to the AIMS in the next round of pilot testing, and we expect to see those items supporting the revised Arizona English and Language Arts standards which will be adopted in 2010-2011.

It is not surprising that multiple-choice holds a certain appeal for those wishing to assess writing. Multiple-choice items take less time away from instruction, can be scored using automated procedures such as those available to users of Galileo K-12 Online, and are scored consistently due to the use of a single correct answer instead of relying on evaluators to score to a rubric. These advantages make multiple-choice a compelling option, but there are other considerations that limit the usefulness and effectiveness of multiple-choice items in the assessment of writing. The use of multiple-choice to assess writing is an attractive but limited approach. Thomas M. Haladyna explains the limits of using multiple-choice to assess writing in Developing and Validating Multiple-Choice Items:

The most direct measure would be a performance-based writing prompt. MC items might measure knowledge of writing or knowledge of writing skills, but they would not provide a direct measure (p.11).

Therefore a crucial concern is the logical connection between item formats and desired interpretations. For instance, an MC test of writing skills would have low fidelity to actual writing. A writing sample would have much higher fidelity (p.12).

To assess writing, it is necessary to apply a standardized rubric and a writing prompt that allows students to express their responses in a manner that represents accurately their ability to compose, convey and communicate in a way that fulfills the designated purpose of a text and that utilizes appropriate information they possess relevant to the topic.

While multiple-choice reading items addressing an analysis standard may not require the student to compose a full analytical expression, they do require the student to utilize the same analytical processes to identify the correct analysis from the distractors provided. However, the ability to identify the best compositional example does not reflect accurately the skills and abilities inherent in good writing as the ability to recognize persuasive, informative or expressive quality does not indicate the ability of the student to create the same level of written content.

Galileo provides content to allow for writing assessments using prompts for the most authentic measure of student writing, while also covering writing knowledge and skills in multiple-choice items that help to establish data for basic skills measurement and test reliability in predicting standardized test performance.

Text Referenced

Haladyna, T.M. (2004). Developing and Validating Multiple-Choice Test Items (3rd ed.). Mahwah, N.J.: Lawrence Erlbaum Associates.

Care must be taken when administering benchmark assessments to subsets of students or to students from multiple grade levels

Galileo K-12 Online benchmark assessments serve two functions simultaneously. One is to provide teachers with timely feedback regarding which standards their students have and have not mastered. The other is to forecast the students’ likely performance on the high stakes statewide assessment such as AIMS in Arizona or MCAS in Massachusetts. Both of these functions are equally important, and in most cases both goals are achieved in harmony by the single benchmark assessment. However, there are some cases where the two goals are in conflict. In today’s post, I want to alert district administrators to a potential problem and to give them a way to avoid it when planning benchmark assessments.

In the typical scenario, a benchmark assessment is given to all students in the district in a given grade level. For example, all fifth-graders in the district might take a fifth-grade math benchmark assessment. It is expected that all of these students will also take the fifth-grade math high-stakes statewide assessment. This is important because the benchmark assessment must be aligned to the statewide assessment in order to generate cut scores for performance levels and to forecast student performance on the statewide assessment. If the same set of students is expected to take both the benchmark assessment and the statewide assessment, then the comparison between the two assessments is essentially a comparison of apples to apples, and all is well. The cut scores that are calculated for the benchmark assessment should provide accurate forecasts of student performance on the statewide assessment and, in fact, the accuracy rate for Galileo K-12 Online benchmark assessments is quite high (see the Galileo K-12 Online Technical Manual.)

There are cases, however, where the set of students taking a benchmark assessment is not the same as the set that will be taking the statewide assessment. In these cases, the calculation of accurate cut scores for benchmark assessments becomes more complicated. A common scenario is one in which advanced 8th-graders are taking a high school algebra course and, quite reasonably, they take the high school math benchmark assessments instead of the 8th grade math benchmark assessments. This makes perfect sense for the first goal of benchmark assessments: providing feedback to teachers regarding student mastery of state standards. It does, however, create problems for the goal of forecasting student performance on the statewide assessment. In most cases these students will be taking the 8th grade statewide assessment, and not the high school statewide assessment, and so the comparison when calculating cut scores becomes one of apples to oranges.

In order to calculate accurate cut scores for the high school math benchmark assessment in the above scenario, the scores from the 8th grade students must be removed from the data set, so that the set of students on the benchmark assessment will be the same as the set of students who will be taking the high school statewide assessment. Additionally, care must be taken when calculating the cut scores for the 8th grade math benchmark assessment. This is because a specific region of the student distribution, the advanced students, will not be present in the distribution of scores for the 8th grade benchmark. If no adjustment is made to account for the absence of the advanced students, then the cut scores that are calculated will be too low, and too many students will be classified as being likely to pass the statewide assessment. This, of course, will result in rude surprises when the statewide assessment results come in.

The take-home message, then, is to be sure to be clear about who will be taking benchmark assessments when you are planning them. Steps can be taken in cases such as the one described here to make sure that the cut scores on benchmark assessments are accurate, but only if ATI knows about the unusual circumstances in advance. If you are designing benchmark assessments in Galileo K-12 Online and there will be any out-of-grade testing, or if the set of students on the benchmark assessment will not be the same as the set that is taking a particular statewide assessment, please let your Field Services or Educational Management Services representative know right away. Forearmed with as much information as possible, ATI can work with your district to make sure that the benchmark assessments provide accurate forecasts of student performance on statewide assessments as well as providing timely feedback regarding the mastery of standards to classroom teachers.

Help for Math Teachers

The purpose of this thread is to provide information and a way for math teachers to converse with each other about specific states standards both interpretations of state provided language and ideas about how to teach these standards to students.

Please comment on posts or add new posts including questions, ideas, and answers about how to teach math standards.

High School: Post #1

AZ-MCW-S3C4-PO10. Determine an effective retirement savings plan to meet personal financial goals including IRAs, ROTH accounts, and annuities.

AZ provided connection: MCWR-S5C2-09. Use mathematical models to represent and analyze personal and professional situations.

AZ provided explanation: An IRA is an “Individual Retirement Account,” and a ROTH is a specific type of IRA, with a more complex tax-advantaged structure.

I have searched for formulas or information about how to figure returns, advantages, and how to figure how much to invest in order to reach a retirement goal, but I have only found calculators not any information about formulas to mathematically figure the answer.

What materials/formulas do you plan to teach students to figure this information?

Middle School: Post #1

AZ-M06-S2C4-01. Investigate properties of vertex-edge graphs
· Hamilton paths,
· Hamilton circuits, and
· shortest route.

How do you teach students to check their answers on the vertex-edge graph items?

How do you know if you found all possible paths on a vertex-edge graph?

AZ provided explanation: A Hamilton path in a vertex-edge graph is a path that starts at some vertex in the graph and visits every other vertex of the graph exactly once. Edges along this path may be repeated. A Hamilton circuit is a Hamilton path that ends at the starting vertex. The shortest route may or may not be a Hamilton path. Depending upon the constraints of a problem, each vertex may not need to be visited.

Elementary School: Post #1

AZ-M02-S5C2-03. Select from a variety of problem-solving strategies and use one or more strategies to arrive at a solution.

What problem strategies do you think are appropriate to teacher primary students?

Which problem strategies are your student’s favorites?

Share Your Lessons With Others

Have you created a lesson that you are incredibly proud of? Do you wish there was an easier way to let your colleagues access the lesson to use with their students? With Galileo sharing is easy. In order to share your content, you will want to attach it to a Dialog. Don’t worry. You needn’t recreate your lesson in a Dialog. We recommend that you do the following:

1. Link your Dialog to state standards. Most of your colleagues will search for lessons based on standards.
2. Give your Dialog a title and add any notes that will be relevant to other users.
3. Add a description. The words you place in the description box will be searchable by other users once you share your lesson. Examples of keywords could include: emerging language learners, hand-held responders, or teacher-facilitated.
4. Attach the lesson as a resource.
5. Automatically generate a follow up quiz. This is optional and only necessary if you’d like to use Galileo’s Formative Test Reports to evaluate students’ learning of the lesson.
6. Publish your lesson.

Once your lesson is published you can share it in two ways. Once your Dialog is published you will see a Share Dialog button. Click this button to add your lesson to the community bank. Sharing your Dialog to the community bank will allow Galileo users in your district and other districts to see your Dialog when searching, and they can schedule and use it with students. If you would prefer only to share your content with colleagues in your district, that is possible as well. You will just need to provide your colleague’s access to your Dialog Library or copy your Dialog into their library. ATI will be more than happy to show you how this is done. For more information on sharing your lessons, e-mail ATI’s Professional Development staff at professionaldevelopment@ati-online.com for assistance. Or call us at 1-800-367-4762 ext. 132.