Counting the Mountains and the Lakes: Quantile Regression and NCLB

I am sure that the title of this post sounds a bit odd. Let me explain.....

A statistics book that I was recently reading starts out in the preface with a quote by Francis Galton in which he teased some of his colleagues for always falling back on averages to the exclusion of other analytic approaches thereby missing much of what could be discovered. Galton chided that they were much the same as a resident of “flat English counties, whose retrospect of Switzerland was that, if its mountains could be thrown into its lakes, two nuisances would be got rid of at once (Natural Inheritance).” The author of this statistics book (which can be seen here) then proceeds to describe the use of a statistical technique called quantile regression which provides a means to examine some of the “mountains and lakes” that might be found in data by those willing to look beyond averages. I’ll get back to this procedure in a bit. Don’t worry… I won’t bore you with its inner workings. One of my colleagues here is rather fond of pointing out that statistics isn’t a topic for polite conversation. Rather, I will try and talk a bit about what sort of real life questions quantile regression is being used to answer. Some of these real life questions concern new ways to look at student growth within the context of NCLB.

We are all very aware of the data that are gathered as part of NCLB and the types of questions that these data are used to address. The fundamental question has been: Are children meeting the standard? If they aren’t, then schools and districts are subject to penalties. Over the course of the years since NCLB was implemented, there have been a growing number of educators and members of the research community arguing that this approach isn’t adequately attentive to issues that are essential to the ultimate success of efforts to raise student achievement. The fundamental issue that has not yet been adequately addressed is student growth. Looking only at whether students have met the standard doesn’t make a distinction between a school in which students started at a low level and are making rapid progress towards ultimately mastering state standards from one in which the students started at a similar level but weren’t progressing. To paraphrase Galton, failing to recognize this particular mountain range could mean that opportunities are missed to support educational intervention efforts that are proving successful. Lack of sensitivity to student growth also has potential implications for high achieving students . Without being attentive to student growth, there is no way to highlight the differences between high achieving students who are growing and those that are not.

In order to get a more complete view, several states have implemented growth models for determining accountability under NCLB. Thus far 15 states make use of such a model for determining AYP. The growth model that is used in Colorado is particularly intriguing because of the fashion in which it applies quantile regression to the question of growth. The Colorado approach allows for a student’s growth to be compared against his or her academic peers. Students can be evaluated to determine if they are making more or less progress than students who are essentially starting from the same place. High achieving students aren’t lumped together with students who are behind. This approach focuses attention both on each student’s current level of skill and on the progress that they are making. Because of this more complete view, the Colorado Department of Education (CDE) is able to give schools credit for moving students forward, even if they haven’t yet got to the point where they will ultimately pass the test at the end of the year. This approach also more clearly identifies student progress at the upper end.

The information that looking at accountability in this fashion can provide is obviously more nuanced and complete than the more basic approaches that have been employed. The question that must be addressed is whether the approach is shining the light on all the mountains and lakes that should ultimately be considered. We believe that tracking growth using quantile regression can provide information that is useful for guiding instruction and that cannot be easily obtained in other ways. For example, quantile regression analysis can be of assistance in determining growth rates for students starting at different ability levels. Information of this kind can be very useful for guiding instruction in ways that elevate student achievement and that are maximally beneficial for all students. This fall, ATI will be developing new reporting tools providing growth information derived from quantile regression. We would be interested in hearing from you regarding this initiative. We are particularly interested in hearing from those of you working in states where such an approach has been put in place. How has it worked in practice? What sort of issues have arisen?