Comments on ATI Town Hall Blog: How can you tell if your intervention is working?

Thank you Christine for your response. A clarific...

2008-11-17T19:58:00.000-08:00

Thank you Christine for your response. A clarification questions plus another thought-provoker:

- Are the DL scores supposed to correspond to the expectations at the END of each grade (e.g., 800 for an average student at the end of 3rd grade)?

- I'm still interested in a simple way to think of growth in terms of DL - is there a way to translate the DL standard deviations into something more user-friendly? If a student gains 50 DL points between benchmark 1 and benchmark 2, is there an alternative to telling the parents that "Johnny gained an entire half of one standard deviation this quarter!" It would be much better to be able to say something like "Johnny has made the equivalent of four months of progress in the last two months. Since he started about six months behind, if he keeps working like this he could be caught up by the spring."

If there is a way to use the power of the psychometrics to inform the common sense understanding of parents and teachers (and students!), we need to do it.

Thank you, Life, for your insightful comments. To...

2008-11-13T14:25:00.000-08:00

Thank you, Life, for your insightful comments. To respond to your first point, yes, looking at the percentage of students who are passing would tell you whether you’re making progress. I phrased it in terms of increased rate of growth because that’s the concept behind the increased percentages, but you’re right. Just looking at the percentages is more straightforward.

Regarding the rest of your comment, the questions you ask are very intriguing and well worth discussing. Two of them require fairly extensive, technical answers, which will be posted soon. In the mean time, I will try to answer the third.

You ask, “Is there a more user-friendly way to describe the DL gains? e.g., in grade-level equivalents, weeks of instruction, etc.?” The DL scales in Galileo K-12 Online for each grade level are meant to reflect the typical age of students at that grade level. For example, third-graders tend to be about 8 years old so the DL scale for third grade is built around a mean of 800, fourth-graders tend to be about 9 years old so the DL scale for fourth grade is built around a mean of 900, and so on. We opted to use student age rather than grade level as a framework for DL scores because, if grade level were used, then kindergarten would be built around a mean of 0, and any students falling below the mean would have negative scores, which would most likely be distressing for parents and teachers alike.

With that in mind, a teacher can monitor a student’s progress with reference to his or her age group. If a third-grader earns a DL score of 750, then the teacher knows that he or she is somewhat below average for the grade. In fact, since the standard deviation at all grade levels is set to be 100, then the teacher knows that the student is one half a standard deviation below the mean for the grade. As the student progresses, he or she should move closer to a score of 800 and perhaps beyond it.

One point that’s important to clarify is that the scales for the different grade levels are not vertically scaled. What that means is that each grade-level scale is independent and DL scores cannot be compared across grade levels. A third-grader who earns a DL score of 850 is performing at one half a standard deviation above the mean for third-graders, but it does not follow that he or she is also performing at one half of a standard deviation below the mean for fourth graders. In order to vertically scale the benchmark assessments, we would have to administer a series of assessments that have a certain number of items in common between adjacent grade levels. In other words, we would have to administer a third-grade level assessment to a large sample of third-graders that contained a set of 2nd-grade items in addition to the 3rd-grade items, and those same items would have to appear on a 2nd-grade assessment. The same pattern would have to be followed up through all of the grade levels. Thus far we have not pursued vertical scaling. As you know, MCAS is currently not vertically scaled. Vertical scaling is useful for measuring progress from one year to the next. In the event that the State were to opt for vertical scaling of MCAS, it would be useful for ATI to provide vertical scaling for benchmark assessments. If the DL scale were vertically scaled across grade levels, the means would no longer land on nice, round, age-relevant DL scores like 800, 900, and so on.

I hope this addresses your question and that it helps teachers who are using Galileo K-12 Online to view student DL scores in a clearer light. Watch for responses to your other questions to be posted soon.

- Christine

This is a helpful explanation of the equipercentil...

2008-11-10T08:52:00.000-08:00

This is a helpful explanation of the equipercentile equating, especially the way that cut scores are set after the first benchmark by using "an expected growth rate that is likely to maintain the status quo." It seems to me that if this is the case, you can tell whether you are making progress by simply looking at the percentage of students who are passing. If the percentage goes up, you are making progress, right? I wasn't sure why you would need to actually look at the increase rate of the DL scores.

I would like to see more posts about the topic of measuring growth. Here at the MA Dept of Ed, our pilot of Galileo has revealed a number of questions along these lines:

- what is a simple way of setting growth targets in Galileo based on the growth targets on our state tests for an individual student, a classroom, school, or district?

- for what amount of growth can you definitively say there is a significant difference at the classroom, school, or district level? when is it valid and not valid to use growth on DL to compare performance (i.e., "added value") at these different levels?

- is there a more user-friendly way to describe the DL gains? e.g., in grade-level equivalents, weeks of instruction, etc.?

Looking forward to more discussion on these issues!