The National Call to Measure Teacher Effectiveness

On March 9th President Obama made a speech about the vision of his administration for education. The speech included calls for some controversial things. One of the hottest topics was the idea of rewarding more effective teachers with extra pay. Conversations around this topic quite rightly raise questions. What measures are fair to determine which teachers are the most effective? How do we account for the fact that teachers don’t get a randomly selected group of students? What kinds of stats are the most fair to evaluate the results?

All of these questions warrant careful consideration. In that light, a quick look at what we already know about some of the issues involved in answering these questions is in order.

The first issue that should be considered is the objective research data which speaks to whether there is a teacher effect on student achievement that can be quantified. In short, the research shows that an effect for teachers can be demonstrated and that the effect lasts beyond the time the student is in that teacher’s class. Some findings have shown that the impact of having an effective teacher can still be measurable 3-4 years later. This finding would suggest that students who are assigned to a particularly effective teacher for several years in a row will likely be far ahead of a student who hasn’t been assigned to equally effective instructors. Interestingly, the links between teacher variables such as credentialing have been at best weakly associated with achievement. This research is nicely, and thoroughly, summarized in a monograph prepared by the RAND Corporation.


While the research world has pretty consistently shown that a teacher effect on student outcomes can be measured, it gets a whole lot more complicated when you dive into the specifics. One of the first nitty gritty questions that must be considered is how exactly should a teacher effect be quantified? Many researchers have employed Value Added Modeling (VAM) to address this question. In short, VAM asks the question of how much variance in a student measure of achievement can be attributed to the teacher. Estimates of teacher impact obtained by utilizing VAM can be influenced by a number of variables including the impact of the way in which teacher variables and other possible confounds are modeled, the measure of student achievement that is selected, and the impact of missing data. The RAND monograph provides a very useful summary of the impact of some of these issues and their impact on conclusions that might be drawn.

All of these questions might lead one to the conclusion that developing a VAM based approach to measuring teacher effectiveness is too complicated to be able to pull off effectively. Such a conclusion would be misplaced. Many papers on the topic of VAM have consistently shown that the data provided by this approach can be useful in answering the types of questions that would face policy makers charged with delivering on President Obama’s call to design a system that can be used to reward teachers who are effective. While VAM is certainly useful, we would suggest that several things happen if it is to be employed for this type of work. The first is that skilled researchers who are familiar with the types of issues that impact VAM estimates should be involved in the design of the system on which policy makers make their decisions. Second, these same researchers also need to be able to engage in research to further understanding of the impact of various issues on VAM estimates.

Before I sign off for this post I want to raise a different but related issue for consideration. Ill bring it up here as an introduction for a post that will follow shortly

Implicit to the merit pay discussion is the idea that providing such incentives will ultimately elevate the level of instruction and lead to higher student achievement. It is our view that the goal of elevating student achievement should be looked at with all the tools that are at our disposal. Just like any other tool, VAM type analyses have certain strengths and notable weaknesses. One of the most notable shortcomings of VAM is that it can tell us little about what effective instruction looks like. It can’t provide any information about what the more effective teachers do that makes a difference. Other types of approaches that I will talk about in subsequent posts can nicely complement findings from VAM based work by addressing this very issue.

As always we look forward to hearing the thoughts of our readers

Different Types of Value Added Questions

Many of us who are involved in education today have heard about the value added approach to analyzing assessment data coming from a classroom. In a nutshell, this is a method of analyzing test results that attempts to answer the question of whether something that is in the classroom, typically the teacher, adds to the growth that the students make above and beyond what would otherwise be expected. The approach made its way onto the educational main stage in the state of Tennessee as the Tennessee Valued Added Assessment System (TVAAS).

This rather dry topic would likely not have been something that was well known outside the ivory towers were it not for the growing question of merit pay for educators. One of my statistics professors used to love to say that “Statistics isn’t a topic for polite conversation”. The introduction of pay into the conversation definitely casts it in a different light. In NYC, consideration is being given to utilizing a value added analyses in tenure decisions for principals. Value added models have been used for determining teacher bonus pay in Tennessee and Texas. Michelle Rhee has argued for using a value added approach to determining teacher performance in the DC school system. One might say it is all the rage, both for the size of the spotlight shining its way and the emotion that its use for this purpose has brought forth.

I will not be using this post to venture into the turbulent waters of discussing who should be getting paid based on results and who shouldn’t. I’ll leave it to others to opine on that very difficult and complicated question. My purpose here is to introduce the idea that the type of questions one asks from a value added perspective, the mindset if you will, can greatly inform instructional decision making through creative application. The thoughts that I will write about here are not intended to say that current applications of the value added type approach are wrong or misguided. I intend only to offer a different twist for everyone’s consideration.

The fundamental question in the value added mind set is whether something that has been added to the classroom positively impacts student learning above and beyond the status quo. One could easily ask this question of new instructional strategies introduced to the classroom that are intended to teach a certain skill. For instance, one might evaluate a new instructional activity designed to teach finding the lowest common denominator between two fractions. Given the limited scope of the activity, this evaluation could be conducted with a great deal of efficiency in very short time by the administration of a few test questions. This sort of evaluation will provide the sort of data that could be used immediately to guide instruction. If the activity is successful then teachers can move on to the next topic. If it is unsuccessful then a new approach may be utilized. The immediacy of the results puts one in a position of being able to make decisions informed by data without having to wait for the year or the semester to end.

Conducting short term small scale evaluations is different from the typical approach in value added analysis of being concerned about impact over a long period. The question of long term impact over time could easily be asked of collection of instructional activities or lessons. In an earlier post, Christine Burnham discusses some of the ways that impact over time could be tested.

As always, we look forward to hearing your thoughts about these issues.