Is Quantifying Teacher Performance Akin to Flipping A Coin?

Last week, on the way home from school, I tuned into a story on the radio titled: “Seattle Releases First Teacher Ratings Based on Student Performance.” Data is great, but if you paid attention to the elections a few weeks ago, there were two kinds of math going on. Nate Silver’s Five Thirty Eight blog predicted 50/50 states. Karl Rove’s analysis of the data had him flummoxed. The difference was that Rove was emotionally attached, was eager to win, and for some reason his analysis of the same polls was way off. Alternatively, Silver simply plugged numbers into complex algorithms.

Mathematicians have noted that test scores and teacher performance don’t necessarily have a strong correlation, yet an incredible weight and cost is attributed to these standardized tests. Math professor Johh Ewing says, “You might as well look at all the teachers and flip a coin and those that get heads, say, are good, and those that get tails are bad, and it’s not much different from using one year of growth to measure teachers,”

Ewings paper, “Mathematical Intimidation: Driven by Data,” Looks at the potential pitfall of trying to create Value Added Measures to teacher evaluation.

Like the election examples earlier, we often attach a lot of emotion to the data creating a lot of noise. This noise had the potential to lead to bias. When a teacher says, “But I’ve done this for 20 years. I know this works,” it is evident that experience plays an important role. But is there bias involved. During those 20 years, did that teacher ever once control the experiment by not utilizing a particular skill? If so was the result the same, better, worse. Without trying to control for various things, how does one really know if what you do works. Is it just a feeling or is it based on empirical data.

Finally, there are so many things that make a good teacher: relationships with students, high expectations, integrity, care, leadership, collaboration, etc. Yet all of these traits can’t be tested for.

Standardized test scores are a reality and here to stay. As long as graduate schools use test scores as a tool to help with admissions, and undergraduate schools do the same, high schools and middle schools won’t have much of a choice. Elementary schools just follow.

There’s a dark side to this. Children as early as Pre-K are getting tutored in test preparation. Like the qualities of teachers, students have many amazing strengths and skills. However just because they struggle with test taking, potential doors my be closed without even giving the child a chance to show the brilliance that lies within.

And what about those 21st Century Skills – Critical Thinking, Communication, Collaboration, Creativity, etc. Will teachers drop integrating teaching these skills in order to meet the demands of the test scores? I hope not.

What I’d Like to Ask Bill Gates Next Week

Next week, people from many places associated with independent schools will be in town for the National Independent Schools Annual Conference here in Seattle. I’m excited about this week for many reasons and hope to write about them in the coming days.

One of the things I’m interested in is what the featured keynote speaker, Bill Gates, has to say. I won’t be able to hear him speak directly on Thursday as I’ll be teaching. I will, however, be able to follow his address through many various channels.

I read his opinion piece in the NYTimes on Friday about his thoughts on New York making teacher performance assessments public. I agree with him on many points. One of these is that making teacher evaluation assessments publicly available isn’t going to do anything to help improve teaching. I also agree with Gates’ statement that “Teaching is multifaceted, complex work.” I also think that his push for robust teacher evaluations that help give direct feedback to teachers so they can improve their practice is a good thing. Mr. Gates calls for trained peers and supervisors to provide this feedback. I would love to invite a team from his foundation come visit me teach, so I can get that direct feedback on how to improve. In return, I’d love to be trained so I can pass it on and give this feedback to others. If there’s a way to sign up, let me know.

Effective teaching requires complicated measures, and I don’t believe that we’ve reliably figured out what combination of those metrics are. Unfortunately, the term ‘teacher accountability’ tends to scare people away from “creating a personnel system that truly helps teachers improve.” As reported in an article titled “Teacher Quality Widely Diffused, Ratings Indicate,” the actual publication of New York’s assessments show that high and low performing teachers exist in every school regardless of wealth, neighborhood, or population.

The theme of the NAISAC12 conference is Innovation. I am a big fan of the work the Bill and Melinda Gates Foundation do and think its research into improving schools will benefit us all.

The Gates foundation recognizes the need to implement new ideas, and even if those attempts at education reform don’t work, analyzing and learning from the data is important. Microsoft, the company Gates founded some time ago took many risks and has been very successful, but along the way, it has also produced some things that didn’t work as well as they’d hope (remember the Kin anyone?). That didn’t stop them. In fact, I’m quite excited to see Microsoft trying to be a player in the mobile world. It promotes innovation from all its competitors.

In today’s op ed section of the NYTimes there’s an article titled “True Innovation” about Bell Labs. Last year I read two great books about innovation and risks: Steven Johnson’s Where Good Ideas Come From: The Natural History of Innovation and Tim Harford’s Why Success Always Starts with Failure: Adapt. Like so many things that end up being polarized, I think many things do not have to be either/or. The article and the books mention the need for both autonomy and collaboration. They are not exclusive of each other. The challenge is finding the balance, so that the continued cycle of improvement promotes both teacher accountability and innovative teaching.

If I had the chance, I’d like to ask Bill Gates this…

To fuel innovation, we often need to take risks. Risks come with many rewards, but they also come with failure. How do you balance teacher accountability while supporting and promoting innovative teaching?

If anyone gets a chance on Thursday to get behind a mic and ask this question, I’d love to hear his response. 


How Do You Measure Teacher Effectiveness: 7 Challenges

I can’t believe when I go in to teach tomorrow it will be December already. The Dec/Jan issue of Ed. Leadership focuses on “The Effective Educator” and its first article, “Measuring Effectiveness: What Will it Take?” focuses on several challenges associated with trying to measure effective educators. Given all the public school incentives, there are many trying to quantify what an effective teacher looks like.

Challenge 1: The Limits of Student Assessment Data

  • Using this data as a measurement of teacher success is relatively untested as a high-stakes measure
  • Testing experts like Popham (2007) say that standardized tests “may not accurately reflect the quality of instruction specifically provided to promote students’ mastery of what is being assessed.” For example, we spend time in our classes for social/emotional learning, social studies, not to mention specialist subjects like art and music that are not on elementary standardized tests. Should kids be deprived of PE or recess in order to get more math or  reading instruction time? Some schools feel pressured enough to do that.
  • “Too many standardized tests do not demonstrate whether a teacher’s instruction had an effect on the students’ performance.”

Challenge 2: Many Untested Subjects

  • Are we going to measure the effectiveness of a second language teacher based on ERB math scores? I hope not – even though there is a good chance foreign language instruction increases vocabulary scores.

Challenge 3: Quality of Evaluators

  • How trained are teacher evaluators to actually measure teacher performance based on student success?
  • The prediction is that evaluator training will go up in many states.

Challenge 4: Individual vs. Team-Based Accountability

  • Who is evaluated for a child who gets learning support from another teacher in addition to their classroom teacher?
  • When I team up with my colleagues and plan a unit and teach it collaboratively, how does one measure that?

Challenge 5: What Else Matters?

  • I’ve mentioned previously, that some kinds of learning support important values, but don’t necessarily increase test scores.

Challenge 6: Working Conditions

  • I work in a great facility, but other things to take into consideration is: time available to each teacher, collaboration, technology equity, the particular group of students, physical spaces, budgets, etc.

Challenge  7: Engaging All Stakeholders

  • We’ve come a long way to involve students in their own assessments, are we doing the same with teachers. Are parents, teachers, students, trustees, and administrators all part of the discussion? According to the article they should be.

At this point, it is really difficult to determine a teacher’s effectiveness. Yet, one knows it when one sees it. There are groups trying to make it more tangible and scientific. They are trying to “shift the model from the sole unit of authority and responsibility to next-generation systems that recognize the importance of professional collaboration, transparent practice, reflective and collective inquiry, and joint accountability.”