Data: Is it Reliable? And What do We do with it?

It’s been almost a couple of months since my last post, and I find myself thinking of data again.

Earlier this month, the Gates Foundation released its cumulative findings on its 3-year Measures of Effective Teaching (MET) research project. They recommend a balanced approach which included observations and student perception surveys in addition to achievement test scores. If you look at the data in the report, much can be gleaned, yet it’s easy to see that effective teaching is a very complex thing to measure.

Also in the local news this week, teachers from a two different Seattle Public Schools, for various reasons, have stated they are going to boycott the district standardized test known as the Measures of Academic Progress (MAP).

There are many reasons standardized tests cause anxiety among students, teachers, parents, and school leaders. Often they are used as sorting mechanisms (admissions into schools, teaching effectiveness, and putting students on a certain track are just a few examples). Yet, if one approaches the data from these assessments with more purpose (to set new goals, to inform ones teaching, provide meaningful feedback, or guide learning), these measures can be useful.

Data today is abundant, but is it the right data? How data is collected, analyzed, and interpreted; how reliable it is; and what we do with it can make all the difference. Though the Gates Foundation and those Seattle Public School teachers are doing it differently, I’m glad there are many out there asking these questions.

Is Quantifying Teacher Performance Akin to Flipping A Coin?

Last week, on the way home from school, I tuned into a story on the radio titled: “Seattle Releases First Teacher Ratings Based on Student Performance.” Data is great, but if you paid attention to the elections a few weeks ago, there were two kinds of math going on. Nate Silver’s Five Thirty Eight blog predicted 50/50 states. Karl Rove’s analysis of the data had him flummoxed. The difference was that Rove was emotionally attached, was eager to win, and for some reason his analysis of the same polls was way off. Alternatively, Silver simply plugged numbers into complex algorithms.

Mathematicians have noted that test scores and teacher performance don’t necessarily have a strong correlation, yet an incredible weight and cost is attributed to these standardized tests. Math professor Johh Ewing says, “You might as well look at all the teachers and flip a coin and those that get heads, say, are good, and those that get tails are bad, and it’s not much different from using one year of growth to measure teachers,”

Ewings paper, “Mathematical Intimidation: Driven by Data,” Looks at the potential pitfall of trying to create Value Added Measures to teacher evaluation.

Like the election examples earlier, we often attach a lot of emotion to the data creating a lot of noise. This noise had the potential to lead to bias. When a teacher says, “But I’ve done this for 20 years. I know this works,” it is evident that experience plays an important role. But is there bias involved. During those 20 years, did that teacher ever once control the experiment by not utilizing a particular skill? If so was the result the same, better, worse. Without trying to control for various things, how does one really know if what you do works. Is it just a feeling or is it based on empirical data.

Finally, there are so many things that make a good teacher: relationships with students, high expectations, integrity, care, leadership, collaboration, etc. Yet all of these traits can’t be tested for.

Standardized test scores are a reality and here to stay. As long as graduate schools use test scores as a tool to help with admissions, and undergraduate schools do the same, high schools and middle schools won’t have much of a choice. Elementary schools just follow.

There’s a dark side to this. Children as early as Pre-K are getting tutored in test preparation. Like the qualities of teachers, students have many amazing strengths and skills. However just because they struggle with test taking, potential doors my be closed without even giving the child a chance to show the brilliance that lies within.

And what about those 21st Century Skills – Critical Thinking, Communication, Collaboration, Creativity, etc. Will teachers drop integrating teaching these skills in order to meet the demands of the test scores? I hope not.

What I’d Like to Ask Bill Gates Next Week

Next week, people from many places associated with independent schools will be in town for the National Independent Schools Annual Conference here in Seattle. I’m excited about this week for many reasons and hope to write about them in the coming days.

One of the things I’m interested in is what the featured keynote speaker, Bill Gates, has to say. I won’t be able to hear him speak directly on Thursday as I’ll be teaching. I will, however, be able to follow his address through many various channels.

I read his opinion piece in the NYTimes on Friday about his thoughts on New York making teacher performance assessments public. I agree with him on many points. One of these is that making teacher evaluation assessments publicly available isn’t going to do anything to help improve teaching. I also agree with Gates’ statement that “Teaching is multifaceted, complex work.” I also think that his push for robust teacher evaluations that help give direct feedback to teachers so they can improve their practice is a good thing. Mr. Gates calls for trained peers and supervisors to provide this feedback. I would love to invite a team from his foundation come visit me teach, so I can get that direct feedback on how to improve. In return, I’d love to be trained so I can pass it on and give this feedback to others. If there’s a way to sign up, let me know.

Effective teaching requires complicated measures, and I don’t believe that we’ve reliably figured out what combination of those metrics are. Unfortunately, the term ‘teacher accountability’ tends to scare people away from “creating a personnel system that truly helps teachers improve.” As reported in an article titled “Teacher Quality Widely Diffused, Ratings Indicate,” the actual publication of New York’s assessments show that high and low performing teachers exist in every school regardless of wealth, neighborhood, or population.

The theme of the NAISAC12 conference is Innovation. I am a big fan of the work the Bill and Melinda Gates Foundation do and think its research into improving schools will benefit us all.

The Gates foundation recognizes the need to implement new ideas, and even if those attempts at education reform don’t work, analyzing and learning from the data is important. Microsoft, the company Gates founded some time ago took many risks and has been very successful, but along the way, it has also produced some things that didn’t work as well as they’d hope (remember the Kin anyone?). That didn’t stop them. In fact, I’m quite excited to see Microsoft trying to be a player in the mobile world. It promotes innovation from all its competitors.

In today’s op ed section of the NYTimes there’s an article titled “True Innovation” about Bell Labs. Last year I read two great books about innovation and risks: Steven Johnson’s Where Good Ideas Come From: The Natural History of Innovation and Tim Harford’s Why Success Always Starts with Failure: Adapt. Like so many things that end up being polarized, I think many things do not have to be either/or. The article and the books mention the need for both autonomy and collaboration. They are not exclusive of each other. The challenge is finding the balance, so that the continued cycle of improvement promotes both teacher accountability and innovative teaching.

If I had the chance, I’d like to ask Bill Gates this…

To fuel innovation, we often need to take risks. Risks come with many rewards, but they also come with failure. How do you balance teacher accountability while supporting and promoting innovative teaching?

If anyone gets a chance on Thursday to get behind a mic and ask this question, I’d love to hear his response. 


Value of Teachers and the 1%

Last week, the nytimes listed several job markets where one would find the top 1% in this country. It also went on to list the degrees in which the top 1% graduated from. It was interesting that they were also running articles on the value of teachers based on the Harvard/Columbia study that came out recently:

Big Study Links Good Teachers to Lasting Gain

Value of Teachers

And an interesting debate about value and measuring teacher effectiveness followed.

The 1% articles talked about the various professions. In the print version, teaching didn’t even make the graphic. On the online graphic, they were there, but a clear side note mentioned that teachers in 1% households were there because of marriage.

And here’s What the Top 1% Majored In. My undergrad degree is in biology, but after that I chose to pursue education (not listed). These two letters responding to those graphs, one by a teacher, and one by a father of two teachers say a lot.

There’s something amiss in the way teachers are compensated. I’m not pro- or anti- union, but see the benefits and challenges with both systems when it comes to teaching. There are districts, charter schools, and independent schools trying a number of schemes and some doing better than others. It be great to look at all the possibilities, find out which ones are working best, try and guess why, and start to try it out. That’s how innovation happens; You look at all the ideas out there, develop your own compensation prototype, take a risk (a calculated one, of course), analyze, modify, and keep looping back refining and revising the prototype. There’s a good chance it’s not going to work right away and will ruffle a few feathers, but I think it’s worth the risk. As long as one acknowledges and learns from mistakes, something good will eventually emerge. That’s part of what innovation is all about. School culture in general is invisible, deep, complex, and very conservative – it’s not an easy task.

Debate About Using Data to Measure Teacher Effectiveness

Worth the read: A good debate on using data to measure teacher effectiveness.

Measuring teacher effectiveness is complex, and I agree with the economists that if all evaluations are based on test scores, teaching to the test will increase, especially among the less effective teachers.

Unlike batting averages, most ‘official’ student test data is acquired only once a year. We’ve also seen some of the negative effects high stakes have had on sports. Also, unlike athletes, teachers don’t all peak at the same time.

I feel like I’m an effective teacher, but I’m not sure how to really assess that. I also feel I have a lot more to learn and plenty of room for improvement.

Two Words I’d Like to See Disappear From Education Articles This Year

Finland & Singapore.

Both countries are on my list of places I’d like to visit one day. What I’m tired of reading is articles that keep trying to compare their education system to the US’s. I agree that it’s important to look for what’s working in education. Unfortunately, comparisons to student achievement and teacher quality in those countries to the US cannot be done easily. There are many challenges that face US education, both independent and public, but simply comparing them to Finland or Singapore is unfair. Finland, Singapore and the US are very different.

In the past few years, Finland has been one of the countries that has consistently placed first on international academic tests. It doesn’t surprise me then that many education reform leaders are trying to look at their practices. In a nytimes article last month, it even mentioned that the president of the National Association of Independent Schools, Pat F. Bassett has made the pilgrimage. The one thing I like about Finland’s education system is that most kids don’t start testing and homework until their teens!

Still, Finland or Singapore are small countries that value teaching. It can be harder to get into a school of education than into law or medicine (and the schooling is fully subsidized). they recognize that the quality of teaching matters and they support that from the beginning.

Apart from that, looking at what Finland or Singapore is doing right is not going to fix the challenges that exist today in the US. I agree with Linda Darling Hammond, that it might be a good model for a state like Kentucky, but both those countries have a much more homogenous society.

The US education system has to stop being reactionary. It cannot repair itself simply by learning from Finland. It has to innovate and lead. Education in this country has always been able to do that, and I am optimistic that it will continue to do so. There are many amazing schools with incredible teachers doing many things right. We should start to look at those as models first. Good teachers eventually find the schools that fuel and support their passion and purpose. It should be the other way around. Schools should be finding those qualified teachers.

Just think of all the people who learned Russian or Japanese not too long ago. I think it’s great that they learned a foreign language, but I wonder if those choices were based on what was going on in the world at that time. Finland and Singapore are both on my list of countries I’d like to visit, but somehow I don’t think the general population is going to be learning Finnish anytime soon.

hyvää yötä

What is the Value of a Good Teacher?

A “study, by economists at Harvard and Columbia universities, finds that if a great teacher is leaving, parents should hold bake sales or pass the hat around in hopes of collectively offering the teacher as much as a $100,000 bonus to stay for an extra year.”

This is taken from an op-ed piece by Nicholas Kristof in today’s nytimes titled, “The Value of Teachers.” Of course that’s never going to happen – at least not in the foreseeable future. But why not? Teachers are still mostly rewarded simply by how long they’ve been teaching. There’s some incentive for those with a higher degree, but it’s minimal in most cases.

In many industries, there are many options for growth opportunities. Not for teachers, though. I am very fortunate that I work with many who are always looking to improve the way they teach, however the opportunity for career advancement is limited. I should clarify. There are a myriad of different opportunities in education for advancement. All of them, however, are far removed from the classroom. One could become an administrator, consultant, researcher, academic, and the list goes on. Unfortunately, none of that list includes remaining in the classroom. At this point in time, good teachers are truly undervalued.