Data: Is it Reliable? And What do We do with it?

It’s been almost a couple of months since my last post, and I find myself thinking of data again.

Earlier this month, the Gates Foundation released its cumulative findings on its 3-year Measures of Effective Teaching (MET) research project. They recommend a balanced approach which included observations and student perception surveys in addition to achievement test scores. If you look at the data in the report, much can be gleaned, yet it’s easy to see that effective teaching is a very complex thing to measure.

Also in the local news this week, teachers from a two different Seattle Public Schools, for various reasons, have stated they are going to boycott the district standardized test known as the Measures of Academic Progress (MAP).

There are many reasons standardized tests cause anxiety among students, teachers, parents, and school leaders. Often they are used as sorting mechanisms (admissions into schools, teaching effectiveness, and putting students on a certain track are just a few examples). Yet, if one approaches the data from these assessments with more purpose (to set new goals, to inform ones teaching, provide meaningful feedback, or guide learning), these measures can be useful.

Data today is abundant, but is it the right data? How data is collected, analyzed, and interpreted; how reliable it is; and what we do with it can make all the difference. Though the Gates Foundation and those Seattle Public School teachers are doing it differently, I’m glad there are many out there asking these questions.

Is Quantifying Teacher Performance Akin to Flipping A Coin?

Last week, on the way home from school, I tuned into a story on the radio titled: “Seattle Releases First Teacher Ratings Based on Student Performance.” Data is great, but if you paid attention to the elections a few weeks ago, there were two kinds of math going on. Nate Silver’s Five Thirty Eight blog predicted 50/50 states. Karl Rove’s analysis of the data had him flummoxed. The difference was that Rove was emotionally attached, was eager to win, and for some reason his analysis of the same polls was way off. Alternatively, Silver simply plugged numbers into complex algorithms.

Mathematicians have noted that test scores and teacher performance don’t necessarily have a strong correlation, yet an incredible weight and cost is attributed to these standardized tests. Math professor Johh Ewing says, “You might as well look at all the teachers and flip a coin and those that get heads, say, are good, and those that get tails are bad, and it’s not much different from using one year of growth to measure teachers,”

Ewings paper, “Mathematical Intimidation: Driven by Data,” Looks at the potential pitfall of trying to create Value Added Measures to teacher evaluation.

Like the election examples earlier, we often attach a lot of emotion to the data creating a lot of noise. This noise had the potential to lead to bias. When a teacher says, “But I’ve done this for 20 years. I know this works,” it is evident that experience plays an important role. But is there bias involved. During those 20 years, did that teacher ever once control the experiment by not utilizing a particular skill? If so was the result the same, better, worse. Without trying to control for various things, how does one really know if what you do works. Is it just a feeling or is it based on empirical data.

Finally, there are so many things that make a good teacher: relationships with students, high expectations, integrity, care, leadership, collaboration, etc. Yet all of these traits can’t be tested for.

Standardized test scores are a reality and here to stay. As long as graduate schools use test scores as a tool to help with admissions, and undergraduate schools do the same, high schools and middle schools won’t have much of a choice. Elementary schools just follow.

There’s a dark side to this. Children as early as Pre-K are getting tutored in test preparation. Like the qualities of teachers, students have many amazing strengths and skills. However just because they struggle with test taking, potential doors my be closed without even giving the child a chance to show the brilliance that lies within.

And what about those 21st Century Skills – Critical Thinking, Communication, Collaboration, Creativity, etc. Will teachers drop integrating teaching these skills in order to meet the demands of the test scores? I hope not.

What Are Teacher Leadership Standards?

It was a marathon of a day with little time between to take in all that I was learning between sessions. One session that resonated with me was the call for identifying teacher leaders and giving them various responsibilities – not as add-ons, but by providing them the structures to take on these responsibilities. According to Kathryn Boles, we lose too many of our best teachers and attrition rates are too high. Many teachers do not want to become a principal/head of school, but they aren’t given the opportunities to be the change agents they want to be while still in the classroom. There are seven domains/standards for teacher leaders that have been identified.

Domain I: Fostering a Collaborative Culture to Support Educator Development and Student Learning

Domain II: Accessing and Using Research to Improve Practice and Student Learning

Domain III: Promoting Professional Learning for Continuous Improvement

Domain IV: Facilitating Improvements in Instruction and Student Learning

Domain V: Promoting the Use of Assessments and Data for School and District Improvement

Domain VI: Improving Outreach and Collaboration with Families and Community

Domain VII: Advocating for Student Learning and the Profession

You can get a lot more information about all these standards at this site (still under construction, but already very good). Every administrator should know about this site. Not only have standards for teacher leaders been developed, but the supporting strategies to support these have also been identified.

  • Increase the capcity to create staffing models that include differentiated career options for teachers. It’s shouldn’t solely be just teacher ==> assistant principal ==> principal ==> superintendent.
  • Develop new structures for licensing and/or credentialing teacher leaders.
  • Engage stakeholders in developing criteria-based models for the selection of teachers to serve in formalized leadership roles.
  • Develop systems for reward and recognition of the contributions of teachers in formal and informal leadership roles.
  • Establish compensation systems that recognize teacher leadership roles, knowledge, and skills.
  • Establish a performance management and evaluation system that is consistent with the identified and varied roles of teacher leaders.

I’ve a lot more to add on this one topic alone, but as I mentioned, the schedule is packed solid – Fantastic, but full. In fact, I’m getting ready for another 12-hour day of learning which starts in 15 minutes. Hopefully, I’ll be able to break some of the things I’m learning down into little chunks and how they apply to the classroom. In the meantime, check out that link above.

Atlanta Public Schools Open Amid a Testing Scandal – NYTimes.com

Atlanta Public Schools Open Amid a Testing Scandal – NYTimes.com.

This article from todays NYTimes is alarming, but not all that surprising to me. What is really being assessed when tests create so much anxiety and pressure on students, parents, teachers, and administrators?

As a teacher, I’ve gone from worrying about how well my students do, to actually focusing on their mistakes. Mistakes actually provide you with a lot more information. Being able to analyze kids’ errors helps me understand and reflect on what I need to change. When I look at a test item, and see that more than half of my class got the item incorrect, it’s a good place to start asking myself why.

Unfortunately, standardized tests are good for expediency, but not always good for learning. The wrong answer doesn’t always provide enough insight. Take 2-digit subtraction with regrouping (borrowing), for example. If a child got the answer wrong, was it due to a misunderstanding of place value, did the child have a directionality issue, did they miss a step in the algorithm, did they simply add by mistake. A good standardized test may include incorrect answers that reveal some of the reasons, but not necessarily all. The only way to know for sure is to observe a child doing the problem and then asking them to explain what they did and why. It’s amazing the kind of insight you can gain from a few simple questions. Furthermore, with a test that provides four possible answers there’s a good chance your student had no clue, but guessed correctly. The correct answer provides very little information.

The other problem with some of the standardized tests out there, is the timeliness of the test-makers correcting and returning the results. By the time many schools get them back, it’s well past the point that they can inform the teacher with something useful about what they can change. With NCLB (No Child Left Behind) and RTTT (Race to the Top), the focus of test scores often becomes, “How did we do?” rather than “What can we learn from this?”

Many test companies are going to computer testing, which I think is great in terms of timeliness, but I wonder how kids 7 and under will do with a mouse. I’d rather the little ones touch their answers on a touch screen, but I suppose a mouse isn’t that far removed from filling in bubbles with a pencil.

This news story isn’t the first of its kind, but I hope it helps change the kind of pressure and anxiety that these tests can place on everyone involved. I’m not opposed to standardized tests; I think there’s a place for them. We have to keep asking, though, what are these tests actually testing and how can they help us be more effective. I hope that the policy makers behind NCLB and RTTT can learn from their mistakes and make student assessment something that’s actually FOR students and teachers rather than an assessment OF them.

I have one other minor criticism about these tests: they create a mindset of having only one right answer to a problem. While this may be true for a test item, we know that innovation comes from thinking outside the bubble and entertaining many possible solutions to more complex problems. By all means use standardized tests, but also include student interviews, their own reflections and assessments, observations, and the myriad of other assessment tools available.

Great Professional Development Resource

It’s been a couple of weeks since I was at the ISTE (International Society for Technology in Education) conference, and my head is still full of resources and information. Today, I got an email from them with a few statistics about this year’s conference.

“More than 17,850 educators and exhibit personnel attended ISTE 2011, held in Philadelphia at the Pennsylvania Convention Center June 26-29. Conference highlights included:

  • 13,336 registered attendees
  • 4,562 exhibit personnel
  • Dozens of workshops with more than 2,611 tickets sold
  • An exhibit hall the size of 5.5 football fields featuring 1,423 booths and 501 companies
  • 149 registered journalists from around the globe
  • 1,025 attendees sent more than 3,000 letters to the U.S. Congress
  • Among the attendees were 1,152 presenters and 940 international attendees from 63 countries”
As I mentioned in an earlier post, it was daunting. Also in that email though, was a link to their ‘white paper’ on Coaching. ISTE’s webpage summarizes the details of the paper like this:
    • Situation: Effective use of technology is essential for teaching and learning in a global, digital age.
    • Problem: Many teachers do not know how to design and support technology-rich learning environments.
    • Solution: Coaching, combined with communities of learning, is a highly effective job-embedded professional development model
    • Result: Teachers experience technology as an effective tool for professional learning and develop the skills to powerfully use technology to improve student learning.
The paper’s content highlights include:
    • Introduction to three coaching models that provide highly effective professional development
    • 10 tips for leveraging technology, coaching, and community
    • 5 key benefits that result from the integration of technology, coaching, and community
    • Introduction to the NETS×C
You can download the whole paper here. It’s really a great read for all teachers/administrators who are trying to make changes in tech to better enhance student learning.

Best PD for Teaching IS Teaching

It’s been a while since I’ve taught during the summer, but this one particular program I started at yesterday intrigued me. First, the objectives of the classes were not written the way  State Standards or Core Curricula are written. For example, one of the objectives in one of the classes I’m teaching is for the student to ‘explore the different ways to employ creativity techniques in the development of a new invention.’ Second the classes are 90 minutes long which really allow for project/problem-based learning activities. Third, these are all multi-aged classes, so I’m seeing kids from ages 5 to 12 throughout the day. Not having committees, faculty meetings, regular email communication with parents, homework to assign, and unbelievable amounts of autonomy to reach or adapt these objectives to the actual kids I’m teaching, I have had time to play with, use, and have kids use technology in the class already. Finally, the program is only three weeks long, so there’s a lot of interesting thought that goes into planning out the courses. There are a lot of books about regular classrooms and how important it is to set the tone and expectations for kids in the first 6 weeks. I’ve only got three!

One can read and see examples of project/problem-based learning, but until you have a solid 90 minute block and figure out how to utilize that time best to suit the needs of the kids, it’s just a theory. By nature of the schools I’ve worked in, I haven’t taught a multi-aged class in over a decade. It’s been a lot of fun (and it’s only been my second day on the job). I am also loving the objectives being so open-ended and relevant to kids’ lives. While objectives for basic skills can be and are appropriate, it is evident that these kids are getting basic skills instruction and practice as part of their project/problem-based objective. Just thinking about the ‘real-world’ product that kids will create as a final assessment has been fun for me. Making the material relevant to them now, not someday in the future increases their motivation incredibly.

Professional Development can happen in so many ways. We can have workshops, attend conferences, teach other teachers, or coach, but in my mind, I think the best way to become a better teacher is to keep trying new ways to teach and adapt to your students.

In our own schools, it is possible for us to develop professional development like this. According to Douglas B. Reeves in his book Transfroming Professional Development into Student Results, he notes that not only does a school have to have vision for this kind of PD, but also implementation. Without implementation, the vision “not only fails to achieve the intended objectives but also engenders cynicism and distrust.”

Reeves also criticizes most schools for what he calls “Institutional Multitasking,” and that we need to FOCUS: Focus on teaching, curriculum, assessment, and leadership. Darling-Hammond and Richardson (2009) stated that the largest effects in teacher improvement were found for programs offering between 30 and 100 hours over 6 to 12 months. We’d have to use all our faculty meetings and in-service days throughout the year just on one topic to reach that goal. So what is one of the biggest factors in supporting this kind of PD? The schedule. Marzano (2009) notes that school “leaders must be the architects of systems and schedules.”

Finally, Reeves talks about recognizing our biases and being willing to fail. “School leaders have a particular responsibility to respect research integrity, particularly when a teacher-researhcer expresses disappointment that a planned intervention was ineffective.” Teachers have to get over their fear of being wrong or making mistakes. It’s how we learn.

This summer job that I’ve got is a great one. Including the work I’m doing prepping for each class, I’m spending about 105 hours. That definitely puts me in Darling-Hammond’s range. Unfortunately, it’s not over 6 to 12 months. The systems and schedules for the next school year, may determine how effective our school’s PD is. I will have to build in my own to maintain what I’m currently learning.

We had a guest speaker talk about the campus's Green initiatives. These kids are examining native and invasive species on the campus's wetlands.

What is the Point of Learning?

Several weeks ago, I was fortunate to attend a TEDx event at Eastside Prep. Several speakers really spoke not only about what’s wrong with much of our education system (that would be too easy to do). They also spoke of changes that would enhance learning – the real kind of learning where you take risks, sometimes fail, but persevere until you get it. The reward is intrinsic. The value and motivation comes from the learning itself. For example, changing the schedule to allow for longer deeper inquiry or assessment without grades. Honestly, if I was evaluated on a grading system rather than through goal setting, feedback, and reflection, I wouldn’t do it. So why do we do this with kids in middle, high, and even some elementary schools? I’m glad people are motivated to put on events TEDx events focused around education. There’s another TED education event in this area, TEDxOverlake (How People Learn,) happening on June 18.

This week, the videos of these talks were posted, and I wanted to highlight a couple of speakers that addressed the above in different but concrete and passionate ways. The first is by Shawn Cornally, a high school math and science teacher called The Future of Education Without Coercion (you should also check out his blog, ThinkThankThunk.

The second is the talk by Dr. Tae whom I wrote about a few weeks back. His talk was titled: Can Skateboarding Save Our Schools?

If you’ve read Daniel Pink’s Drive, read Alfie Kohn’s Punished by Rewards, seen the movie Race to Nowhere, or heard Sir Ken Robinson speak or read his book The Element, a very similar theme emerges in all of them.