Disconnects, fallacies and dogma

Be sure to catch Education at the Brink 's excellent analysis of this CSM article on NCLB.

But reading the CSM article got me going on a different tangent.

Can I just bang my head against the keyboards?

One of my pet peeves is the fallacy that test scores show ‘achievement’.

I’m glad the CSM article points out one reason why it’s a fallacy: if you lower the bar for ‘achievement’, then you do get some kind of increased level of ‘achievement’ ,whatever ‘achievement’ really is. I don’t know what achievement is, by the way. It’s not defined anywhere. I’m presuming it means ‘higher test scores’. But there’s more to this than that little problem.

But here’s the second reason why I have apoplectic fits about connecting test scores to ‘achievement’…

Test scores are test scores. To me, they simply show how well people take that particular test.

But until you prove to me that a test score actually means something else other than that, I simply can’t and won’t buy it.

Anyone can make a test. You or I or Harcourt Brace or DK Publishing or whoever is making the big bucks these days off NCLB imperatives.

But to show whether a test really tests what it says it does, you’ve got to do research which shows that the test scores are connected to what you are testing. You might just be showing an association between test scores and airconditioning levels. And you’ve got to do lots of work like this, not just one study. You can’t simply publish a test and say, woo-hoo! Higher scores mean higher ‘achievement’. Sorry; to do this is bogus, a freaking disconnect, dogma.

There’s an official research term describing whether test scores really have relevance: it’s called determining external validity. Does your test that you’ve developed really test for what it says it does? I frankly haven’t seen any research for external validity anywhere.

Even worse, no one even thinks to question whether these tests really test for achievement. The assumption is that ‘hard numbers’ is proof enough. This is still another a huge fallacy, and it just gets me upset to see how unquestioning people are about scores.

The third reason I have heart trouble about testing is more meta in nature. Can a test really tell you that the kids are getting a good education? Is it right to determine whether kids are getting a good education just by one test?

A good education involves so many different components. I haven’t yet delved deeply into the education research world but I have a bet there is no consensus on what defines an excellent education. There’s too many worldviews, too many factors, too many constructs. It’s ludicrous to think that just one test can determine anything about education and yet that is the state of the art in public education today.

Despite all this, don’t get me wrong. I’m not as anti-testing as you might think. Testing can be very useful when used correctly. Really and truly. But I come from a place where I know that tests have weaknesses, and test scores require careful and nuanced interpretations. And, ahem, this ain’t happening.

Using test scores as a scythe to slash public education into little bleeding bits is not even close to being scientific, just tragic and nasty.