The Tests Don’t Test What They Purport to Test

test

My third graders (they’re eight or nine years old — keep that in mind) have finished their state testing for the year. They got off easy — just two tests, ELA and Math. The State of Michigan, a few weeks before the testing window opened, sent a letter to parents explaining the purpose of the tests:

“[The tests] are designed to provide information on student knowledge and ability to be career- and college-ready upon graduation. Schools and districts use the results for curriculum planning and school improvement initiatives that benefit all students.” [Source]

Let’s ignore for the moment the dubious claim that any test can predict how “career- and college-ready” a person will be nine years later and focus on what the tests are supposedly designed to do: “provide information on student knowledge.”

If that were true, most teachers would be fine with them. Want to find out if kids can read? Give them something to read and ask them a handful of questions about it. Need to determine if teachers are teaching kids math? Give them 20 math problems that they might someday encounter in the real world and see if they can figure them out.

Confucious said, “Life is really simple, but we insist on making it complicated.”

The same is true of those who make these tests.

Because instead of assessing whether or not my third-graders can read or do math, here is what they were really tested on:

Stamina

It took my best student nearly three hours to finish the ELA test and almost two to complete the math test. Let me state this as clearly as I can: You don’t need three hours to find out if a kid can read or do math. A three-hour test doesn’t assess ability; it tests stamina.

How to Navigate Foreign Formats

The state of Michigan provides a way for students to practice using the tools they will encounter on the tests. And, in fairness, it does a decent job. However, of the 30 questions provided, not one of them required students to click on the tool necessary to enter a fraction for an answer. So I’ll give you one guess what happened when my third-graders had to enter a fraction on the actual exam.

Intrinsic Motivation

There are, as yet, no stakes attached to these tests for the students. And they know this. The state of Michigan makes sure of it. From the same letter as the one referenced above:

“State assessment results do not impact student grades.”

They don’t impact anything else, either, as far as the students are concerned. Which means that there are really only two reasons for them to try their best. Either they’ve learned to always give their all, or they want to please adults. No wonder, then, that a handful of students breeze through the test every year. I can’t say I don’t understand why.

In an earlier article, I suggested a potential remedy: bribery. That’s because studies show that it works. In one, researchers concluded that if the U.S. had used financial incentives during the 2012 PISA test, the country’s math ranking would have risen from 36th to 19th. In another, the impact of incentives had an effective size similar to a one standard deviation increase in teacher quality or a 20% reduction in class size.

Grit

For the record, I have no problem with these tests assessing student perseverance. Persistence, grit, or whatever you want to call it is a trait that serves people well in school and in life. The tests do an excellent job of assessing it with long reading passages lined up one after another and multistep story problems embedded in a test that students know will take more than two hours.

But if grit is what we’re testing, then be honest about it. Don’t say it’s a reading test or a math test, when it’s really a test of character. School districts use these results to make curricular decisions and journalists report the results to inform the public on the status of our schools. So when a test doesn’t test what it purports to test, it leaves districts fixing a problem that may not exist while ignoring those that do, and newspapers describing the wrong deficiencies.

Reading Ability

You have to be a good reader to do well on the ELA test. That’s good, to a certain extent. But what the test doesn’t do a good job of doing is determining what reading skills students have.  If you want to find out if students understand cause and effect relationships, you can do that with a fairly simple text. Same goes for every other skill students are supposed to learn K-3. But the test doesn’t include below grade level passages, which means that if you can’t read the text, it doesn’t matter how well you can find the main idea, or understand the organizational structure of a non-fiction article, or differentiate between your and the author’s point of view,  or literally any other thing your teacher did an excellent job teaching you.

And if you aren’t reading at grade level come test time, you’re really screwed on the math test. Because the math test only sort of tests math knowledge. Mostly, it’s another reading test. So when the state of Michigan claims that schools “use the results for curriculum planning” and the results show that your students aren’t very good at math, you might want to think twice before throwing out your math curriculum, because you may have a reading problem.

Don’t Trust the Results

The tests don’t test what they purport to test, which makes the results confusing and not very useful. Schools believe they have a problem here when they may actually have a problem there. Journalists write stories with headlines like:

Less than half of 12th-graders can read or do math proficiently

65 Percent of Public School 8th Graders Not Proficient in Reading 

Only 25% of Nashville elementary, middle school students on grade level in reading, math

 

Those are misleading, and they are the gift that keeps giving to those who want to dismantle public schools.  If you set out to design a system to undermine public education, you could do a lot worse than designing tests that are harder than they used to be, longer than they need to be, and have no stakes for the people who take them.

There’s a saying, “Don’t believe the hype,” which suggests people ignore the marketing and media buzz around a phenomenon. When it comes to the standardized tests students are taking today, I suggest people “not believe the tripe.” Because the tests just don’t test what they claim to test.

 

4 Replies to “The Tests Don’t Test What They Purport to Test”

  1. Yes to all of this. My state may claim that the tests are used to make curriculum decisions, but they are really judgments of schools and teachers, and unfair judgments at that. To further the point they are not used for curriculum decisions, in my subject of language arts, the only “data” (quotations used to denote sarcasm) we get upon which we can base such a decision are in broad, vague categories such as remediate, monitor, or accelerate learning in reading/vocabulary and writing/language. Which standards need strengthening? No one knows. Which type of reading is weak–literary or informational? Sorry, out of luck. As a result, we keep feeling our way around in a completely darkened room that has no light switch, candle, or even a flashlight.

  2. It sounds like the nonsense being inflicted on primary and secondary students in Australia (equivalent US elementary to about freshman level) called the NAPLAN tests.Back when I was teaching in NSW such a thing did not exist – but since its introduction – English and Mathematics – the levels of Australian students has fallen from way up there – down to the dismal levels of the US (as politicians keep reminding us all – devising with their testing buddies more ways of control over curriculum and public education). Your essay here is a corrective re their ignorance – and certainly not that of the students – if only the politicians would remove their deadening hand and control and their hubristic assumption that being a politician makes them an immediate expert on everything – but especially on what the educational/learning field should do/be! They are abysmally wrong. And traits to the young people over whom they inflict control testing mechanisms! Ugly is not the half of it!

Leave a Reply

Your email address will not be published. Required fields are marked *