Up the Down School Tests

When school begins in September, class placement decisions will have been made for elementary and middle school students based on the math scores released by the New York public schools last Tuesday.

Without an independent analysis of the scores and the tests themselves, though, it is wishful, at best, for educators and politicians to suggest that the 6.9 point increase at the state level and the 8.1 point gain for city students, are a true reflection of improved proficiency.

Meanwhile, the education commissioner and chancellor bask in the news, attributing the results to higher standards, wiser policies, and the implementation of good practices.

In terms of academic growth, however, the reality is that learning basic skills on a mass scale is a gradual process that unfolds over time. To appraise student achievement meaningfully requires tests that can measure such incremental progress in a carefully calibrated and reliable way.

That is why, compared to the statewide tests that are in use, the National Assessment of Educational Progress is a well-regarded yardstick. Its design is sound and its outcomes don't vary wildly from one occasion to another, as they do with statewide tests.

The crux of the problem in New York State and elsewhere is that there is enormous and increasing pressure to do well on whatever test series is being used annually to gauge achievement. Departments of education and their officers become clients, who hire publishers to provide the instruments by which they will be judged. Officials want data to evaluate how well "the system" is working. What they really hope for is evidence to prove that, under their leadership, everything is getting better.

Education agencies select from a limited number of publishers a company that is able to prepare a test that will do the job. These outfits are overburdened and spread thin because most are already working under contracts with other states to provide testing in compliance with the No Child Left Behind Act.

The relation between a client and publisher usually develops into a multi-year partnership. One of the claimed benefits of working with a particular outfit is that this provides continuity of results from one year to the next.

Much interaction between the two parties occurs as the tests and items are being constructed. State education department staff, in consultation with the publisher, have much to say about where the standards are set. The way the cutoff points are decided — the boundary line that separates a Level 2 from a Level 3 student — is a matter of judgment, not a scientifically derived quantum.

Deciding how many correct answers are needed to pass a certain level of the test will determine how many students cross the line. The idea that standards are constant reference points is a misconception that must be challenged. Simply put, some years it is easier to clear the proficiency hurdle than others because the standard varies.

These issues cannot be left to both working partners to address. The tests are "high stakes" for them too. Alone and together, they have a vested interest in positive test outcomes.

In fact, if the authenticity of the results is questioned, say, at an occasional education committee hearing, publishers agree they will step forward and attest to the test's validity, the competence of their psychometricians, the comparability of the results from year to year, the integrity of their scaling procedures, etc. This appeal to authority has always worked.

Then the publisher goes back to preparing tests that will be used next year. The educrats take credit and move forward. The teachers, whose talent, honest efforts, and classroom time are abused in anxious devotion to frantic test preparation activity, soldier on. And the education of the kids is sacrificed so that no child is left behind.

This cycle must be broken. Publishers, with the assent of their clients, are involved in a complicated special interest relationship. They know it behooves them to deliver tests that yield good news results.

Unless there's an independent auditing mechanism that allows the books to be opened to impartial analysis by outside experts that makes truth in testing the norm and that protects parents and students from this self-perpetuating arrangement, things may well be going downhill, even as we're told the results are going up.

Mr. Smith, a retired administrative analyst for the city's public school system, is writing a book about testing called "Failing the Test."