WHAT'S WRONG WITH RANKINGS, WITHIN AND OUTSIDE THE LAW--AND WHY THEY'RE USEFUL NONETHELESS

This is the time of year for lists: New Year's Resolutions, worst-dressed celebrities of 2002, and of course, the leading psychics' predictions for 2003.

Lists, especially those that rank people and institutions, are a source of amusement and information, but they can also be a source of consternation. No one wants his city to be ranked the 437^th best (out of 440) places to live in America, and you can bet that the Mayor of City 437 will have a bone or two to pick with the ranking criteria. So too with the Dean of the law school U.S. News & World Report puts last on its list.

Nonetheless, quality rankings can serve a useful function, not only because they enable the public to make informed choices, but also because of the incentives they create for the entities they rank. Unfortunately, though, the rankings that are easiest to compile do not always have the right incentive effects.

Let's take a brief look at the costs and benefits of rankings in a number of settings.

The Tyranny of College and Graduate School Rankings

University admissions officers frequently complain about the "tyranny of US News & World Report," a magazine that publishes annual rankings of "the best" colleges and graduate programs. Although the US News editors invariably preface their rankings with a warning to prospective students not to place undue weight on small differences in numerical rankings, students in fact do just that.

Someone deciding between a small college in the Northeast and a large university in the Midwest may well overlook minor differences in rank. But the truth is, a prospective student who does not have a strong geographic, size, or other style preference has little on which to base his decision other than relative rank.

For this reason, I'll bet that substantial numbers of law students now choose UCLA over USC, and Columbia over NYU, because UCLA and Columbia come out just barely ahead of their crosstown rivals on the US News rankings. I'll also bet that substantial numbers of students would make the opposite choice on the basis of a flip in these schools' relative positions were it to occur, as it has more than once in recent years.

Indeed, it is hard to say that deciding among colleges and graduate schools on the basis of relative US News ranking is wholly irrational. Perhaps one could argue that students should focus on the components, not the ranking itself; the magazine provides data on each institution's score across a variety of dimensions. But that is often not feasible; prospective students are not well-equipped to assign relative weights to the sub-category scores. (After all, how can you tell until you attend law school whether a low faculty-student ratio is more important than classmates with better numerical credentials?) As a result, prospective students would be justified in assuming that the aggregation formula chosen by the rankings' compilers is better than any formula they could devise.

Rankings as a Self-Fulfilling Prophecy

Moreover, rankings are, to some extent, a self-fulfilling prophecy. Suppose that in year 1, Fredonia State's advantage relative to Faber College is due entirely to Fredonia's large number of inactive emeritus faculty who (let us assume for purposes of this illustration) nonetheless count in the faculty-student ratio. One might then conclude that if Joe - or Josephine - College picks Fredonia based on its ranking, he or she has been misled.

But that conclusion might be wrong; indeed, Mr. or Ms. College may get even more of an advantage than he or she bargained for from picking Fredonia. That is because the rankings gap may well widen by year 5, when Joe (or Josephine) College graduates.

Why? By that time, four years' worth of students will have chosen Fredonia over Faber, so that Fredonia will have become the more selective institution (selectivity is a ranking factor too). Based on its initially undeserved higher ranking, Fredonia will have been able to choose students with better GPAs and LSATs than Faber; they will get better jobs; the school's prestige will grow, and so on.

The result is that even if, choosing solely by his or her own lights, Joe or Josephine College would rank Faber even with Fredonia, he or she should still choose Fredonia. Knowing that substantial numbers of other prospective students will choose Fredonia because of its (at this point still undeserved) rankings advantage, Joe or Josephine acts rationally by matriculating at Fredonia.

An undeserved rankings advantage based on an essentially inaccurate faculty-student ratio is thus transformed in an actual rankings advantage based on an accurate selectivity figure.

Thus rankings - even wrong rankings - become a self-fulfilling prophecy. Note that an uncorrected typo switching Faber and Fredonia in the rankings would have exactly the same effect.

The Baleful Impact of Rankings For Minority and Other Candidates

One of the most common complaints about the US News rankings is that they distort admissions offices' priorities. A prospective student with mediocre grades or test scores may have other extraordinary qualities that make her worthy of admission, but these qualities are unlikely to show up in the school's rankings.

Accordingly, the admissions office may feel obliged to reject her lest the inclusion of her data in the next year's report will harm the school's rank. One more point on the LSAT and one more point on the GPA may negate such factors as, say, Peace Corps experience, or particular brilliance in specific academic fields that isn't consistent across the board. A student with solid B pluses through college may be favored over one who grew up and by the end, achieved As.

This effect is especially painful for institutions that practice affirmative action for minority candidates. However, it is somewhat mitigated by the fact that US News calculates a school's student body quality by reference to grades and scores at the 25^th and 75^th percentiles.

That means that for matriculating students in the bottom quarter of the class numerically, no negative impact on rankings arises from lower grades and scores. For example, if a student at the 25^th percentile for a college has a high school GPA of 3.3, then it doesn't matter whether the rest of the bottom of the class is filled with students who had a 3.2 or with students with a 2.0.

What is the real impact of US News' policy of calculating 25^th and 75^th percentiles rather than mathematical means calculated based on every single student's statistics? As noted above, it can help minority candidates - and others whose test scores may not be the only relevant factor - but only to a point.

The real impact of the 25^th/75^th system of rankings on admissions is to restrict the total number of places in an entering class that can be devoted to students who--for whatever reason--are desirable despite relatively low grades or scores. Put another way, for every low-scoring student above the 25^th percentile, a school pays a price in its ranking. (Other distortions also follow from this approach: Why should scores be irrelevant at the top? Isn't a law school's ability to attract the very most high-scoring students at least somewhat relevant?)

Whether the 25^th/75^th restriction will continue to be an important one, of course, depends on whether the Supreme Court overturns its 1978 ruling permitting race-based affirmative action, when it considers the issue in the coming year. If such programs are ruled illegal, the distorting effect of rankings on schools' ideal affirmative action plans may be irrelevant, since those plans will have to be scrapped.

Still, even if the Supreme Court moots the impact of US News rankings on affirmative action, the general phenomenon will remain. Perhaps the clearest example comes from primary and secondary education, where the widespread use of standardized tests to rank schools and districts often leads teachers to "teach to the test," emphasizing skills that enable students to earn high scores without absorbing the knowledge they need to succeed later in life.

Gaming the System

Another complaint about rankings is that they can be gamed - manipulated dishonestly, or at least strategically, by schools and other institutions for whom rankings are a priority.

Consider hospital rankings, which are typically based on, among other things, mortality rates. A hospital administrator eager to move up in the rankings could take the very expensive measures of recruiting better doctors and nurses, improving equipment and routines, and so forth, or he could take the easy way out and simply turn away (or subtly steer away) the sickest patients. In fact, hospitals have used just this strategy of "creaming" the healthiest patients to improve their mortality statistics.

Other rankings can also be gamed. In the legal academy, rumors abound about law school libraries purchasing cheap and largely useless books, to boost their total-volumes score. One also hears of admissions offices artificially lowering the size of the first-year class so as to boost its credentials - only to admit large numbers of transfer students (whose numerical scores do not figure in the US News statistics).

The truth is that US News should not be, in effect, choosing the way colleges and law schools pick their students; the schools themselves should be. Instead, schools cater to the ranking system by which they are judged.

Improve the Rankings, But Don't Junk Them

Some critics of ranking regimes lament their very existence, arguing that prospective students or hospital patients would be better served by raw data. The argument, if not disingenuous, is at best naive.

The public craves information that ordinary people can use, and that means data aggregated into a small number of indices. And given press freedom, that in turn means that someone somewhere will inevitably provide rankings.

If current rankings systems distort educational priorities, patient care, and the myriad other goods and services provided by entities subject to ranking, then the right response is to create better measures.

If teachers teach to the test, then educational policy makers should try to design tests so that the best way to achieve high scores is to learn what the tests are designed to measure--and not simply to learn how to take the test. If hospitals cream patients, then hospital report cards should (as they now typically do) control for patient health, so that those who take the sickest patients willingly do not suffer for it in the rankings.

Likewise, as one critic argues, the US News criteria for law school rankings systematically favor private schools over public schools--by crediting schools for giving financial aid but not for having low tuitions, as most state schools do. Then perhaps the proper remedy is to devise a ranking system that more fairly accounts for the financial burden of three years of law school.

More generally, if the editors of US News do not change their formula, then someone else should calculate and publicize ratings based on a better formula.

Indeed, Professor Brian Leiter of the law school and philosophy department of the University of Texas has done just that. His biannual Educational Quality Ranking system "focuses exclusively on the three factors central to a good legal education: the quality of the faculty, the quality of the student body, and the quality of teaching."

Leiter makes a persuasive argument that his system better captures what should matter to students than the US News system. (And I don't just say that because I personally do well on one of Leiter's other set of rankings!)

Unfortunately for Leiter's excellent system, many superior systems don't catch on because they are overwhelmed by larger rivals--the Sony Betamax, the Apple Macintosh, the Dvorak keyboard, and so on. Leiter's rankings are unlikely to displace those of US News anytime soon--though readers of this column may well want to spread the word about the Leiter alternative.

Thus, in the short to medium term, law school deans and others who rightly object to the flaws in the US News system will have to keep making their case for prospective students to heed the magazine's own warnings, and put the rankings in perspective, realizing that they are approximate and there may not actually be much of a difference between closely-ranked schools.

The Value of Flawed Measures

Of course, many prospective students will not heed the warnings. But even though the flaws in the US News methodology will lead some students to enroll in the "wrong" schools and distort admissions priorities, their impact is not all negative.

The problem--in analyzing the US News ranking system and others--is that we tend to think of rankings as a method of figuring out who's on top, when in fact they're most useful for figuring out who's at the bottom.

Given small differences between the scores of, say, the number 14-ranked and the number-17 ranked universities, it may be possible for number 14 to move up a few places by manipulating its program in ways that have no educational benefit. But unless the measurement criteria bear no relation to the underlying value, an institution in the bottom ten percent will not be able to show substantial improvement without, well, substantially improving. So for very low-ranking schools, the incentives from ranking systems are, in sum, very good.

The furor over standardized testing of elementary and secondary school students illustrates this proposition well. The most vocal critics of such testing tend to be parents in affluent public school districts, whose students, by and large, perform very well on standardized tests. Nonetheless, because state mandates often tie rewards to overall performance on standardized tests, even teachers of generally high-performing students feel obliged to teach to the test.

But of course the state and national movements to require standardized tests were never intended to improve already high-performing schools. Standardized tests--even if flawed--were seen as a way of holding schools and districts accountable for the poor performance of the students in the worst schools.

And for that purpose, although there is considerable room for improvement, the tests do reasonably well. A school that does disastrously badly on its tests often becomes the focus for reform efforts, and for rightful parental outcry.

The Ranking Itch: The Urge to Know Who's On Top

The problem at the elementary and secondary levels is that standardized tests are not used simply to identify at-risk students and schools; they are used all the way up the quality chain.

Likewise, US News does not just sell its rankings issues to prospective students hoping to learn which colleges and graduate programs to avoid. Readers want to know who's on top. And although there is no unique answer to that question, the editors satisfy their readers' curiosity by generating a list anyway. Who's #1 this year? That school makes news, even though #2 may not only try harder, but be better for a large number of students.

That should not come as a surprise. For a democratic country with no aristocracy or royalty--and perhaps, ironically, because of that fact--we are obsessed with calculating everybody's relative place and most of all, with crowning a winner.

That's something to think about this New Year's while you watch three Big 10 teams compete in bowl games, but none in the Big 10's traditional post-season home, the Rose Bowl matchup with the Pacific 10 champion. Because of our "need" to crown a national football champion, Ohio State, the undefeated Big 10 champion, will play undefeated Miami in the Fiesta Bowl, while the Rose Bowl settles for also-rans Washington State and Oklahoma. Americans really care a lot about who's number one-- and not only in football.

Happy holidays and happy new year. Here's hoping you enjoyed this, one of my top 26 columns of 2002.