Academics strike back at spurious rankings

Academics strike back at spurious rankings
D Butler, Nature 447, 514-515 (31 May 2007) doi:10.1038/447514b

Below are excerpts from the Nature article, followed by some
references:
Universities seek reform of ratings.

[A] group of US colleges [called for a] boycott [of] the most influential university ranking in the United States...
Experts argue that these are based on dubious methodology and spurious data, yet they have huge influence...

"All current university rankings are flawed to some extent; most, fundamentally,"

The rankings in the U.S. News & World Report and those published by the British Times Higher Education Supplement (THES) depend heavily on surveys of thousands of experts - a system that some contest. A third popular ranking, by Jiao Tong University in Shanghai, China, is based on more quantitative measures, such as citations, numbers of Nobel prizewinners and publications in Nature and Science. But even these measures are not straightforward.

Thomson Scientific's ISI citation data are notoriously poor for use in rankings; names of institutions are spelled differently from one article to the next, and university affiliations are sometimes omitted altogether. After cleaning up ISI data on all UK papers for such effects... the true number of papers from the University of Oxford, for example, [were] 40% higher than listed by ISI...

Researchers at Leiden University in the Netherlands have similarly recompiled the ISI database for 400 universities: half a million papers per year. Their system produces various rankings based on different indicators. One, for example, weights citations on the basis of their scientific field, so that a university that does well
in a heavily cited field doesn't get an artificial extra boost.

The German Center for Higher Education Development (CHE) also offers rankings... for almost 300 German, Austrian and Swiss universities... the CHE is expanding the system to cover all Europe.

The US Commission on the Future of Higher Education is considering creating a similar public database, which would offer competition to the U.S. News & World Report.

---------------------------------------------------------------------------
Bollen, Johan and Herbert Van de Sompel. Mapping the structure of science through usage. Scientometrics, 69(2), 2006
http://dx.doi.org/10.1007/s11192-006-0151-8

Hardy, R., Oppenheim, C., Brody, T. and Hitchcock, S. (2005) Open Access Citation Information. http://eprints.ecs.soton.ac.uk/11536/

Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35.
http://www.ecs.soton.ac.uk/~harnad/Temp/Ariadne-RAE.htm

Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects,
chapter 21. Chandos. http://eprints.ecs.soton.ac.uk/12453/

Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. Invited Keynote, 11th Annual Meeting of the International Society for Scientometrics and Informetrics. Madrid,
Spain, 25 June 2007 http://arxiv.org/abs/cs.IR/0703131

Kousha, Kayvan and Thelwall, Mike (2006) Google Scholar Citations and Google Web/URL Citations: A Multi-Discipline Exploratory Analysis. In Proceedings International Workshop on Webometrics, Informetrics and Scientometrics & Seventh COLLNET Meeting, Nancy (France).
http://eprints.rclis.org/archive/00006416/

Moed, H.F. (2005). Citation Analysis in Research Evaluation. Dordrecht (Netherlands): Springer.

van Raan, A. (2007) Bibliometric statistical properties of the 100 largest European universities: prevalent scaling rules in the science system. Journal of the American Society for Information Science and Technology http://www.cwts.nl/Cwts/Stat4AX-JASIST.pdf

Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
To join or leave the Forum or change your subscription address:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html