Wednesday, June 23, 2010

Skewing the results - heavily

With thousands of scientific journals being published and more arriving each year, research journals constantly struggle to make sure that they are considered as important – filled with noteworthy research and a place in which other researchers want their research published. One way to quantify this nebulous quality is with “Impact Factors”, which is the average number of times a journal’s articles have been cited in the last two years, divided by the number of articles the journal published.

Any attempt at quantification is going to have problems because you are trying to measure something that is vague. The Wikipedia link above has a few of the standard criticisms aimed at impact factors:

• It can't be compared across different disciplines as citation behavior varies greatly between disciplines.
• The average number of citations per paper is not a normal distribution, but is instead a Bradford distribution, meaning that the arithmetic mean is not a valid measure.
• Authors can self-cite their work and these citations are included in the impact factor.

One big criticism was missed however: the impact of one article can completely skew the results. This is seen in a discussion (courtesy of "The Scientist") on new impact factors just released.

“Specifically, the publication with second highest impact factor in the "science" category is Acta Crystallographica - Section A, knocking none other than the New England Journal of Medicine from the runner's up position. This title's impact factor rocketed up to 49.926 this year, more than 20-fold higher than last year. A single article published in a 2008 issue of the journal seems to be responsible for the meteoric rise... "A short history of SHELX," by University of Göttingen crystallographer George Sheldrick, which reviewed the development of the computer system SHELX, has been cited more than 6,600 times, according to ISI. This paper includes a sentence that essentially instructs readers to cite the paper they're reading -- "This paper could serve as a general literature citation when one or more of the open-source SHELX programs (and the Bruker AXS version SHELXTL) are employed in the course of a crystal-structure determination." (Note: This may be a good way to boost your citations.)”

6600 citations! Ignore what that does to the impact factor; that is an incredible, higher than most people get in their entire careers.

But fear not, the result is temporary for the journal.

“"Without another, similarly important article in 2010, Acta Crystallographica - Section A is likely to return in 2011 to its prior Journal Impact Factor of between 1.5 and 2.5,"”

If this anomaly become more widely known, I have to believe that it will really hurt the impact of the impact factor.

1 comment:

Anonymous said...

What do you mean?