Category: General

In 2005 Jorge E. Hirsch published an article in the Proceedings of the National Academy of Science (link), proposing the “h-index”, a metric for the impact of an academic’s publications.

Your h-index is the largest number n such that you have n papers with n or more citations. So, for example, if you have 21 papers with 21 or more citations, but don’t yet have 22 papers with 22 or more citations, then your h-index is 21.

Hirsch claims that this measure is a better (or at least different) measure of impact than standard measures such as the total number of citations. He gives a number of apparently persuasive reasons why this might be the case.

In my opinion, for nearly all practical purposes, this claim is incorrect. In particular, and as I’ll explain below, you can to a good approximation work out the h-index as a simple function of the total number of citations, and so the h-index contains very little information beyond this already standard citation statistic.

Why am I mentioning this? Well, to my surprise the h-index is being taken very seriously by many people. A Google search shows the h-index is spreading and becoming very influential, very quickly. Standard citation services like the Web of Science now let you compute h-indices automatically. Promotion and grant evaluation committees are making use of them. And, of course, the h-index has been extensively discussed in the blogosphere (e.g., here, here, here, here, here, here, and here).

Hirsch focuses his discussion on physicists, and I’ll limit myself to that group, too; I expect the main conclusions to hold for other groups, with some minor changes in the constants. For the great majority of physicists (I’ll get to the exceptions), the h-index can be computed to a good approximation from the total number of citations they have received, as follows. Suppose T is the total number of citations. For most physicists, the following relationship holds to a very good approximation:

(*) h ~ sqrt(T)/2.

Thus, if someone has 400 citations, then their h-index is likely to be about half the square root of 400, which is 10. If someone has 2500 citations, then their h-index is likely to be about half the square root of 2500, which is 25.

The relationship (*) actually follows from the data Hirsch analysed. He notes in passing that he found empirically that T = a h^2, where a is between between 3 and 5. Inverting the relationship, we find that (*) holds to within an accuracy of about plus or minus 15%. That’s accurate enough – nobody cares whether your h-index is 20 or 23, particularly since citation statistics are already quite noisy. Provided a is in this range, h contains little additional information beyond T, which is already a commonly used citation statistic.

What about the exceptions to this rule? I believe there are two main sources of exception.

The first class of exception is people with very few papers. Someone with 1-4 papers can easily evade the rule, simply because their distribution of citations across papers may be very unusual. In practice, though, this doesn’t much matter, since in such cases it’s possible to look at a person’s entire record, and measures of aggregate performance are not used so much in these cases, anyway.

The second class of exceptions is people who have one work which is vastly more cited than any other work. In that case the formula (*) tends to overstate the h-index. The effect is much smaller than you might think, though, since it seems to be that for the great majority of physicists their top-cited publication has many more citations than their next-most cited publication.

In any case, I hypothesize that this effect is mostly corrected by using the formula:

(**) h approx b sqrt(T’)

where T’ is a the total number of citations, less the most cited publication, and b is a constant which needs to be empirically determined. At a guess I’d believe that omitting the top two cited publications would work even better, but after that we’d hit the point of diminishing returns.

Returning to the main point, my counter-claim to Hirsch is that the h-index contains very little additional information beyond the total number of citations. It’s not that the h-index is irrelevant, it’s just that in the great majority of cases the h-index is not very useful, given that the total number of citations is likely to already be known.

After 12 years of work on quantum information and quantum computation, I’ve decided to shift my creative work to a completely new direction.

I’m making this shift because I believe I can contribute more elsewhere.

I became interested in quantum information and computation in 1992, and started working fulltime on it in 1995. When I started it was a tiny little field with a handful of practitioners around the world. Most scientists hadn’t even heard of quantum computers. Those few who had would often use what they’d heard to pour cold water on the idea of ever being able to build one. Now, in 2007 the field is one of the hottest in physics, and many researchers, myself included, believe it is only a matter of time and concentrated effort before a large-scale quantum computer is built.

To me this seems a propitious time to change direction.

The new direction I’ll be working toward is the development of new tools for scientific collaboration and publication. This is a tremendously exciting area, and it’s also one where my skills and interests seem likely to be useful. I’m a beginner in the area, and so for the next few months, I’ll be doing a “reconnaissance in force”, orienting myself, figuring out what I need to learn, where I might be able to make a contribution, and launching some small projects. It ought to be a blast.

By guest blogger Robin Blume-Kohout

In 2005, Slate published twelve essays on “How to reinvent higher education”. The opening paragraphs of one, by Alison Gopnik, still burn in my mind:

I’m a cognitive scientist who is also a university professor. There is a staggering contrast between what I know about learning from the lab and the way I teach in the classroom. … I know that children, and even adults, learn about the everyday world around them in much the way that scientists learn. …Almost none of this happens in the average university classroom, including mine. In lecture classes, the teacher talks and the students write down what the teacher says. In seminars, the students write down what other students say. This is, literally, a medieval form of learning…

In short, we are screwing up — and we should know better.

Scientific publishing — the primary means by which we communicate with other scientists — is in the same boat:

1. We’re doing it badly,
2. Our methods are medieval,
3. We should know better.

Technically, point #2 is unfair. Scientific publishing dates from the 1660s, when Proceedings of the Royal Society emerged from Henry Oldenburg‘s voluminous scientific correspondence. If you wanted to show off your research in 1665, you wrote a letter to Henry. When he got it (a month or two later), he forwarded it to someone who could tell him whether it was any good. If the referee liked it, then (after a few more month-long postal delays), Henry read your letter out loud to the Royal Society, and it got recorded in the Proceedings.

These days, it’s quite different. Specifically:

1. We write letters in LaTeX, and email them,
2. There are so many journals that nobody reads most of them,
3. Henry doesn’t read your letter out loud.

The rest of the system is unchanged. This raises a bunch of questions, like “Why does publication take 6 months?”, “Why is it so expensive?”, and “Does anybody read journals, what with the arXiv?” I’m not going to discuss these questions, but if you’re interested, you might try the Wikipedia article on scientific journals. Which is a perfect example of why we should know better.

I’m not talking about the content. I’m talking about the article itself, and how I referenced it — with a hyperlink. I’ve given you incredible power. Quickly and easily, you can:

• Verify my sources,
• Find answers to questions I’ve raised — if you’re interested,
• Get more detailed explanations,
• Discover and explore related topics.

Enabling you this way is part of the core mission: The purpose of scientific communication is to educate, extensibly and efficiently. Education: After months of research, I publish a paper so that you can learn what I know — without all the hard work. Extensibility: I include proofs, arguments, figures, explanations, and citations — so that you can verify my work and place it in the context of prior work. Efficiency: Writing this way takes more months — but thousands of my colleagues can save months by reading my paper.

We are failing at efficiency, for Wikipedia illustrates a more efficient way of educating — or, if you prefer, a source for more efficient learning. I don’t mean that Wikipedia is The Answer. We need to build a new medium, replacing medieval features with the best features of Wikipedia. For instance,

• Hypertext revolutionizes scientific writing, by organizing content as a tree instead of a list. Articles and textbooks have a linear structure. To find a specific answer, I have to read (on average) half the text. In a hypertext environment like Wikipedia, I can search through a cluster of ideas for answers — even to questions I haven’t been able to formulate yet. Hyperlinking specifically enables…
• “Choose your own adventure” approaches to a body of work. Scientific papers represent a cluster of related ideas. Different readers, with different background knowledge, will benefit from different paths. A well-structured (and judiciously hyperlinked) electronic text can become the reader’s personalized guide. Parts of several such texts can be combined by a customized path, to form an entirely new text. This requires…
• Modular content, dividing a text into bite-sized chunks. Modularity also offers intrinsic benefits. One is reusability; a single explanation can be referenced in many contexts. Current scientific writing is necessarily terse. Hyperlinks and modularity allow the text to be larded with optional explanations, which clarify potential confusion without breaking the flow. Modularity also allows alternative approaches, providing the reader with multiple analyses of the same concept. Such alternatives are particularly useful when combined with…
• Distributed editing by a large community of contributors. This is a vast can of worms that I shan’t open here, but two things are clear. First, a forum for scientific communication cannot adopt Wikipedia’s “anyone can edit” motto. Second, the potential benefits of post-publication editing, combined with an unlimited pool of “editors”, are too great to ignore. Balancing these imperatives is an outstanding challenge, but a relatively uncontroversial technique is…
• Attached commentary, either critical or explanatory, by readers. Consider, for example, the Talmud, where post-publication analysis (the Gemara) attempts to clarify the original text (the Mishnah). More recently, commenting systems have proliferated on blogs and (with much, much less intellectual rigor) news-sites like Slashdot. In a scientific publishing context, commentary can
  • correct mistakes, either technical or factual, in the original text,
  • provide an alternative to a module that (the reader feels) could be improved,
  • critique and question the original work,
  • update older work in light of new research.

These points are not a prescription. They are a manifesto (“We can do better, see!”), and a plea (“Help make it better!”). Published scientific communications are the collective memory of scientists. If we cannot access it quickly and efficiently, we are effectively brain damaged. Improving our access makes us — quite simply — smarter. All we need to do is to use the computing tools before us intelligently.

We’ve taken first steps — the preprint arXiv, central repositories like PROLA, and online publishing by the likes of Nature. These are baby steps. We’re doing the same old thing a little better with new technology. Sooner or later, scientific communication is going to be restructured to really take advantage of what we can do now… and it’s going to make us (collectively) a lot smarter.

I can’t wait.

How to write consistently boring scientific literature, by Kaj Sand-Jensen

Although scientists typically insist that their research is very exciting and adventurous when they talk to laymen and prospective students, the allure of this enthusiasm is too often lost in the predictable, stilted structure and language of their scientific publications. I present here, a top-10 list of recommendations for how to write consistently boring scientific publications. I then discuss why we should and how we could make these contributions more accessible and exciting.

Sadly, this is hidden behind a publisher pay wall. I particularly enjoyed the opening quote:

“Hell – is sitting on a hot stone reading your own scientific publications”
– Erik Ursin, fish biologist

“The work reported in this paper is obvious, and wrong. Besides, I did it all 5 years ago, anyway.”

(I heard this from my PhD supervisor, Carl Caves, about 10 years ago. At the time, I thought it was funny…)

It is the greatest game in the history of chess. The sheer number of ideas, the complexity, and the contribution it has made to chess make it the most important game ever played.
-Garry Kasparov (World Chess Champion) in a Reuters interview conducted during his 1999 game against the World

In 1999, world chess champion Garry Kasparov, widely acknowledged as the greatest player in the history of the game, agreed to participate in a chess match sponsored by Microsoft, playing against “the World”. One move was to be made each 24 hours, with the World’s move being decided by a vote; anyone at all was allowed to vote on the World Team’s next move.

The game was staggering. After 62 moves of innovative chess, in which the balance of the game changed several times, the World Team finally resigned. Kasparov revealed that during the game he often couldn’t tell who was winning and who was losing, and that it wasn’t until after the 51st move that the balance swung decisively in his favour. After the game, Kasparov wrote an entire book about it. He claimed to have expended more energy on this one game than on any other in his career, including world championship games.

What is particularly amazing is that although the World Team had input from some very strong players, none were as strong as Kasparov himself, and the average quality was vastly below Kasparov’s level. Yet, collectively, the World Team produced a game far stronger than one might have expected from any of the individuals contributing, indeed, one of the strongest games ever played in history. Not only did they play Kasparov at his best, but much of the deliberation about World Team strategy and tactics was public, and so accessible to Kasparov, an advantage he used extensively. Imagine that not only are you playing Garry Kasparov at his best, but that you also have to explain in detail to Kasaparov all the thinking that goes into your moves!

How was this remarkable feat achieved?

It is worth noting that another “Grandmaster versus the world” game was played prior to this game, in which Grandmaster and former world champion Anatoly Karpov crushed the World Team. However, Kasparov versus the World used a very different system to co-ordinate the World Team’s efforts. Partially through design, and partially through good luck, this system enabled the World Team to co-ordinate their efforts far better than in the earlier game.

The basic idea used was that anyone in the world could register a vote for their preferred next move. The move taken was whichever garnered the most votes. Microsoft did not release detailed statistics, but claimed that on a typical move more than 5000 people voted. Furthermore, votes came from people at all levels of chess excellence, from chess grandmasters to rank amateurs. On one move, Microsoft reported that 2.4 percent of the votes were cast for moves that were not merely bad, but actually illegal! On other occasions moves regarded as obviously bad by experts obtained up to 10 percent of the vote. Over the course of the match, approximately 50,000 individuals from more than 75 countries participated in the voting.

Critical to the experiment were several co-ordinating devices that enabled the World Team to act more coherently.

An official game forum was set up by Microsoft so that people on the World Team could discuss and co-ordinate their ideas.

Microsoft appointed four official advisors to the World Team. These were outstanding teenage chess players, including two ranked as grandmasters, all amongst the best of their age in the world, although all were of substantially lower caliber than Kasparov. These four advisors agreed to provide advice to the World Team, and to make public recommendations on what move to take next.

In addition these formal avenues of advice, as the game progressed various groups around the world began to offer their own commentary and advice. Particuarly influential, although not always heeded, was the GM school, a strong Russian chess club containing several grandmasters.

Most of these experts ignored the discussion taking place on the game forum, and made no attempt to engage with the vast majority of people making up the World Team, i.e., the people whose votes would actually decide the World’s moves.

However, one of the World Team’s advisors did make an effort to engage the World Team. This was an extraordinary young chess player named Irina Krush. Fifteen years old, Krush had recently become the US Women’s chess champion. Although not as highly rated as two of the other World Team advisors, or as some of the grandmasters offering advice to the World Team, Krush was certainly in the international elite of junior chess players.

Unlike her expert peers, Krush focused considerable time and attention on the World Team’s game forum. Shrugging off flames and personal insults, she worked to extract the best ideas and analysis from the forum, as well as building up a network of strong chess-playing correspondents, including some of the grandmasters now offering advice.

Simultaneously, Krush built a publicly accessible analysis tree, showing possible moves and countermoves, and containing the best arguments and refutations for different lines of play, both from the game forum, and from her correspondence with others, including the GM school. This analysis tree enabled the World Team to focus its attention much more effectively, and served as a reference point for discussion, for further analysis, and for voting.

As the game went on, Krush’s role on the World Team gradually became more and more pivotal, despite the fact that according to their relative rankings, Kasparov would ordinarily have beaten Krush easily, unless he made a major blunder.

Part of the reason for this was the quality of Krush’s play. On move 10, Krush suggested a completely novel move that Kasparov called “A great move, an important contribution to chess”, and which all expert analysts agree blew the game wide open, taking it into uncharted chess territory. This raised her standing with the World Team, and helped her assume a coordinating role. Between moves 10 and 50 Krush’s recommended move was always played by the World Team, even when it disagreed with the recommendations of the other three advisors to the World Team, or with influential commentators such as the GM school.

As a result, some people have commented that the game was really Kasparov versus Krush, and Kasparov himself has claimed that he was really playing Smart Chess, Krush’s management team. Krush has repudiated this point of view, commenting on how important many other people’s input was to her recommendations. It seems likely that a more accurate picture is that Krush was at the center of the co-ordination effort for the World Team, and so had a better sense of the best overall recommendation made by the members of the World Team. Other, ostensibly stronger players weren’t as aware of all these different points of view, and so didn’t make as good decisions about what move to make next.

Krush’s coordinating role brought the best ideas of all contributors into a single coherent whole, weeding out bad moves from the good. As the game went on, much stronger players began to channel their ideas through her, including one of the strongest players from the GM school, Alexander Khalifman. The result was that the World Team emerged stronger than any individual player, indeed, arguably stronger than any player in history with the exception of Kasparov at his absolute peak, and with the advantage of being able to see the World “thinking” out loud as they deliberated the best course of action.

Kasparov versus the World is a fascinating case study in the power of collective collaboration. Most encouragingly for us, Kasparov versus the World provides convincing evidence that large groups of people acting in concert can solve creative problems well beyond the reach of any of them alone.

More practically, Kasparov versus the World suggests the value of providing centralized repositories of information which can serve as reference points for decision making and for the allocation of effort. Krush’s analysis tree was critical to the co-ordination of the World Team. It prevented duplication of effort on the part of the World Team, who didn’t have to chase down lines of play known to be poor, and acted as a reference point for discussion, for further analysis, and for voting.

Finally, Kasparov versus the World suggests the value of facilitators who act to channel community opinion. These people must have the respect of the community, but they need not be the strongest individual contributor. If such facilitators are flexible and responsive (without being submissive), they can co-ordinate and focus community opinion, and so build a whole stronger than any of its parts.

Further reading

This essay is an abridged extract from a book I’m writing about “The Future of Science”. If you’d like to be notified when the book is available, please send a blank email to
the.future.of.science@gmail.com with the subject “subscribe book”. I’ll email you to let you know in advance of publication. I will not use your email address for any other purpose, nor reshare it with anyone else!

Subscribe to my
blog here.

Anton Zeilinger now has a blog.

In the comments, Franklin writes on the subject of open source research:

On the other side of the coin, what would be the incentives for contributing to other people’s research?

This is an excellent question. Generalizing, any proposed change to how people do research, collaborate, or publish necessarily must face the question: what are the incentives to participate in the change? One must find a migration path which provides positive incentives at each step of the way, or else the migration is doomed to failure. I am proposing very significant changes to how research is done, and so the incentives along the migration path necessarily require considerable thought. Addressing these issues systematically is one of the main reasons I’ve written a book.

Commenter Martin points to the January 2007 issue of Physics World, which contains lot of very interesting information about Web 2.0 and Science. In a similar vein, Corie Lok has some thoughtful recent reflections on getting scientists to adopt new tools for research. Finally, let me mention Jon Udell’s interview with Lewis Shepherd, talking about the US Defense Intelligence Agency’s use of wikis, blogs, Intellipedia, and many other interesting things. Some of the challenges he faced in bringing such social tools to Defense are similar to the problems in bringing them to science.

On a completely different topic, let me mention a fantastic presentation about green technology given earlier this year by John Doerr at the TED conference. I’ve been working my way through all the online TED talks, many of which are really good. While I’m at it, I may as well plug the Long Now talks, which is also a great series, with talks by people like Danny Hillis, John Baez, Stewart Brand, Jimmy Wales and many others.