Is scientific publishing about to be disrupted?

Part I: How Industries Fail

Until three years ago, the oldest company in the world was the construction company Kongo Gumi, headquartered in Osaka, Japan. Kongo Gumi was founded in 578 CE when the then-regent of Japan, Prince Shotoku, brought a member of the Kongo family from Korea to Japan to help construct the first Buddhist temple in Japan, the Shitenno-ji. The Kongo Gumi continued in the construction trade for almost one and a half thousand years. In 2005, they were headed by Masakazu Kongo, the 40th of his family to head Kongo Gumi. The company had more than 100 employees, and 70 million dollars in revenue. But in 2006, Kongo Gumi went into liquidation, and its assets were purchased by Takamatsu Corporation. Kongo Gumi as an independent entity no longer exists.

How is it that large, powerful organizations, with access to vast sums of money, and many talented, hardworking people, can simply disappear? Examples abound – consider General Motors, Lehman Brothers and MCI Worldcom – but the question is most fascinating when it is not just a single company that goes bankrupt, but rather an entire industry is disrupted. In the 1970s, for example, some of the world’s fastest-growing companies were companies like Digital Equipment Corporation, Data General and Prime. They made minicomputers like the legendary PDP-11. None of these companies exist today. A similar disruption is happening now in many media industries. CD sales peaked in 2000, shortly after Napster started, and have declined almost 30 percent since. Newspaper advertising revenue in the United States has declined 30 percent in the last 3 years, and the decline is accelerating: one third of that fall came in the last quarter.

There are two common explanations for the disruption of industries like minicomputers, music, and newspapers. The first explanation is essentially that the people in charge of the failing industries are stupid. How else could it be, the argument goes, that those enormous companies, with all that money and expertise, failed to see that services like iTunes and Last.fm are the wave of the future? Why did they not pre-empt those services by creating similar products of their own? Polite critics phrase their explanations less bluntly, but nonetheless many explanations boil down to a presumption of stupidity. The second common explanation for the failure of an entire industry is that the people in charge are malevolent. In that explanation, evil record company and newspaper executives have been screwing over their customers for years, simply to preserve a status quo that they personally find comfortable.

It’s true that stupidity and malevolence do sometimes play a role in the disruption of industries. But in the first part of this essay I’ll argue that even smart and good organizations can fail in the face of disruptive change, and that there are common underlying structural reasons why that’s the case. That’s a much scarier story. If you think the newspapers and record companies are stupid or malevolent, then you can reassure yourself that provided you’re smart and good, you don’t have anything to worry about. But if disruption can destroy even the smart and the good, then it can destroy anybody. In the second part of the essay, I’ll argue that scientific publishing is in the early days of a major disruption, with similar underlying causes, and will change radically over the next few years.

Why online news is killing the newspapers

To make our discussion of disruption concrete, let’s think about why many blogs are thriving financially, while the newspapers are dying. This subject has been discussed extensively in many recent articles, but my discussion is different because it focuses on identifying general structural features that don’t just explain the disruption of newspapers, but can also help explain other disruptions, like the collapse of the minicomputer and music industries, and the impending disruption of scientific publishing.

Some people explain the slow death of newspapers by saying that blogs and other online sources [1] are news parasites, feeding off the original reporting done by the newspapers. That’s false. While it’s true that many blogs don’t do original reporting, it’s equally true that many of the top blogs do excellent original reporting. A good example is the popular technology blog TechCrunch, by most measures one of the top 100 blogs in the world. Started by Michael Arrington in 2005, TechCrunch has rapidly grown, and now employs a large staff. Part of the reason it’s grown is because TechCrunch’s reporting is some of the best in the technology industry, comparable to, say, the technology reporting in the New York Times. Yet whereas the New York Times is wilting financially [2], TechCrunch is thriving, because TechCrunch’s operating costs are far lower, per word, than the New York Times. The result is that not only is the audience for technology news moving away from the technology section of newspapers and toward blogs like TechCrunch, the blogs can undercut the newspaper’s advertising rates. This depresses the price of advertising and causes the advertisers to move away from the newspapers.

Unfortunately for the newspapers, there’s little they can do to make themselves cheaper to run. To see why that is, let’s zoom in on just one aspect of newspapers: photography. If you’ve ever been interviewed for a story in the newspaper, chances are a photographer accompanied the reporter. You get interviewed, the photographer takes some snaps, and the photo may or may not show up in the paper. Between the money paid to the photographer and all the other costs, that photo probably costs the newspaper on the order of a few hundred dollars [3]. When TechCrunch or a similar blog needs a photo for a post, they’ll use a stock photo, or ask their subject to send them a snap, or whatever. The average cost is probably tens of dollars. Voila! An order of magnitude or more decrease in costs for the photo.

Here’s the kicker. TechCrunch isn’t being any smarter than the newspapers. It’s not as though no-one at the newspapers ever thought “Hey, why don’t we ask interviewees to send us a polaroid, and save some money?” Newspapers employ photographers for an excellent business reason: good quality photography is a distinguishing feature that can help establish a superior newspaper brand. For a high-end paper, it’s probably historically been worth millions of dollars to get stunning, Pulitzer Prizewinning photography. It makes complete business sense to spend a few hundred dollars per photo.

What can you do, as a newspaper editor? You could fire your staff photographers. But if you do that, you’ll destroy the morale not just of the photographers, but of all your staff. You’ll stir up the Unions. You’ll give a competitive advantage to your newspaper competitors. And, at the end of the day, you’ll still be paying far more per word for news than TechCrunch, and the quality of your product will be no more competitive.

The problem is that your newspaper has an organizational architecture which is, to use the physicists’ phrase, a local optimum. Relatively small changes to that architecture – like firing your photographers – don’t make your situation better, they make it worse. So you’re stuck gazing over at TechCrunch, who is at an even better local optimum, a local optimum that could not have existed twenty years ago:


local_optimum.jpg

Unfortunately for you, there’s no way you can get to that new optimum without attempting passage through a deep and unfriendly valley. The incremental actions needed to get there would be hell on the newspaper. There’s a good chance they’d lead the Board to fire you.

The result is that the newspapers are locked into producing a product that’s of comparable quality (from an advertiser’s point of view) to the top blogs, but at far greater cost. And yet all their decisions – like the decision to spend a lot on photography – are entirely sensible business decisions. Even if they’re smart and good, they’re caught on the horns of a cruel dilemma.

The same basic story can be told about the dispruption of the music industry, the minicomputer industry, and many other disruptions. Each industry has (or had) a standard organizational architecture. That organizational architecture is close to optimal, in the sense that small changes mostly make things worse, not better. Everyone in the industry uses some close variant of that architecture. Then a new technology emerges and creates the possibility for a radically different organizational architecture, using an entirely different combination of skills and relationships. The only way to get from one organizational architecture to the other is to make drastic, painful changes. The money and power that come from commitment to an existing organizational architecture actually place incumbents at a disadvantage, locking them in. It’s easier and more effective to start over, from scratch.

Organizational immune systems

I’ve described why it’s hard for incumbent organizations in a disrupted industry to change to a new model. The situation is even worse than I’ve described so far, though, because some of the forces preventing change are strongest in the best run organizations. The reason is that those organizations are large, complex structures, and to survive and prosper they must contain a sort of organizational immune system dedicated to preserving that structure. If they didn’t have such an immune system, they’d fall apart in the ordinary course of events. Most of the time the immune system is a good thing, a way of preserving what’s good about an organization, and at the same time allowing healthy gradual change. But when an organization needs catastrophic gut-wrenching change to stay alive, the immune system becomes a liability.

To see how such an immune system expresses itself, imagine someone at the New York Times had tried to start a service like Google News, prior to Google News. Even before the product launched they would have been constantly attacked from within the organization for promoting competitors’ products. They would likely have been forced to water down and distort the service, probably to the point where it was nearly useless for potential customers. And even if they’d managed to win the internal fight and launched a product that wasn’t watered down, they would then have been attacked viciously by the New York Times’ competitors, who would suspect a ploy to steal business. Only someone outside the industry could have launched a service like Google News.

Another example of the immune response is all the recent news pieces lamenting the death of newspapers. Here’s one such piece, from the Editor of the New York Times’ editorial page, Andrew Rosenthal:

There’s a great deal of good commentary out there on the Web, as you say. Frankly, I think it is the task of bloggers to catch up to us, not the other way around… Our board is staffed with people with a wide and deep range of knowledge on many subjects. Phil Boffey, for example, has decades of science and medical writing under his belt and often writes on those issues for us… Here’s one way to look at it: If the Times editorial board were a single person, he or she would have six Pulitzer prizes…

This is a classic immune response. It demonstrates a deep commitment to high-quality journalism, and the other values that have made the New York Times great. In ordinary times this kind of commitment to values would be a sign of strength. The problem is that as good as Phil Boffey might be, I prefer the combined talents of Fields medallist Terry Tao, Nobel prize winner Carl Wieman, MacArthur Fellow Luis von Ahn, acclaimed science writer Carl Zimmer, and thousands of others. The blogosophere has at least four Fields medallists (the Nobel of math), three Nobelists, and many more luminaries. The New York Times can keep its Pulitzer Prizes. Other lamentations about the death of newspapers show similar signs of being an immune response. These people aren’t stupid or malevolent. They’re the best people in the business, people who are smart, good at their jobs, and well-intentioned. They are, in short, the people who have most strongly internalized the values, norms and collective knowledge of their industry, and thus have the strongest immune response. That’s why the last people to know an industry is dead are the people in it. I wonder if Andrew Rosenthal and his colleagues understand that someone equipped with an RSS reader can assemble a set of news feeds that renders the New York Times virtually irrelevant? If a person inside an industry needs to frequently explain why it’s not dead, they’re almost certainly wrong.

What are the signs of impending disruption?

Five years ago, most newspaper editors would have laughed at the idea that blogs might one day offer serious competition. The minicomputer companies laughed at the early personal computers. New technologies often don’t look very good in their early stages, and that means a straightup comparison of new to old is little help in recognizing impending dispruption. That’s a problem, though, because the best time to recognize disruption is in its early stages. The journalists and newspaper editors who’ve only recognized their problems in the last three to four years are sunk. They needed to recognize the impending disruption back before blogs looked like serious competitors, when evaluated in conventional terms.

An early sign of impending disruption is when there’s a sudden flourishing of startup organizations serving an overlapping customer need (say, news), but whose organizational architecture is radically different to the conventional approach. That means many people outside the old industry (and thus not suffering from the blinders of an immune response) are willing to bet large sums of their own money on a new way of doing things. That’s exactly what we saw in the period 2000-2005, with organizations like Slashdot, Digg, Fark, Reddit, Talking Points Memo, and many others. Most such startups die. That’s okay: it’s how the new industry learns what organizational architectures work, and what don’t. But if even a few of the startups do okay, then the old players are in trouble, because the startups have far more room for improvement.

Part II: Is scientific publishing about to be disrupted?

What’s all this got to do with scientific publishing? Today, scientific publishers are production companies, specializing in services like editorial, copyediting, and, in some cases, sales and marketing. My claim is that in ten to twenty years, scientific publishers will be technology companies [4]. By this, I don’t just mean that they’ll be heavy users of technology, or employ a large IT staff. I mean they’ll be technology-driven companies in a similar way to, say, Google or Apple. That is, their foundation will be technological innovation, and most key decision-makers will be people with deep technological expertise. Those publishers that don’t become technology driven will die off.

Predictions that scientific publishing is about to be disrupted are not new. In the late 1990s, many people speculated that the publishers might be in trouble, as free online preprint servers became increasingly popular in parts of science like physics. Surely, the argument went, the widespread use of preprints meant that the need for journals would diminish. But so far, that hasn’t happened. Why it hasn’t happened is a fascinating story, which I’ve discussed in part elsewhere, and I won’t repeat that discussion here.

What I will do instead is draw your attention to a striking difference between today’s scientific publishing landscape, and the landscape of ten years ago. What’s new today is the flourishing of an ecosystem of startups that are experimenting with new ways of communicating research, some radically different to conventional journals. Consider Chemspider, the excellent online database of more than 20 million molecules, recently acquired by the Royal Society of Chemistry. Consider Mendeley, a platform for managing, filtering and searching scientific papers, with backing from some of the people involved in Last.fm and Skype. Or consider startups like SciVee (YouTube for scientists), the Public Library of Science, the Journal of Visualized Experiments, vibrant community sites like OpenWetWare and the Alzheimer Research Forum, and dozens more. And then there are companies like WordPress, Friendfeed, and Wikimedia, that weren’t started with science in mind, but which are increasingly helping scientists communicate their research. This flourishing ecosystem is not too dissimilar from the sudden flourishing of online news services we saw over the period 2000 to 2005.

Let’s look up close at one element of this flourishing ecosystem: the gradual rise of science blogs as a serious medium for research. It’s easy to miss the impact of blogs on research, because most science blogs focus on outreach. But more and more blogs contain high quality research content. Look at Terry Tao’s wonderful series of posts explaining one of the biggest breakthroughs in recent mathematical history, the proof of the Poincare conjecture. Or Tim Gowers recent experiment in “massively collaborative mathematics”, using open source principles to successfully attack a significant mathematical problem. Or Richard Lipton’s excellent series of posts exploring his ideas for solving a major problem in computer science, namely, finding a fast algorithm for factoring large numbers. Scientific publishers should be terrified that some of the world’s best scientists, people at or near their research peak, people whose time is at a premium, are spending hundreds of hours each year creating original research content for their blogs, content that in many cases would be difficult or impossible to publish in a conventional journal. What we’re seeing here is a spectacular expansion in the range of the blog medium. By comparison, the journals are standing still.

This flourishing ecosystem of startups is just one sign that scientific publishing is moving from being a production industry to a technology industry. A second sign of this move is that the nature of information is changing. Until the late 20th century, information was a static entity. The natural way for publishers in all media to add value was through production and distribution, and so they employed people skilled in those tasks, and in supporting tasks like sales and marketing. But the cost of distributing information has now dropped almost to zero, and production and content costs have also dropped radically [5]. At the same time, the world’s information is now rapidly being put into a single, active network, where it can wake up and come alive. The result is that the people who add the most value to information are no longer the people who do production and distribution. Instead, it’s the technology people, the programmers.

If you doubt this, look at where the profits are migrating in other media industries. In music, they’re migrating to organizations like Apple. In books, they’re migrating to organizations like Amazon, with the Kindle. In many other areas of media, they’re migrating to Google: Google is becoming the world’s largest media company. They don’t describe themselves that way (see also here), but the media industry’s profits are certainly moving to Google. All these organizations are run by people with deep technical expertise. How many scientific publishers are run by people who know the difference between an INNER JOIN and an OUTER JOIN? Or who know what an A/B test is? Or who know how to set up a Hadoop cluster? Without technical knowledge of this type it’s impossible to run a technology-driven organization. How many scientific publishers are as knowledgeable about technology as Steve Jobs, Sergey Brin, or Larry Page?

I expect few scientific publishers will believe and act on predictions of disruption. One common response to such predictions is the appealing game of comparison: “but we’re better than blogs / wikis / PLoS One / …!” These statements are currently true, at least when judged according to the conventional values of scientific publishing. But they’re as irrelevant as the equally true analogous statements were for newspapers. It’s also easy to vent standard immune responses: “but what about peer review”, “what about quality control”, “how will scientists know what to read”. These questions express important values, but to get hung up on them suggests a lack of imagination much like Andrew Rosenthal’s defense of the New York Times editorial page. (I sometimes wonder how many journal editors still use Yahoo!’s human curated topic directory instead of Google?) In conversations with editors I repeatedly encounter the same pattern: “But idea X won’t work / shouldn’t be allowed / is bad because of Y.” Well, okay. So what? If you’re right, you’ll be intellectually vindicated, and can take a bow. If you’re wrong, your company may not exist in ten years. Whether you’re right or not is not the point. When new technologies are being developed, the organizations that win are those that aggressively take risks, put visionary technologists in key decision-making positions, attain a deep organizational mastery of the relevant technologies, and, in most cases, make a lot of mistakes. Being wrong is a feature, not a bug, if it helps you evolve a model that works: you start out with an idea that’s just plain wrong, but that contains the seed of a better idea. You improve it, and you’re only somewhat wrong. You improve it again, and you end up the only game in town. Unfortunately, few scientific publishers are attempting to become technology-driven in this way. The only major examples I know of are Nature Publishing Group (with Nature.com) and the Public Library of Science. Many other publishers are experimenting with technology, but those experiments remain under the control of people whose core expertise is in others areas.

Opportunities

So far this essay has focused on the existing scientific publishers, and it’s been rather pessimistic. But of course that pessimism is just a tiny part of an exciting story about the opportunities we have to develop new ways of structuring and communicating scientific information. These opportunities can still be grasped by scientific publishers who are willing to let go and become technology-driven, even when that threatens to extinguish their old way of doing things. And, as we’ve seen, these opportunites are and will be grasped by bold entrepreneurs. Here’s a list of services I expect to see developed over the next few years. A few of these ideas are already under development, mostly by startups, but have yet to reach the quality level needed to become ubiquitous. The list could easily be continued ad nauseum – these are just a few of the more obvious things to do.

Personalized paper recommendations: Amazon.com has had this for books since the late 1990s. You go to the site and rate your favourite books. The system identifies people with similar taste, and automatically constructs a list of recommendations for you. This is not difficult to do: Amazon has published an early variant of its algorithm, and there’s an entire ecosystem of work, much of it public, stimulated by the Neflix Prize for movie recommendations. If you look in the original Google PageRank paper, you’ll discover that the paper describes a personalized version of PageRank, which can be used to build a personalized search and recommendation system. Google doesn’t actually use the personalized algorithm, because it’s far more computationally intensive than ordinary PageRank, and even for Google it’s hard to scale to tens of billions of webpages. But if all you’re trying to rank is (say) the physics literature – a few million papers – then it turns out that with a little ingenuity you can implement personalized PageRank on a small cluster of computers. It’s possible this can be used to build a system even better than Amazon or Netflix.

A great search engine for science: ISI’s Web of Knowledge, Elsevier’s Scopus and Google Scholar are remarkable tools, but there’s still huge scope to extend and improve scientific search engines [6]. With a few exceptions, they don’t do even basic things like automatic spelling correction, good relevancy ranking of papers (preferably personalized), automated translation, or decent alerting services. They certainly don’t do more advanced things, like providing social features, or strong automated tools for data mining. Why not have a public API [7] so people can build their own applications to extract value out of the scientific literature? Imagine using techniques from machine learning to automatically identify underappreciated papers, or to identify emerging areas of study.

High-quality tools for real-time collaboration by scientists: Look at services like the collaborative editor Etherpad, which lets multiple people edit a document, in real time, through the browser. They’re even developing a feature allowing you to play back the editing process. Or the similar service from Google, Google Docs, which also offers shared spreadsheets and presentations. Look at social version control systems like Git and Github. Or visualization tools which let you track different people’s contributions. These are just a few of hundreds of general purpose collaborative tools that are lightyears beyond what scientists use. They’re not widely adopted by scientists yet, in part for superficial reasons: they don’t integrate with things like LaTeX and standard bibliographical tools. Yet achieving that kind of integration is trivial compared with the problems these tools do solve. Looking beyond, services like Google Wave may be a platform for startups to build a suite of collaboration clients that every scientist in the world will eventually use.

Scientific blogging and wiki platforms: With the exception of Nature Publishing Group, why aren’t the scientific publishers developing high-quality scientific blogging and wiki platforms? It would be easy to build upon the open source WordPress platform, for example, setting up a hosting service that makes it easy for scientists to set up a blog, and adds important features not present in a standard WordPress installation, like reliable signing of posts, timestamping, human-readable URLs, and support for multiple post versions, with the ability to see (and cite) a full revision history. A commenter-identity system could be created that enabled filtering and aggregation of comments. Perhaps most importantly, blog posts could be made fully citable.

On a related note, publishers could also help preserve some of the important work now being done on scientific blogs and wikis. Projects like Tim Gowers’ Polymath Project are an important part of the scientific record, but where is the record of work going to be stored in 10 or 20 years time? The US Library of Congress has taken the initiative in preserving law blogs. Someone needs to step up and do the same for science blogs.

The data web: Where are the services making it as simple and easy for scientists to publish data as it to publish a journal paper or start a blog? A few scientific publishers are taking steps in this direction. But it’s not enough to just dump data on the web. It needs to be organized and searchable, so people can find and use it. The data needs to be linked, as the utility of data sets grows in proportion to the connections between them. It needs to be citable. And there needs to be simple, easy-to-use infrastructure and expertise to extract value from that data. On every single one of these issues, publishers are at risk of being leapfrogged by companies like Metaweb, who are building platforms for the data web.

Why many services will fail: Many unsuccessful attempts at implementing services like those I’ve just described have been made. I’ve had journal editors explain to me that this shows there is no need for such services. I think in many cases there’s a much simpler explanation: poor execution [8]. Development projects are often led by senior editors or senior scientists whose hands-on technical knowledge is minimal, and whose day-to-day involvement is sporadic. Implementation is instead delegated to IT-underlings with little power. It should surprise no one that the results are often mediocre. Developing high-quality web services requires deep knowledge and drive. The people who succeed at doing it are usually brilliant and deeply technically knowledgeable. Yet it’s surprisingly common to find projects being led by senior scientists or senior editors whose main claim to “expertise” is that they wrote a few programs while a grad student or postdoc, and who now think they can get a high-quality result with minimal extra technical knowledge. That’s not what it means to be technology-driven.

Conclusion: I’ve presented a pessimistic view of the future of current scientific publishers. Yet I hope it’s also clear that there are enormous opportunities to innovate, for those willing to master new technologies, and to experiment boldly with new ways of doing things. The result will be a great wave of innovation that changes not just how scientific discoveries are communicated, but also accelerates the way scientific discoveries are made.

Notes

[1] We’ll focus on blogs to make the discussion concrete, but in fact many new forms of media are contributing to the newspapers’ decline, including news sites like Digg and MetaFilter, analysis sites like Stratfor, and many others. When I write “blogs” in what follows I’m usually referring to this larger class of disruptive new media, not literally to conventional blogs, per se.

[2] In a way, it’s ironic that I use the New York Times as an example. Although the New York Times is certainly going to have a lot of trouble over the next five years, in the long run I think they are one of the newspapers most likely to survive: they produce high-quality original content, show strong signs of becoming technology driven, and are experimenting boldly with alternate sources of content. But they need to survive the great newspaper die-off that’s coming over the next five or so years.

[3] In an earlier version of this essay I used the figure 1,000 dollars. That was sloppy – it’s certainly too high. The actual figure will certainly vary quite a lot from paper to paper, but for a major newspaper in a big city I think on the order of 200-300 dollars is a reasonable estimate, when all costs are factored in.

[4] I’ll use the term “companies” to include for-profit and not-for-profit organizations, as well as other organizational forms. Note that the physics preprint arXiv is arguably the most successful publisher in physics, yet is neither a conventional for-profit or not-for-profit organization.

[5] This drop in production and distribution costs is directly related to the current move toward open access publication of scientific papers. This movement is one of the first visible symptoms of the disruption of scientific publishing. Much more can and has been said about the impact of open access on publishing; rather than review that material, I refer you to the blog “Open Access News”, and in particular to Peter Suber’s overview of open access.

[6] In the first version of this essay I wrote that the existing services were “mediocre”. That’s wrong, and unfair: they’re very useful services. But there’s a lot of scope for improvement.

[7] After posting this essay, Christina Pikas pointed out that Web of Science and Scopus do have APIs. That’s my mistake, and something I didn’t know.

[8] There are also services where the primary problem is cultural barriers. But for the ideas I’ve described cultural barriers are only a small part of the problem.

Acknowledgments: Thanks to Jen Dodd and Ilya Grigorik for many enlightening discussions.

About this essay: This essay is based on a colloquium given June 11, 2009, at the American Physical Society Editorial Offices. Many thanks to the people at the APS for being great hosts, and for many stimulating conversations.

Further reading:

Some of the ideas explored in this essay are developed at greater length in my book Reinventing Discovery: The New Era of Networked Science.

You can subscribe to my blog here.

My account of how industries fail was influenced by and complements Clayton Christensen’s book “The Innovator’s Dilemma”. Three of my favourite blogs about the future of scientific communication are “Science in the Open”, “Open Access News” and “Common Knowledge”. Of course, there are many more excellent sources of information on this topic. A good source aggregating these many sources is the Science 2.0 room on FriendFeed.

200 comments

  1. Yup, well-written. Archiving science blogs would be especially nice!

    Note also that blogs have sometimes backreation into the traditionnal publishing world, for instance Terence Tao turns each year of his blog into an AMS book, and there are probably many more examples.

    As for peer-review, I’m dreaming of a way to make all the interactions between authors, reviewers and paper citations visible (some kind of customizable graph). This would allow to easily localize solid new trends, as well as low-quality researchers (the ones which self-cite a lot without innovation and get reviewed by friends just to add a line in their publication list).

  2. Exceptionally insightful essay. You present the challenges faced by scientific publishing persuasively.

    What is going to be interesting is seeing whether the scientific community can adapt and develop strategies to embrace the opportunities you outline. Perhaps one of the greatest dangers of limited proactive action is effective peer to peer science communication and archiving falling between the gaps – the current local optimum being undermined before a new one – that serves the community equally well or better – is established.

    (I’m cynically assuming of course that social/market forces could lead to a new paradigm that kills off mainstream publishing without leading to an equivalent or superior alternative)

  3. Nice essay – and lots of links & references I’d like to check out.

    I agree that the potential of the web for improving communication/collaboration is huge and largely untapped. I work on several large international projects and it’s frustrating how unwilling people are to use new technologies – even as simple as obviously useful as a wiki. I guess for the generation of scientists that are currently “in charge” blogs, wikis, social networking platforms, are considered new and … frivolous?

  4. Absolutely Right. Well deserved of the Techcrunch RT and post.

    I dont think it will take 10 years to supplant academic journals though. All that is missing, is a means to replicate the reputation building and content filtering for “quality” that journals currently do better than online, open-source means. Blogs have already done this to print media; how long will it take to do the same to journals?

  5. There is no metric by which a single photograph in a newspaper would cost a thousand dollars. Most photogs will shoot hundreds of frames per week and are paid salaries to do so. Let’s say, for the sake of argument, that a photog is paid $60g per year. In order for the paper to spend a grand per photo, that photog would be shooting less than twice a week. As a newsroom manager, I assure you it doesn’t work that way. At least not in 99.5% of all papers.

  6. @Cmaury

    Don’t underestimate the inertia. Some fields are moving quickly, but most of my colleagues in the social sciences do not even know yet that they can download research papers electronically. They are still vastly reliant on a paper-based culture.

    We have come a long, long way in the last ten years. It is now “accepted” to blog as a professor, but it is hardly common. I work for a large university with over 1,000 professors. It is a bona fide research university too. How many professors have serious and long-standing blogs? Stevan Harnad has one (http://openaccess.eprints.org/). Seb Paquet also (http://openresearch.sebpaquet.net/). Short of that… I can’t think of anyone. As a blogger, I still get the usual “I don’t have time to blog” (meaning: it is a frivolous activity) from older professors. Thankfully, people no longer mock me, but if I were to state that blogging counts as “academic publishing” they would mock me.

    There are good and bad reasons why academia moves slowly, but it is undeniable that it does. It has a very, very strong immune system.

  7. A great post!

    Another thought I would like to share is that I see there being a potential difference between two attitudes of big companies and why they can be overthrown by a market disruption:

    On the one hand, some companies might be purely “ignoring” new ideas, similar to what Andrew Rosenthal’s comment further above might suggest, or similar to having the opinion “but we are better than X”. These companies might fail because they just ignore innovative ideas or start-ups. The problem with this setup is that in the end there can only be one winner: either it’s the start-up, or the incumbent firm.

    On the other hand, I believe there are also big companies which actually don’t ignore market changes, but which just have not found a way to evolve quickly enough in order to keep up with a changing market. These companies are looking for innovative solutions, but could still fail because they are not able to change with market conditions due to the reasons you mentioned above, or because they haven’t found a strong partner in a quickly evolving environment. Since they are willing to discuss and change, however, such a competitive setup can actually create a win-win situation for both the start-up and the incumbent firm.

    Luckily, at Mendeley, so far we’ve mostly encountered companies of the second type (probably because the companies of the first type just ignore us…)., and I feel that start-ups could also help big companies to find ways of how to create a business model in a changing competitive environment, and therefore create a win-win situation. It’s not that we would bet on this, but it shows additional potential of what can be done.

  8. I found your article to be of immense interest. I do believe all industries must have a look at their senior staff and heavy weight them with technical backgrounds in some manner. Those who move quickest to do so will lead market share in their respective industry.

    The world markets are competitive to those who are forward thinking, reach mass markets in a timely manner, control costs, and can reinvent themselves.

    Those who do will thrive. Technology and the ability to understand the business and social implications of the web is a job in and of itself. Make sure you have a technical researcher on your staff.

    Scientific Papers are the issue of this today but technology is all encompassing, no one, nor will any industry remain untouched from it’s inclusion on everyday life.

  9. Great post – just a few comments:
    wrt disruptive industries – this was hot when I was getting my masters. Evans and Wuster, Blown to Bits, gave a bunch of examples. Britannica was one – beat out by much poorer quality Encarta because they failed to pay attention. I think Thomas Stewart had a few examples,too, but I lost some respect for his work when he gave Enron as a *positive* (in ~1998?)

    wrt newspapers and photographers – they also have a secondary business in storing and reusing images later and they have to have the rights for that – they also sometimes re-license or sell images so that’s another reason they need to fully own the picture, when the blogs don’t

    Maybe a little Weber bureaucracy here – and organizational psychology….

    And there were news aggregators – lots of them – before Google. DIALOG, Factiva (oh, what was it called before that?), Nexis… the difference is that Google is an advertising company, and these others are subscription or services companies.

    wrt publishers becoming technology companies – some/many/most? publishers outsource the technological parts of putting journals up.. oops! What about societies like *cough* ACS who use publication revenue to pay for society activities and then practice immune responses to try to disable competitors?

    Elsevier (I know, I know) is doing a lot of innovating with technology, too. And, as you say, RSC but they’re a society.

    Both WoS and Scopus have APIs that our programmer has used to provide “related article”, and citation links in our openurl resolver. actually. I use RSS and e-mail alerts from them, too. I’ll give you scholar, though 🙂 Oh, and both are actually providing some mining tools – particularly to try to map science, find related papers, and find sleeping beauties.

    couldn’t agree more about preserving science blogs – that is one positive for journals, they do have that figured out.

  10. Michael..a well written and thoughtful post about the challenges ahead for the scientific publishing industry.

    Regarding “Scientific blogging and wiki platforms”: for Chemistry specifically I am passionate about making the internet searchable by “Chemistry”..specifically structures and reactions. It’s possible.

    I highlight the last section “Why many services will fail:” and believe that the skills we have taken to the RSC, if we get to focus them appropriately, will show that such issues will not always win over the possibilities that exist.

    We share a lot of common views and I am glad that you took the time to put together this post in such a succinct and focused manner. Sign me up for the book (I’ll send the email..)

  11. Thank you for this, Michael: a compelling analysis, forcefully presented, and a way forward for those within STM publishing who have ears to hear, and the organizational discipline to overcome their ‘immune systems’, as you put it.

    I’ll confess to having laughed out loud at some elements, so perfectly do they map on to my experiences within STM publishing over the past couple of years.

  12. Really interesting post! 🙂

    Am really looking forward to the time things start to change in publishing.

    Currently I am working on LogiLogi.org, a project trying to create an informal form of publishing between journals and good conversations (http://en.logilogi.org).

    And in september I am going to do the Digital Humanities MA at Kings College London, to be able to continue working on this and similar projects…

  13. This is a wonderful blog post, and I am sure the APS editors found your talk very stimulating!

    There seems a slight difference between newspapers and science publishing though. In science publishing it still matters (to some and to some degree) where something is published. Google News is successful because it mostly doesn’t matter from which newspapers it pulls the news together.

    At the moment, publication of a scientific paper carries an implicit (albeit not always accurate) notion of trust and relative value depending on place of publication. As you mention, this may eventually be replaced by some sort of page rank algorithm or a voting system – albeit these approaches only work with a certain delay. But what would be interesting might be a more instant expression of potential quality, similar to what we have now with the journal system.

    Last but not least, what is discussed so far is a change in publication medium and technology, and not the system of publishing papers (why publishing science in these little pieces at all?), and in the peer review process. Things like open notebook science might be truly disruptive in this area….

  14. Another example is of course Microsoft.

    Internally it faces battles analogous to those of the newspaper industry.

    The funny thing is that Microsoft’s recent competitive posturing involves even closer ties with the newspaper and big media – industries facing their own massive downward spirals.

    Just amazing that people (and organizations) still hold on to their shares of Microsoft when the writing on the wall is so clear.

  15. Dear Michael,

    thank you very much for sharing this important set of insights. I am looking forward to similar posts on other aspects of the scientific research cycle, and to the integration of all this into some more permanent and more collaborative medium than this blog.

  16. Andrew Maynard – I certainly think that’s a possibility. To some extent, that seems to be what’s going on with the newspapers: they’re dying, but it’s not yet entirely clear how (or if) their services will be replaced.

  17. Sarah – Yes, it can certainly be tough to convince senior people to be involved! In part with good reason: some of these tools aren’t ready for prime time, and I guess they’re understandably reluctant to be early adopters / beta testers. With that said, I think that showing successful existing uses can really help convince people to take the plunge. I tried to do that in a talk I gave earlier this year (http://michaelnielsen.org/blog/?p=545 ).

  18. Asciidan: Do you mean that the photographer is shooting hundreds of shots that end up in the paper each week? Or just hundreds of shots? That latter wouldn’t surprise me, but the former certainly would: every time I’ve seen a photographer at work, it’s usually taken them a couple of hours work (including travel, setup etc) to get one photo that’s actually used in the paper. And sometimes nothing at all gets into the paper. Of course, in those shoots they’ll often take dozens of photos, but most are discarded.

    My cost estimate was based not just on the photographer’s salary, but also on all the associated costs: captioning, the sub-editors, overhead, etc. You may well be right that the estimate is high (it was just a back-of-the-envelope calculation), but I think it’s very likely that TechCrunch is spending at least an order of magnitude less per photo than the papers.

  19. I was General Chairman of the recent 5th International Conference on Open Source Systems, sponsored by IFIP. By pre-existing agreement, IFIP conference proceedings are commercially published in closed format (book and PDF articles) by Springer Verlag. We are actively exploring the possibility of finding alternatives for future conferences. We are also planning to publish selected papers with an open access agreement in JAIS (Journal of the Association for Information Systems, having rejected the International Journal on Open Source Systems, a closed journal commercially published by IGI-Global.

    Readers of this blog may also be aware that Bloomsbury Academic has announced its intent to publish using a Creative Commons license. Also, scribd (www.scribd.com) offers low-cost commercial versions of works from MIT Press and others.

    Academic promotion remains a barrier to widespread disruption of traditional publication of scientific works. In each field, there are established journals and conferences which are highly regarded by senior faculty who make the tenure decisions for junior faculty. Junior faculty must make every effort to publish their research in these peer-reviewed journals and conferences, or risk their promotion. The online journal First Monday has high quality content, but still lacks the academic cred needed to attract junior faculty as contributors.

  20. Jan – Thanks for the comment. A related point is that sometimes the old companies eventually respond by buying up the new companies. This can work, but is risky: often the acquired company goes downhill. For example, my impression is that MySpace has lost a lot of momentum under Murdoch. Sometimes the old companies wait too long to act, and they instead end up being acquired by the new companies, or simply fade. Of course, even if an acquisition happens and is successful, it’s debatable to what extent the company remains the same company. I doubt the apparent success of the New York Times’ blogging endeavours gives much solace to its beat reporters.

  21. This analysis of the naturally destructive/constructive forces that are influencing the future scientific publishing offers hope because:
    1. It exemplifies the high quality writing/thinking that should be disseminated
    2. It offers hope that the closed-door society of scientific publishing will be opened.
    3. It explains a process that could also be our best hope of disupting the seemingly entrenched public school system.
    4. It provides evidence of its own prognastication: I found it on Twitter.

  22. Excellent article Michael, definitely food for thought. A few random comments:

    Is TechCrunch really the best example? Their reputation appears to be dropping lower and lower along with their credibility. Examples here and here.

    One question I have is whether the newspapers are directly competing with the blogs as much as you think. Certainly there are aspects of newspapers that are going head to head with blogs, gossip columns (and things like TechCrunch would qualify here), coverage of niche areas (TechCrunch fits here as well), editorials, science sections, etc. But what about straightforward general news reporting? Aren’t most “news” blogs just link blogs to actual newspapers? I’d argue that for newspapers, the biggest issue has been a lowering of quality to a point where they’re all nearly identical. Since they’re all available for free online with little differentiation (they’re all the same wire feeds), this has led to a saturation of the market, resulting in lower ad rates (along with a loss of classified income to Craigslist). Since ads won’t support an online newspaper, the obvious option is to require a subscription, but you can’t do this because your product can be easily replaced by one of the many other nearly identical newspapers that are still freely available. So they’re stuck in a game of chicken, waiting to see who dies off first.

    Take a look at some of the science innovators you praise: JOVE and SciVee both had to switch from an open access model to become more like traditional publishers because they were losing money hand over fist. PLOS doesn’t seem to make any money from their journals except for those like PLOS-One which rely on high volume and low editorial input (I’ll leave it to the individual reader to judge how this affects content quality), and they’re reliant on donated funding. The Nature Network can’t really be called a success at this point either. It’s cost a huge amount of money and resulted in a fairly small clubhouse for a limited number of scientists. Rumor has it that it’s facing big changes in the future. Mendeley is going to face some serious legal challenges if they ever gain any traction, and beyond that, it’s unclear what exactly their business model is, other than a vague statement about making everything free but selling advanced functionality, functionality that someone else will come along and make free on their own site. The experimentation is great, but so far almost none of it is sustainable.

    And I’d also be cautious about declaring the impact of science blogging on research in general. It’s easy to lose sight of things when one lives in the blogosphere, but think this way–there are an estimated 20+ million working scientists in the USA alone. How many science blogs are there relative to that number, let alone the number worldwide? I still don’t know many scientists, whether old and established or young and fresh in the lab who read science blogs. It’s nice to get to read blogs from senior tenured scientists, but the few you list are outliers, and represent only a tiny minority of their peers. They may just be ahead of the game, and predictive of a massive movement later to follow, but that movement is certainly not here yet.

    Of your recommendations–GoPubMed is a great example of a really useful science search engine. Many journals have invested a lot of time and effort in creating commenting and rating systems for scientific papers. So far, the readers don’t seem all that interested in using them. Nature has invested in building their network, and it’s resulted in around 50 regular bloggers. Probably not the numbers one would hope to get.

    We are still in early days for many of these efforts, and I have a feeling that the things that eventually catch on are yet to come.

  23. a couple additions to the discussion about science publishing.

    A great talk from some open notebook science practitioners here:
    http://scholcomm.columbia.edu/open-science-good-research-good-researchers

    slides from Jean-Claude Bradley here on his open notebook science work:
    http://www.slideshare.net/jcbradley/columbia-talk-on-open-notebook-science

    Jean-Claude and his group are truly impressive in their use of the the free and distributed web2.0/social networking sites to increase the documentation and accessibility of their process and data.

    and additional drive for change is coming from funding agencies which realize their dollars are being wasted in closed publishing systems that don’t help the real goal of the research.
    http://www.surffoundation.nl/en/themas/openonderzoek/Pages/default.aspx

    and the canadian govt help sponsor the creation of open journal systems, http://pkp.sfu.ca, which helps people set up open communities for research publishing.

  24. Michael,

    This was a truly insightful and interesting post. In the ensuing discussion I have to agree with Tony Wasserman that the issue of academic promotion, and the value of traditional publication to this, remains a major hurdle that must be overcome before we really see major disruptions in scientific publishing.

    I’m not close enough on the ground to know if changes are afoot in college science departments that would give more consideration to online publishing when it comes to promotion.

  25. As far as I know, it’s still publish or parish. When the annual review (that determines if you will get grant money) comes around, the determining factor is where you published (i.e. name brand journal like Science or Nature (if you lucky)) and how many times you were cited. The new technology has existed for a long time to increase the dissemination of research, but some scientists still like to keep their work internal. At least until it gets published.

  26. Christina – Thanks for the comments! I don’t know all that much about how Elsevier works internally. Do you know to what extent do they have technology experts actually driving the decision making?

    (It’s hard to generalize for such a huge company, I know.)

    I didn’t know WoS and Scopus had APIs. I wonder how full-fledged they are? I did know about the alerting services and rudimentary data mining, but they are very limited, at least last time I checked — that’s why I talked about, e.g., having “decent alerting services”. Admittedly, while I use WoS often, I use Scopus only very occasionally, and it may be better than I thought.

  27. ChemSpiderman – Congrats on the acquisition, and good luck with the RCS! To my admittedly non-chemist eyes, Chemspider looks great, and, as you say, there’s still so much potential to do great things. I really hope it grows and thrives at RCS.

  28. Wonderful essay, with many key insights. I’m really looking forward to reading your book, and writing positively about it. I particularly liked your links to all the cool emerging tools for scientific collaboration and research. I loved the first part of your essay–spot on. But in the second part, I think some of your remarks are a bit silly. Being a science media professional myself, I hope you’ll excuse my ensuing “classic immune response.”

    While I agree with your thesis that the science publishers (and media companies) who will survive and flourish will be those that exploit technology to its fullest, I think you’re wildly off-base in your extrapolation from that. Your suggestion that programmers and other people who are “deeply technically knowledgeable” should be in charge is short-sighted and biased. Shocking, that a physicist/programmer thinks people just like him should be calling all the shots!

    If that stings, don’t worry, it’s just a “classic immune response.”

    Allow me to make my case by mirroring one of the statements from your essay. You asked: “How many scientific publishers are as knowledgeable about technology as Steve Jobs, Sergey Brin, or Larry Page?”

    You talk about such people (without naming them–do you know who they even are?) as if they have no idea of the “day-to-day” operations, as if they have no “hands-on” knowledge of their bread-and-butter business, but this criticism cuts both ways.

    How many programmers or web developers or network administrators are as knowledgeable about science media or science publishing, or, well, SCIENCE as Geoff Carr at The Economist or Tim Lincoln and Mitchell Waldrop at Nature, or any other number of science-media professionals?

    Your proposal that the Library of Congress archive all science blogs is similarly laughable, precisely because the vast majority of the material on so-called “science blogs” has little to do with actual science, and is mostly insipid pablum. And when it *is* really about science, only the slimmest segment of this is actually original research — most of it is reactionary, in response to research that was vetted and popularized via established science publishers and science media. The blogging researchers you cite, and the content of their blogs, are exceptions to the rule. Your claim that blogs and other “online sources” are not “news parasites” is so easily refuted I’m not going to waste my space and time doing it here–just look at the statistics yourself.

    Despite the increasing popularity of delusional ideas otherwise, technology is not an end in and of itself. It is a means. And by all means, science publishing and media desperately needs more people with deep technical knowledge. But by solely focusing on gee-whiz enabling technologies and the people who make them possible, you’re woefully neglecting the other professionals who actually produce the content that gives those technologies meaning. The notion that all creativity can be automated is specious at best, and willfully ignorant at worst. Without good content and good ideas to traffic, all the technologies you’re discussing are empty vessels.

    Amidst the crisis in publishing, in journalism, scientists and technologists who promulgate happy just-so stories about how this or that technology makes science publishers or science journalists obsolete are essentially fiddling while Rome burns. When you find the various online user-generated “flourishing ecosystems” resembling more the typical comment thread on YouTube rather than anything of erudition, don’t say you and your ilk weren’t warned.

  29. Tony Wasserman got to one of the roots of the matter: “Academic promotion remains a barrier to widespread disruption of traditional publication of scientific works. In each field, there are established journals and conferences which are highly regarded by senior faculty who make the tenure decisions for junior faculty. Junior faculty must make every effort to publish their research in these peer-reviewed journals and conferences, or risk their promotion.”

    As long as promotion is dependent on publishing in old-style paper journals, those journals won’t go away.

    The other root is funding for research: how do granting agencies decide which proposals to fund? As long as the fogeys of the “old boy network” depend on that same network of journals and paper-only conference proceedings, those sources will never die. Only when they are replaced on funding committees by the new generation of media-capable researchers will those media get a fair chance in playing a significant role in scientific progress.

    In addition to the intuitions of members of promotion committees and grant evaluation committees, automated measures of impact can transform the knowledge landscape. ISI’s Science Citation Index was an enormously valuable resource in its day that has been completely replaced by citeseer. But a side effect of the data needed to compile SCI was perhaps equally important, when ISI started publishing impact ratings for journals and other publications. Now evaluators had an objective measure to back up their intuitions that a single article in Cell was worth 20 in Western Biosystems or somesuch unknown journal.

    With a source such as Technorati for science media, the hotspots in new scientific media can be identified. This will drive even more attention to the really good sources, but because of the ability to exhaustively spider the entire visible web, this time around it might help us avoid the tragic fate of Gregor Mendel’s work, which languished in obscurity for 40 years before becoming a foundational portion of modern biology.

  30. Joerg – Thanks!

    I’m not sure I understood you correctly about the differentiation between news sources. Surely people do care (or did, historically) about the difference between different newspapers? E.g., the New York Post versus the New York Times versus the Wall Street Journal?

    I certainly agree that other approaches than the paper-based format are potentially quite revolutionary, and very interesting!

  31. I’ll add my voice to the chorus: this was a wonderful essay!

    Michael, I think many people would welcome more such essays … how about (for example) an essay on traditional biomedical research labs versus robotic/drylab biomedical research labs?

    The point being, it isn’t only in the newspaper business that radical/inexorable changes are happening.

  32. Thanks for the comments, David.

    Re: TechCrunch. I know some people dislike it a lot, while others can’t live without it. (And others manage to do both). But in some particlar ways it’s certainly superior to the Times tech coverage. In other ways, the Times is better. Which it is depends on your exact interests and needs. Personally, neither is all that well adapted to my interests, and I use other sources.

    I largely agree with your comments about online news: there’s a massive and unsustainable oversupply of undifferentiated content.

    Your comments about PLoS One, Mendeley et al are already addressed in my essay.

    Your comments about science blogs are also already addressed in the essay. I made no claim that the impact of research blogs is yet very large: I’m happy to explicitly say that it’s still quite small. But it’s very striking that, apparently, outstanding scientists are over-represented in the blogosphere (10% of the living Fields medallists), and that those are some of the people who are most aggressively expanding the range of the medium. Journals are, by comparison, standing still.

    Thanks for the pointer to GoPubMed.

  33. You’ve done a pretty good job of describing the problems, but like the rest of us, just guessing at the solutions. You’ve done a pretty good job there too, but only time will tell how this shakes out.

    I agree with posters above about competition between Blogs and Newspapers. Frankly, I read news from newspaper web sites, but not the printed version. It becomes obsolete too fast. I’d say that the biggest competitor for newspapers is local and national television. There are sound economic reasons that newspapers don’t publish hourly, but the same economics don’t apply to TV news.

    In the early days of the internet, I noticed that TV news was usually a week or two behind the real news. That gap closed around 2000 and even local stations catch on to trends and significant events pretty quickly. They probably have good technologists, you’re right there.

    Overall, I think the problem with any industry being upset by technology developments is where the company’s management has lost sight of their core business. In the case of music publishers, they were focused on keeping their revenue model intact rather that exploring a new revenue model that fit their new market. The results have been pretty predictable.

    The same is true of scientific publishing. To paraphrase a campaign slogan, “It’s the Information Stupid”. A poster above cited the problems with the ACS, and I agree. They don’t see the scientific value of their information. They see a revenue stream to defend. When I was in college (last century) we were taught to research using Chem Abstracts. The value of CA to the chemistry community was indexing the available literature. Checking through five years of accumulated indexes is no longer efficient, but they charged the same amount for the online information as they charged for the printed volumes. This only works until someone decides to take a more cost-effective approach.

    The internet is indeed the death of the traditional journal. Why? Because I really don’t want to be forced to buy an issue of a journal with two articles of interest and twenty that aren’t. That old delivery model was the most efficient for its day, but not any more. We’re just waiting for a better service model to come along.

    I don’t think that the companies that cling for dear life to their old business models are stupid or malevolent. I do think that they’re badly managed, and in need of some visionary leadership. That will take some time. As another blogger pointed out, Google wasn’t the first in search, it turned out to be the best with a viable business model. We probably won’t recognize the newest successful Scientific publisher until there’s almost no one left to compare them to.

    Thanks for your inspiring article.

  34. Internet makes things go fast. I found the post by googling PLoS. I posted a link on Twitter and saw it retweeted a number of times (including probably some of the commenters above). A couple of hours later I was on a panel about science publishing and completely changed a chunk of what I was saying to include some of the ideas from your essay (which I attributed to you, by name, if you are wondering). Very thought-provoking and very good. Another one of those posts of yours that people keep coming back to over and over again.

  35. You mentioned EtherPad, but you might want to also look at the demo of Google Wave for enhanced real time on line collaboration. It looks very promising, and is intended to be made as open as possible with public API methods to interact with other on line services.

    http://wave.google.com/

Comments are closed.