“The” Scientific Method

Cosma Shalizi ponders the Scientific Method and the Philosophy of Science:

Philosophy of science these days seems largely concerned with questions of method, justification and reliability — what do scientists do (and are they all doing the same thing? are they doing what they think they’re doing?), and does it work, and if so why, and what exactly does it produce? There are other issues, too, like, do scientific theories really tell us about the world, or just give us tools for making predictions (and is there a difference there?). The whole reductionism—emergence squabble falls under this discipline, too. But (so far as an outsider can judge), method is where most of the debate is these days.

Of course, most scientists proceed in serene indifference to debates in methodology, and indeed all other aspects of the philosophy of science. What Medawar wrote thirty years ago and more is still true today:

If the purpose of scientific methodology is to prescribe or expound a system of enquiry or even a code of practice for scientific behavior, then scientists seem to be able to get on very well without it. Most scientists receive no tuition in scientific method, but those who have been instructed perform no better as scientists than those who have not. Of what other branch of learning can it be said that it gives its proficients no advantage; that it need not be taught or, if taught, need not be learned?

(Actually, has anyone done a controlled study of that point?) One of the things a good methodology should do is, therefore, either explain why scientists don’t have to know it.

An observation I find fascinating is that scientists employ very different norms when evaluating what it means to know something in their area of expertise versus what they know about doing science. An experimental physicist may have extremely rigorous standards of what it means to determine something experimentally, and a far more seat-of-the-pants means of evaluating knowledge about what it means to be a good experimental physicist. And, of course, they apply different standards again to everyday knowledge.

Cosma continues a little later:

Now of course working scientists do employ lots of different methods, which are of varying quality. The same is true of all learned professions, and it is probably also true that most professionals (lawyers, architects, doctors) pay no heed to foundational debates about what they are doing. Instead methods seem to breed within the profession — this technique is unreliable under these circumstances, that procedure works better than the old one, etc. — without, as it were, the benefit of philosophical clergy.

Feyerabend had a nice term for this – “anything goes”. I don’t think he meant this literally. Rather, he meant that method was something that scientists invented on a case-by-case basis, with formal methodology being only a heuristic guide, not gospel.

Published

StartupCamp Waterloo

StartupCamp is on in Waterloo next Tuesday night, 6pm to 9pm, at the Waterloo Accelerator Centre, 295 Hagey Blvd., Waterloo. I went to a similar event, DemoCamp Guelph, in June, and it was terrific – lots of energy and great ideas in the room, with a nice informal atmosphere that made it easy to meet people. Great for anyone interested in entrepeneurship, or just interesting new technology.

More generally, if you’re in the Greater Toronto area, and haven’t already done so, you should check out the TorCamp page, which lists all the BarCamps, DemoCamps and so on going on in the area. There’s a lot of really amazing stuff going on!

Published

Academic Reader Database

Just a brief note for users of the Academic Reader – we lost a few hours worth of database updates today. It was an error on my part, and I apologize if anyone lost anything significant because of it. Ironically, the error occurred as I was updating the code so that database backups are made more frequently than once every 24 hours.

Published

Open source software at centralized servers?

Does anyone know of examples of open source software projects which are developing software that is run on large centralized servers? I can think of one example off the top of my head – Second Life – but can’t think of any others.

(I am, of course, asking for a reason – I’m interested in whether open source might be a viable development model for tools for scientific collaboration and publication.)

My impression at the moment is that there are few centralized web services which are open source. I can think of a couple of natural reasons why this might be the case.

First are security issues. In any software development one needs to be sure the programmers are honest, and not slipping back doors into the code, or making unethical use of the database. This is potentially harder to control in an open source software project.

Second, although the software may in some sense be owned by the wider community, it does not necessarily follow that the server is owned by the wider community. The group that owns the servers has a much greater incentive to contribute, and other people less so, which lessens the advantages to be had by open sourcing the project.

Are there any reasons I’m missing? Centralized services other than Second Life which are open source?

Published

Scientific communication in the 21st century

By guest blogger Peter Rohde

In the last year the number of papers I have fully read can easily be counted on your hands. For the larger part I only read abstracts. Why is this? Because for most academic works I’m not especially interested in the details of calculations or the nitty gritty fine points of results. That’s something I’ll refer back to when/should I need it. For the larger part I’m only interested in understanding what it is that’s been done, what approaches were used to obtain the results, and what the remaining unanswered questions are. Typically these things can be characterized much more compactly than via a full scientific paper.

Aside from reading abstracts I gain much of my knowledge by speaking to people. This is a particularly useful way of learning for two reasons. Firstly, it is efficient, unlike verbose papers, and secondly it is interactive. If a particular point is not clear to me, I can grill for more detail. So, for the larger part, verbose scientific papers are far less useful to me than are their abstracts or talking to other people. Both of these points concur with the suggestions made in Robin Blume-Kohout’s contribution to this blog, where he advocates the “choose you own adventure” or hierarchically structured model. Evidently, speaking to other people is an example of this model – we prefer the terse over the verbose, with elaborations only when required. In such a structure, as I would envisage it, the abstract would be the root node of a tree. It would summarize the paper in a condensed, but completely self contained way – a micro-publication in very compact form. Each of the components in the abstract could be folded out to reveal further underlying details. This way the content is tailored to every reader. It means that I can continue doing what I normally do – only reading abstracts – with the bonus that if a particular aspect of the abstract is of interest to me, I can delve into it a little further without requiring me to read the entire paper.

This type of scientific communication lends itself exclusively to online publication. Indeed electronic media provides a plethora of new ways to structure and modularize information. Despite this, scientific publication has been stuck in a time warp where the archaic form of publication has been preserved. Essentially, present day electronic publications are structured and organized in exactly the same way as printed publications were 50 years ago, the only difference being that an LCD replaces paper. This is a sad misuse of resources.

Almost every other aspect of e-society has adopted, to some extent, the ideas advocated here and by Robin. The Wikipedia is the obvious, and perhaps most sophisticated example of this. Here every point in every article cross-references to other articles, creating a highly modularized and hierarchical structure. There are also less obvious examples. These days I never purchase newspapers, and it’s not an issue of saving money, it’s an issue of structural design. If I go to any major online news source, I’m presented with a very elegantly structured, hyperlinked front page. At the top of the page are all the headlines, each with a single line summary. Below this are divisions for international news, politics, technology, science etc, each with their own headlines and single line summary. In principle I could read just the front page and have a pretty good idea of what’s going on in the world and if I want more detail I can follow the links. This is much more efficient than the style adopted by many conventional newspapers of having one main story on the front page in addition to a few other headlines crammed at the bottom of the page, and all the rest jammed into separate pullouts.

Another area where the e-world is a step ahead of the paper world is in creating awareness of content. In present day scientific communication awareness of articles is created via two primary means. The first is by speaking with fellow scientists who draw our attention to articles that interested them. The second is by stumbling across things by oneself, for example, by reading the daily arXiv feeds. The trouble is that nowadays there is so much throughput that it becomes increasingly difficult to keep track of it all. A good analogy is the internet itself. Clearly the amount of material becoming available online is impossibly large to manage oneself. So to increase awareness of things that are of general interest, sites such as Slashdot, reddit and Digg have emerged. All these sites use some voting mechanism to create a list of pages that are of most interest to the online community. I think it is rapidly reaching the point where coping with the massive quantity of scientific communication will necessitate these kinds of approaches.

Another example of awareness creation, which is perhaps more suited to scientific publication, is that of recommendation systems. Some well known examples of recommendation systems are Amazon, iTunes, StumbleUpon and Last.fm. Here users’ preferences for pages/books/music are tracked, but not with the intention of creating a popularity list. Instead the preferences are hidden and only used internally by the service provider, who cross correlates your preferences with other users’ to suggest pages/books/music that might be of interest to you. This approach to discovering material is clearly much more effective than trawling through the immense amount of material out there on my own. Instead I can exploit the fact that others have done it for me.

In summary, the structure of present day scientific communication is inherently archaic. It replaces paper with LCD while taking little advantage of the abundance of possibilities for structuring information. Second, the sheer magnitude of scientific communication necessitates new means for creating awareness of material, using, for example, recommendation systems. While it’s very easy for me to sit here and bawl criticism at the current system, it’s not so straightforward to actually effect a transition to a different model. One route would be to convince a major publisher to adopt some of the aforementioned suggestions, and hope that it’s a success. The other would be set up a new system (e.g. a wiki or the like) and convince a group of reputable scientists to transition to that system. In either case, the success of the pursuit would require a certain critical mass.

Published

Changing fields

After 12 years of work on quantum information and quantum computation, I’ve decided to shift my creative work to a completely new direction.

I’m making this shift because I believe I can contribute more elsewhere.

I became interested in quantum information and computation in 1992, and started working fulltime on it in 1995. When I started it was a tiny little field with a handful of practitioners around the world. Most scientists hadn’t even heard of quantum computers. Those few who had would often use what they’d heard to pour cold water on the idea of ever being able to build one. Now, in 2007 the field is one of the hottest in physics, and many researchers, myself included, believe it is only a matter of time and concentrated effort before a large-scale quantum computer is built.

To me this seems a propitious time to change direction.

The new direction I’ll be working toward is the development of new tools for scientific collaboration and publication. This is a tremendously exciting area, and it’s also one where my skills and interests seem likely to be useful. I’m a beginner in the area, and so for the next few months, I’ll be doing a “reconnaissance in force”, orienting myself, figuring out what I need to learn, where I might be able to make a contribution, and launching some small projects. It ought to be a blast.

Published
Categorized as General

Reinventing scientific papers

By guest blogger Robin Blume-Kohout

In 2005, Slate published twelve essays on “How to reinvent higher education”. The opening paragraphs of one, by Alison Gopnik, still burn in my mind:

I’m a cognitive scientist who is also a university professor. There is a staggering contrast between what I know about learning from the lab and the way I teach in the classroom. … I know that children, and even adults, learn about the everyday world around them in much the way that scientists learn. …Almost none of this happens in the average university classroom, including mine. In lecture classes, the teacher talks and the students write down what the teacher says. In seminars, the students write down what other students say. This is, literally, a medieval form of learning

In short, we are screwing up — and we should know better.

Scientific publishing — the primary means by which we communicate with other scientists — is in the same boat:

  1. We’re doing it badly,
  2. Our methods are medieval,
  3. We should know better.

Technically, point #2 is unfair. Scientific publishing dates from the 1660s, when Proceedings of the Royal Society emerged from Henry Oldenburg‘s voluminous scientific correspondence. If you wanted to show off your research in 1665, you wrote a letter to Henry. When he got it (a month or two later), he forwarded it to someone who could tell him whether it was any good. If the referee liked it, then (after a few more month-long postal delays), Henry read your letter out loud to the Royal Society, and it got recorded in the Proceedings.

These days, it’s quite different. Specifically:

  1. We write letters in LaTeX, and email them,
  2. There are so many journals that nobody reads most of them,
  3. Henry doesn’t read your letter out loud.

The rest of the system is unchanged. This raises a bunch of questions, like “Why does publication take 6 months?”, “Why is it so expensive?”, and “Does anybody read journals, what with the arXiv?” I’m not going to discuss these questions, but if you’re interested, you might try the Wikipedia article on scientific journals. Which is a perfect example of why we should know better.

I’m not talking about the content. I’m talking about the article itself, and how I referenced it — with a hyperlink. I’ve given you incredible power. Quickly and easily, you can:

  • Verify my sources,
  • Find answers to questions I’ve raised — if you’re interested,
  • Get more detailed explanations,
  • Discover and explore related topics.

Enabling you this way is part of the core mission: The purpose of scientific communication is to educate, extensibly and efficiently. Education: After months of research, I publish a paper so that you can learn what I know — without all the hard work. Extensibility: I include proofs, arguments, figures, explanations, and citations — so that you can verify my work and place it in the context of prior work. Efficiency: Writing this way takes more months — but thousands of my colleagues can save months by reading my paper.

We are failing at efficiency, for Wikipedia illustrates a more efficient way of educating — or, if you prefer, a source for more efficient learning. I don’t mean that Wikipedia is The Answer. We need to build a new medium, replacing medieval features with the best features of Wikipedia. For instance,

  • Hypertext revolutionizes scientific writing, by organizing content as a tree instead of a list. Articles and textbooks have a linear structure. To find a specific answer, I have to read (on average) half the text. In a hypertext environment like Wikipedia, I can search through a cluster of ideas for answers — even to questions I haven’t been able to formulate yet. Hyperlinking specifically enables…
  • Choose your own adventure” approaches to a body of work. Scientific papers represent a cluster of related ideas. Different readers, with different background knowledge, will benefit from different paths. A well-structured (and judiciously hyperlinked) electronic text can become the reader’s personalized guide. Parts of several such texts can be combined by a customized path, to form an entirely new text. This requires…
  • Modular content, dividing a text into bite-sized chunks. Modularity also offers intrinsic benefits. One is reusability; a single explanation can be referenced in many contexts. Current scientific writing is necessarily terse. Hyperlinks and modularity allow the text to be larded with optional explanations, which clarify potential confusion without breaking the flow. Modularity also allows alternative approaches, providing the reader with multiple analyses of the same concept. Such alternatives are particularly useful when combined with…
  • Distributed editing by a large community of contributors. This is a vast can of worms that I shan’t open here, but two things are clear. First, a forum for scientific communication cannot adopt Wikipedia’s “anyone can edit” motto. Second, the potential benefits of post-publication editing, combined with an unlimited pool of “editors”, are too great to ignore. Balancing these imperatives is an outstanding challenge, but a relatively uncontroversial technique is…
  • Attached commentary, either critical or explanatory, by readers. Consider, for example, the Talmud, where post-publication analysis (the Gemara) attempts to clarify the original text (the Mishnah). More recently, commenting systems have proliferated on blogs and (with much, much less intellectual rigor) news-sites like Slashdot. In a scientific publishing context, commentary can
    • correct mistakes, either technical or factual, in the original text,
    • provide an alternative to a module that (the reader feels) could be improved,
    • critique and question the original work,
    • update older work in light of new research.

These points are not a prescription. They are a manifesto (“We can do better, see!”), and a plea (“Help make it better!”). Published scientific communications are the collective memory of scientists. If we cannot access it quickly and efficiently, we are effectively brain damaged. Improving our access makes us — quite simply — smarter. All we need to do is to use the computing tools before us intelligently.

We’ve taken first steps — the preprint arXiv, central repositories like PROLA, and online publishing by the likes of Nature. These are baby steps. We’re doing the same old thing a little better with new technology. Sooner or later, scientific communication is going to be restructured to really take advantage of what we can do now… and it’s going to make us (collectively) a lot smarter.

I can’t wait.

Published
Categorized as General

How to write consistently boring scientific literature

How to write consistently boring scientific literature, by Kaj Sand-Jensen

Although scientists typically insist that their research is very exciting and adventurous when they talk to laymen and prospective students, the allure of this enthusiasm is too often lost in the predictable, stilted structure and language of their scientific publications. I present here, a top-10 list of recommendations for how to write consistently boring scientific publications. I then discuss why we should and how we could make these contributions more accessible and exciting.

Sadly, this is hidden behind a publisher pay wall. I particularly enjoyed the opening quote:

“Hell – is sitting on a hot stone reading your own scientific publications”
– Erik Ursin, fish biologist

Published
Categorized as General

Non-abelian money

What would happen if we replaced the current monetary system, which is based on an abelian group [*] by a non-abelian currency system?

[*] If someone gives you x dollars, then y dollars, the result is the same as if you were given y dollars first, then x.

I’ve been puzzling about this for a few years. It raises lots of big questions. How would markets function differently? Might this lead to more efficient allocation of resources, at least in some instances? (At the very least, it’d completely change our notion of what it means to wealthy!) Might new forms of co-operation emerge? How would results in game theory change if we could use non-abelian payoffs?

More generally, it seems like this sort of idea might be used to look at all of economics through an interesting lens.

A nice toy model in this vein is to work with the group of 2 by 2 invertible matrices, with the group operation being matrix multiplication. By taking matrix logarithms, it can be shown that this model is a generalization of the current monetary system.

Electronic implementation of non-abelian money would be a snap. The social implementation might be a bit tougher, however – convincing people that their net wealth should be a matrix would be a tough sell, at least initially. Still, if non-abelian money changed some key results from economics, then in some niches it may be advantageous to make the switch, and possible to convince people that this is a good idea.

(It should, of course, be noted that there are in practice already many effects which make money act in a somewhat non-abelian fashion, e.g., inflation. From the point of view of this post, these are kludges: I’m talking about changing the underlying abstraction to a new one.)

Published
Categorized as ideas

The standard negative referee report

“The work reported in this paper is obvious, and wrong. Besides, I did it all 5 years ago, anyway.”

(I heard this from my PhD supervisor, Carl Caves, about 10 years ago. At the time, I thought it was funny…)

Published
Categorized as General