Open source Google

Why can’t we ask arbitrarily complex questions of the whole web?

Consider the questions we can ask the web. Type a name into Google and you see, very roughly, the top sites mentioning that name, and how often it is mentioned on the web. At a more sophisticated level, Google makes available a limited API (see here, here, and here) that lets you send simple queries to their back-end database.

Compare that to what someone working internally for Google can do. They can ask arbitrarily complex questions of the web as a whole, using powerful database query techniques. They can even apply algorithms that leverage all the information available on the web, incorporating ideas from fields like machine learning to extract valuable information. This ability to query the web as a whole, together with Google’s massive computer cluster, enables not only Google search, but also many of the dozens of other applications offered by Google. To do all this, Google constructs a local mirror of the web, which they then enhance by indexing and structuring it to make complex queries of the web possible.

What I want is for all developers to have full access to such a mirror, enabling anyone to query the web as a whole. Such a mirror would be an amazing development platform, leading to many entirely new types of applications and services. If developed correctly it would, in my opinion, eventually become a public good on a par with the electricity grid.

A related idea was announced last week by Wikipedia’s Jimbo Wales: the Search Wikia search engine is making available an open source web crawler which can be improved by the community at large. This great idea is, however, just the tip of a much larger iceberg. Sure, an open source search tool might improve the quality and transparency of search, and provide some serious competition to Google. But search is just a single application, no matter how important; it would be far more valuable to open up the entire underlying platform and computing infrastructure to developers. I predict that if Search Wikia is successful, then the developers contibuting to it will inevitably drive it away from being a search application, and towards being a development platform.

I believe such a platform can be developed as an open source project, albeit a most unconventional one. So far as I am aware, no-one has ever attempted to develop an open source massively distributed computing platform. Many of the required ideas can of course be found in massively distributed applications such as SETI@Home, Folding@Home, and Bram Cohen’s BitTorrent. However, this project has many very challenging additional problems, such as privacy (who gets to see what data?) and resource allocation (how much time does any party get on the platform?)

Once these problems are overcome, such an open source platform will enable us to query not only the web as a whole, but also what John Battelle has called the “database of human intentions” – all the actions ever taken by any user of the platform. Indeed, Google’s most powerful applications increasingly integrate their mirror of the web with their proprietary database of human intentions. It’d be terrific if these two databases – the web as a whole, and the database of human intentions – were available to and fully queryable by humanity at large.

Was the Universe formerly a black hole?

This is a question that’s bugged me for a while.

First, here’s why I think this is a reasonable question to ask.

Suppose you cram a mass M into a spherical volume of radius R such that R is less than the Schwarzschild radius, i.e., R \leq 2GM/c^2. Then it’s a pretty well understood consequence of general relativity that the mass will collapse to form a black hole.

Current estimates of the mass of the (observable) Universe vary quite a bit. This webpage seems pretty representative, though, giving a value for the mass of 3 x 10^52 kg.

This gives a value for the corresponding Schwarzschild radius of about 6 billion light years.

The radius of the observable Universe is, of course, quite a bit bigger than this. But the Universe is also expanding, and at some point in the past its radius was quite a bit less than six billion light years.

If that was the case, why didn’t it collapse to form a singularity? In short, how come we’re still here?

Any cosmologists out there who can enlighten me?

Update: In comments, Dave Bacon points to an enlightening essay from John Baez, explaining some of what’s going on.

My interpretation of the essay is that the standard lore I learned as an undergraduate (namely, that if you take a mass M and compress it into a smaller radius than the Schwarzschild radius then a black hole must inevitably form) is wrong, and that the FRW cosmology provides a counterexample.

This begs the question of when, exactly, a black hole can be guaranteed to form.

Published
Categorized as Physics

The Academic Reader and RSS readers

In comments, Yue Li writes of the Academic Reader:

This seems very similar with a production of Google, the google reader, a quantum specific google reader:)

In the existing site, the main difference between the Academic Reader and RSS readers like the Google Reader is that we have a variety of ways of searching and browsing older papers. This means the Academic Reader allows you to both (1) keep abreast of your current reading, and (2) look back into the past, discovering older papers and so on. RSS readers typically focus on just the first of these problems.

This functionality will be greatly extended in coming months!

The Academic Reader V0.03

New features at www.academicreader.org:

  • The statistics and nonlinear dynamics feeds from arxiv have now been added. These parts of the arxiv aren’t as active as the physics arxiv, so don’t be surprised to see empty feeds occasionally.
  • The physics arxiv is largely complete back to the beginning of 2007.
  • Adding new feeds to your feed list is a lot easier.
  • Many minor improvements in the user interface.

Introduction to Yang-Mills theories

Yang-Mills theories are a class of classical field theory generalizing Maxwell’s equations. When quantized, Yang-Mills theories form the basis for all successful modern quantum field theories, including the standard model of particle physics, and grand unified theories (GUTs) that attempt to go beyond the standard model.

This post contains working notes (32 pages, pdf only) that I wrote in an attempt to come to a satisfactory personal understanding of Yang-Mills theory. They are part of a larger project of understanding the standard models of particle physics and of cosmology – some related earlier notes are here.

Caveat: The current notes take a geometric approach to Yang-Mills theory, and include quite a bit of background on differential geometry. After completing a first draft, I realized that if I was to write either a pedagogical introduction or a review of Yang-Mills theory, this geometric approach is not the approach I’d prefer to take. Rather, I’d start with a bare statement of the Yang-Mills equations, considered as a generalization of Maxwell’s equations, and then work through a series of examples, only gradually mixing in the geometric approach. This would have the advantage of bringing readers up to speed much more quickly, without needing to absorb reams of differential geometry upfront.

Because of this, I haven’t polished these notes – they remain primarily my personal working notes, and there are various inaccuracies and shortcomings in the notes. I’m content to ignore these – why spend time polishing when you know a better approach is possible – but would appreciate advisement if you spot any serious misconceptions.

Despite these caveats, I believe the notes may be useful to some readers. In particular, if you’d like to understand the approach to Yang-Mills theory from differential geometry, these notes may serve as a useful first step, to be supplemented by additional reading such as the book by Baez and Muniain (“Knots, Gauge Fields and Gravity”, World Scientific 1994) on which the notes are primarily based.

Enjoy!

Update: If you’re reading the notes in detail, then you might want to take a look at the comments, esepcially those by David Speyer and Aaron Bergman, who provide some important corrections and extensions.

The Academic Reader V0.02

We’ve introduced an updated version. New features include 31 extra feeds (all of the mathematics feeds at www.arxiv.org), the ability to browse backwards in time through feeds, and an announcement blog. Many more things are still to come! The announcement blog is at:

http://academicreader.wordpress.com

We have had a few problems with feed continuity – some articles from the arxiv have failed to show up, or have showed up late. We’re working on this, and things seem to be smoothing out.

Thanks to everyone who signed up, especially those who provided feedback – we have nearly 250 registered users!

Published
Categorized as General

Introducing the Academic Reader

The Academic Reader is a new web site that makes it easier to keep track of your scientific reading. Rather than going to multiple websites every day to keep up, we pull all the sources together in a single location, so you can keep track easily. Sources include the preprint arXiv, the Physical Review, and Nature, and many new sources will be added in the months to come, including sources outside physics.

Visit version 0.01 of the site at http://www.academicreader.org.

Update: Last week the preprint ArXiv overhauled its paper numbering. A side effect, which became visible just as we announced (!), is that their back-end interface for extracting paper data is currently completely down. They promise it’ll be back soon. In the meantime, the ArXiv feeds on the Academic Reader will be bit dated. Our apologies for that.

Update 2: The ArXiv appears to be back to normal.

Update 3: We had some server downtime for about 15 mins (around 0700 UTC) due to a hastily scheduled memory upgrade needed to speed things up (thanks everyone for registering!) Sorry if you got booted off the server – we’ll try to make this more transparent in future.

Published
Categorized as General

What is the Universe made of? Part I

This post is, more than usual, a work in progress. It is the first draft of the first installment of a longer article on the subject “What is the Universe made of”. I intend to revise this draft and finish the longer article over the next few weeks, posting it to my blog as I make progress.

This first installment gives a bird’s-eye view of the subject, describing in very broad terms how ideas from particle physics, from cosmology, and from quantum gravity have contributed to our current understanding of what the Universe is made of. It’s really just a warm-up – subsequent installments will be meatier, describing in more detail each of these ideas, how they fit together, and some of the big questions that remain. The next installment will describe the standard model of particle physics in some detail.

The article is intended for a general audience, albeit one with a good grounding in basic science. Physicists hoping for a technical treatment will be disappointed. I certainly can’t claim any great expertise in the subject; while I’m a theoretical physicist, my work has been mostly on quantum information, not particle physics, cosmology, or quantum gravity, and I’m far from being expert on the topics discussed here. If you are an expert, and spot any errors, I’d appreciate hearing about them.

Here’s a link to the article. (PDF only, I’m afraid).

Fields Medal

Update: The four medallists are Perelman (declined), Tao, Russian Andrei Okounkov, and Wendelin Werner. There is a nice article about Tao’s work at the UCLA website. The Nevanlinna Prize went to John Kleinberg.

Tao’s Medal will cause a lot of of excitement in Australia, as he’s the first Autralian ever awarded the Fields Medal.

I’m particularly pleased, as I greatly admire some work Tao did with Allen Knutson on a problem of some interest in quantum information, known as Horn’s problem: given that A+B=C, where A, B and C are Hermitian, what can be said about the relationship between the eigenvalues of A, B and C? There is now a complete (and deep) solution to this problem, and Tao and Knutson played a central role in obtaining it.

This problem at first might not seem all that related to quantum mechanics. It turns out that there are a lot of connections between the techniques used, a fact you can see in some beautiful work by Hayden and Daftuar and by Klyachko, who made important independent contributions to the solution of Horn’s problem.

Of course, this is just one of many things Tao is known for – he’s probably better known for his proof (with Ben Green) that there exist arbitrarily long sequences of primes in arithmetic progression. His web page lists an amazing array of papers, books, expository notes, and other activities (he’s a vegemite fan); it’s a lot of fun to read through!

Published
Categorized as General

Lives of quiet desperation

Noted for posterity: a site to download quantum computing term papers.

(Addendum: After posting, I had some qualms about the wisdom of posting this link. However, upon a nanosecond’s reflection, I would guess that anyone desperate enought to be looking for quantum computing term papers is unlikely to be reading my blog. )

Published
Categorized as General