Comments on: Using your laptop to compute PageRank for millions of webpages http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/ Tue, 09 Mar 2010 22:14:18 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Michael Nielsen http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-26214 Michael Nielsen Wed, 30 Sep 2009 21:06:03 +0000 http://michaelnielsen.org/blog/?p=523#comment-26214 Daniel - Thanks for the note. When preparing the notes I considered doing something similar, but decided the presentation would be a little easier to understand without it. It's been a while, but as I recall, my line of thought was simply that I wanted to avoid numpy, probably because the notes were originally prepared for people not at all familiar with numpy. The purpose of the notes is, of course, pedagogical - production code would be vastly different. Daniel – Thanks for the note. When preparing the notes I considered doing something similar, but decided the presentation would be a little easier to understand without it. It’s been a while, but as I recall, my line of thought was simply that I wanted to avoid numpy, probably because the notes were originally prepared for people not at all familiar with numpy. The purpose of the notes is, of course, pedagogical – production code would be vastly different.

]]> By: Daniel Smilkov http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-26213 Daniel Smilkov Wed, 30 Sep 2009 18:55:29 +0000 http://michaelnielsen.org/blog/?p=523#comment-26213 You know, instead of using column matrices, you can use n-dim arrays: v = numpy.matrix(numpy.zeros((n,1))) becomes v = numpy.zeros(n,dtype=numpy.float32) and p = numpy.matrix(numpy.ones((n,1)))/n becomes p = numpy.ones(n,dtype=numpy.float32)/n and putting the initialization of the v vector out of step function, initializing it only ones in pagerank function, you can get a lot of speed-up. You know,

instead of using column matrices, you can use n-dim arrays:
v = numpy.matrix(numpy.zeros((n,1)))
becomes
v = numpy.zeros(n,dtype=numpy.float32)

and
p = numpy.matrix(numpy.ones((n,1)))/n
becomes
p = numpy.ones(n,dtype=numpy.float32)/n

and putting the initialization of the v vector out of step function, initializing it only ones in pagerank function,

you can get a lot of speed-up.

]]> By: Michael Nielsen http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-17334 Michael Nielsen Sun, 25 Jan 2009 19:10:40 +0000 http://michaelnielsen.org/blog/?p=523#comment-17334 Thanks for that, Henry! I've reshared your comment to the FriendFeed room. It definitely suggests I should use sparse matrices if possible in my distributed implementation. Thanks for that, Henry! I’ve reshared your comment to the FriendFeed room. It definitely suggests I should use sparse matrices if possible in my distributed implementation.

]]> By: Henry Haselgrove http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-17322 Henry Haselgrove Sun, 25 Jan 2009 09:06:07 +0000 http://michaelnielsen.org/blog/?p=523#comment-17322 For those of you who like Matlab, Matlab's inbuilt sparse matrix type appears to be a fairly efficient way to compute pagerank. For example, this Matlab function that I wrote computes a 2 million page pagerank in about 45 seconds: http://users.on.net/~henry/pagerank1.m Apparently the "scipy" package provides sparse matrix types in python. I'll give it a go sometime to see whether that is just as fast. For those of you who like Matlab, Matlab’s inbuilt sparse matrix type appears to be a fairly efficient way to compute pagerank. For example, this Matlab function that I wrote computes a 2 million page pagerank in about 45 seconds: http://users.on.net/~henry/pagerank1.m

Apparently the “scipy” package provides sparse matrix types in python. I’ll give it a go sometime to see whether that is just as fast.

]]> By: What is Authority? | Fidicaro.net http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-16663 What is Authority? | Fidicaro.net Tue, 30 Dec 2008 08:22:32 +0000 http://michaelnielsen.org/blog/?p=523#comment-16663 [...] do believe there is room in social media for authority. I believe Google’s pagerank algorithm may be a great start. A Twitter application that monitors retweets, recency of tweets, frequency of [...] [...] do believe there is room in social media for authority. I believe Google’s pagerank algorithm may be a great start. A Twitter application that monitors retweets, recency of tweets, frequency of [...]

]]> By: What is Authority? | The Marketing Technology Blog, Indianapolis http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-16647 What is Authority? | The Marketing Technology Blog, Indianapolis Tue, 30 Dec 2008 03:25:06 +0000 http://michaelnielsen.org/blog/?p=523#comment-16647 [...] do believe there is room in social media for authority. I believe Google’s pagerank algorithm may be a great start. A Twitter application that monitors retweets, recency of tweets, frequency of [...] [...] do believe there is room in social media for authority. I believe Google’s pagerank algorithm may be a great start. A Twitter application that monitors retweets, recency of tweets, frequency of [...]

]]> By: Coderies | taggle.org http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-16632 Coderies | taggle.org Mon, 29 Dec 2008 08:30:43 +0000 http://michaelnielsen.org/blog/?p=523#comment-16632 [...] Lancer un algorithme de PageRank sur un ordinateur portable, c’est possible [...] [...] Lancer un algorithme de PageRank sur un ordinateur portable, c’est possible [...]

]]> By: John Sidles http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-16553 John Sidles Wed, 24 Dec 2008 23:08:39 +0000 http://michaelnielsen.org/blog/?p=523#comment-16553 Oh boy! Now we're getting to a portion of this lecture series that I was *definitely* looking forward to, namely, <i>O</i>(<i>n</i>) matrix-vector multiplication algorithms. For reasons that (AFAICT) nobody clearly understands, in recent decades <i>O</i>(<i>n</i>) matrix-vector multiplication algorithms have appeared ubiquitously in modern physics, chemistry, engineering, and information theory. Which is good, because with the help of these algorithms (as Michael observes), systems having <i>n</i>~10^6-10^8 degrees of freedom become tractable to analyze and/or simulate. A key issue (whose answer is not obvious to me from the lecture) is whether <i>M</i> is Hermitian (apparently not) and/or can be related to a Hermitian matrix via an <i>O</i>(<i>n</i>) pre-conditioning algorithm (which is the bit that I'm not clear on at present). When this is true, then <i>M</i> (typically) can be regarded as a metric on a (typically) tensor network space, and the act of matrix-vector multiplication can be viewed (typically) as the fast computation of the so-called <i>musical isomorphism</i> between one-forms and vectors. It turns out that this geometric point of view is congenial in many large-scale simulation and information-processing algorithms, but (despite my modest efforts) it has never yet inspired in me any geometric understanding of Google's PageRank algorithms ... and yet I feel that it should. Anyway, that's how one reader is processing these fine lectures ... a reader who would be pathetically grateful for rays of geometric illumination. Just to mention, a fine textbook relating to these issues is Anne Greenbaum's <i>Iterative methods for solving linear systems</i>. ------------- @book{Author = {Greenbaum, Anne}, Publisher = {Society for Industrial and Applied Mathematics (SIAM)}, Series = {Frontiers in Applied Mathematics}, Title = {Iterative methods for solving linear systems}, Volume = {17}, Year = {1997}} Oh boy! Now we’re getting to a portion of this lecture series that I was *definitely* looking forward to, namely, O(n) matrix-vector multiplication algorithms.

For reasons that (AFAICT) nobody clearly understands, in recent decades O(n) matrix-vector multiplication algorithms have appeared ubiquitously in modern physics, chemistry, engineering, and information theory. Which is good, because with the help of these algorithms (as Michael observes), systems having n~10^6-10^8 degrees of freedom become tractable to analyze and/or simulate.

A key issue (whose answer is not obvious to me from the lecture) is whether M is Hermitian (apparently not) and/or can be related to a Hermitian matrix via an O(n) pre-conditioning algorithm (which is the bit that I’m not clear on at present).

When this is true, then M (typically) can be regarded as a metric on a (typically) tensor network space, and the act of matrix-vector multiplication can be viewed (typically) as the fast computation of the so-called musical isomorphism between one-forms and vectors.

It turns out that this geometric point of view is congenial in many large-scale simulation and information-processing algorithms, but (despite my modest efforts) it has never yet inspired in me any geometric understanding of Google’s PageRank algorithms … and yet I feel that it should.

Anyway, that’s how one reader is processing these fine lectures … a reader who would be pathetically grateful for rays of geometric illumination.

Just to mention, a fine textbook relating to these issues is Anne Greenbaum’s Iterative methods for solving linear systems.
————-
@book{Author = {Greenbaum, Anne}, Publisher = {Society for Industrial and Applied Mathematics (SIAM)}, Series = {Frontiers in Applied Mathematics}, Title = {Iterative methods for solving linear systems}, Volume = {17}, Year = {1997}}

]]> By: WWGD: Understanding Google’s Technology Stack http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-16543 WWGD: Understanding Google’s Technology Stack Wed, 24 Dec 2008 16:34:19 +0000 http://michaelnielsen.org/blog/?p=523#comment-16543 [...] Nielsen has a nice post on using your laptop to compute PageRank for millions of webpages. His posts reviews PageRank and how to compute it and shows a short, but reasonably efficient, [...] [...] Nielsen has a nice post on using your laptop to compute PageRank for millions of webpages. His posts reviews PageRank and how to compute it and shows a short, but reasonably efficient, [...]

]]> By: Michael Nielsen http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/comment-page-1/#comment-16542 Michael Nielsen Wed, 24 Dec 2008 16:29:48 +0000 http://michaelnielsen.org/blog/?p=523#comment-16542 Brad - Thanks! Turns out there is a bug in the script I use to convert my source file (which is LaTeX) to WordPress, and that messes up the Python. I'll fix it. I'll be interested to hear how your experiment with Shed Skin goes. Brad – Thanks! Turns out there is a bug in the script I use to convert my source file (which is LaTeX) to WordPress, and that messes up the Python. I’ll fix it. I’ll be interested to hear how your experiment with Shed Skin goes.

]]>