Comments on: Write your first MapReduce program in 20 minutes http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/ Tue, 09 Mar 2010 22:14:18 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Jonathan Stray » Why We Need Open Search, and How to Make Money Doing It http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-26173 Jonathan Stray » Why We Need Open Search, and How to Make Money Doing It Sun, 27 Sep 2009 09:51:16 +0000 http://michaelnielsen.org/blog/?p=529#comment-26173 [...] data processing is now well understood. To be precise, I want an open search company that sells map-reduce access to their index. Map-reduce is a standard framework for breaking down large computational [...] [...] data processing is now well understood. To be precise, I want an open search company that sells map-reduce access to their index. Map-reduce is a standard framework for breaking down large computational [...]

]]> By: Michael Nielsen » Consistent hashing http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-23140 Michael Nielsen » Consistent hashing Thu, 04 Jun 2009 03:06:38 +0000 http://michaelnielsen.org/blog/?p=529#comment-23140 [...] interested in distributed dictionaries is because they’re used as input and output to the MapReduce framework for distributed computing. Of course, that’s not the only reason distributed [...] [...] interested in distributed dictionaries is because they’re used as input and output to the MapReduce framework for distributed computing. Of course, that’s not the only reason distributed [...]

]]> By: Michael Nielsen http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-20406 Michael Nielsen Wed, 15 Apr 2009 12:02:31 +0000 http://michaelnielsen.org/blog/?p=529#comment-20406 Asokan - Thanks, fixed. Asokan – Thanks, fixed.

]]> By: Asokan Pichai http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-20381 Asokan Pichai Wed, 15 Apr 2009 04:40:08 +0000 http://michaelnielsen.org/blog/?p=529#comment-20381 The Dave Spencer link is wrong. AFAICS, it says tropo instead of chencer The Dave Spencer link is wrong. AFAICS, it says tropo instead of chencer

]]> By: Neal Richter http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-18123 Neal Richter Sun, 22 Feb 2009 03:17:13 +0000 http://michaelnielsen.org/blog/?p=529#comment-18123 Obviously Dean and Ghemawat are due some credit here, yet map and reduce have been around in languages like Lisp for approx 25 years prior to their paper. There were even companies/groups that parallelized their Lisp implementations in the 1980s: http://ieeexplore.ieee.org/iel3/1244/519/00010373.pdf?arnumber=10373 http://search.barnesandnoble.com/Parallel-LISP/Takatoshi-Ito/e/9780387527826 http://www.springerlink.com/content/k560u307713j57r4/ Did Dean and Ghemawat 'finish the job' and do a great implementation at exactly the right time and at the right company to start a distributed computing revolution? Absolutely and congrats to them for that. Yet when first introduced to it, it was completely obvious to me and others who learned the map and reduce lisp/scheme primitives as undergrads. And 'difference' in their methodology is trivial.. of course this does describe the best revolutions.. they are all trivial in retrospect. It's also worth reading David DeWitt's savaging of MapReduce to cure you of any further illusion of it being 'invented' by Dean and Ghemawat. (I think he goes way to far.. modern MapReduce is here to stay - we're not going back to RDBMs systems for distributed computing anytime soon) http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html Obviously Dean and Ghemawat are due some credit here, yet map and reduce have been around in languages like Lisp for approx 25 years prior to their paper. There were even companies/groups that parallelized their Lisp implementations in the 1980s:

http://ieeexplore.ieee.org/iel3/1244/519/00010373.pdf?arnumber=10373
http://search.barnesandnoble.com/Parallel-LISP/Takatoshi-Ito/e/9780387527826
http://www.springerlink.com/content/k560u307713j57r4/

Did Dean and Ghemawat ‘finish the job’ and do a great implementation at exactly the right time and at the right company to start a distributed computing revolution?
Absolutely and congrats to them for that.

Yet when first introduced to it, it was completely obvious to me and others who learned the map and reduce lisp/scheme primitives as undergrads. And ‘difference’ in their methodology is trivial.. of course this does describe the best revolutions.. they are all trivial in retrospect.

It’s also worth reading David DeWitt’s savaging of MapReduce to cure you of any further illusion of it being ‘invented’ by Dean and Ghemawat. (I think he goes way to far.. modern MapReduce is here to stay – we’re not going back to RDBMs systems for distributed computing anytime soon)
http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html

]]> By: JK http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-18054 JK Thu, 19 Feb 2009 01:22:40 +0000 http://michaelnielsen.org/blog/?p=529#comment-18054 Please, don't take that too serious :) I just thought of it because I'd instantly rewrite that line to > filenames = map(partial(os.path.join, 'text'), flist) when in the mood of doing things functional. That, however, doesn't save many characters and additionally requires an import to work. In any case, I really appreciate your article and I'm looking forward to more of that quality. Oh, and let me point to the Disco project at http://discoproject.org/ which uses an Erlang server and a client library to write applications in Python. Their API for the map and reduce functions is a little different, though (which is interesting to note as there are similar implementation approaches that work). Please, don’t take that too serious :) I just thought of it because I’d instantly rewrite that line to

> filenames = map(partial(os.path.join, ‘text’), flist)

when in the mood of doing things functional. That, however, doesn’t save many characters and additionally requires an import to work.

In any case, I really appreciate your article and I’m looking forward to more of that quality.

Oh, and let me point to the Disco project at http://discoproject.org/ which uses an Erlang server and a client library to write applications in Python. Their API for the map and reduce functions is a little different, though (which is interesting to note as there are similar implementation approaches that work).

]]> By: Michael Nielsen http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-18050 Michael Nielsen Wed, 18 Feb 2009 22:17:08 +0000 http://michaelnielsen.org/blog/?p=529#comment-18050 Thanks JK, I'll look into it. Thanks JK, I’ll look into it.

]]> By: JK http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-18032 JK Wed, 18 Feb 2009 15:12:34 +0000 http://michaelnielsen.org/blog/?p=529#comment-18032 > filenames = [ os.path.join(”text”, f) for f in flist ] Heh, you could even use `functools.partial` and then MapReduce to spread that over a whole bunch of clustered machines, if I got MR right from this well-explanatory article :D > filenames = [ os.path.join(”text”, f) for f in flist ]

Heh, you could even use `functools.partial` and then MapReduce to spread that over a whole bunch of clustered machines, if I got MR right from this well-explanatory article :D

]]> By: Matt Williams http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-17142 Matt Williams Sun, 18 Jan 2009 16:55:29 +0000 http://michaelnielsen.org/blog/?p=529#comment-17142 I've provided a high level overview of MapReduce in my brand spanking new blog here: http://wordflows.com/matt/2009/01/18/understanding-mapreduce/ Feedback is appreciated, as I've only just started blogging today! I am in agreement that it is far from a new concept. Unless i am missing something obvious in their whitepaper, its a technique we have been using in distributed systems for at least a decade. I’ve provided a high level overview of MapReduce in my brand spanking new blog here: http://wordflows.com/matt/2009/01/18/understanding-mapreduce/

Feedback is appreciated, as I’ve only just started blogging today!

I am in agreement that it is far from a new concept. Unless i am missing something obvious in their whitepaper, its a technique we have been using in distributed systems for at least a decade.

]]> By: Understanding Map/Reduce | Matt Williams http://michaelnielsen.org/blog/write-your-first-mapreduce-program-in-20-minutes/comment-page-1/#comment-17141 Understanding Map/Reduce | Matt Williams Sun, 18 Jan 2009 16:48:59 +0000 http://michaelnielsen.org/blog/?p=529#comment-17141 [...] few references you may find interesting:   Google Research Publication Write your first Map/Reduce function in 20 mins Misconceptions about [...] [...] few references you may find interesting:   Google Research Publication Write your first Map/Reduce function in 20 mins Misconceptions about [...]

]]>