Added “MapReduce” to the “Software Innovations” list
Ken Krugler’s recent blog said that my article of The Most Important Software Innovations was “very good”, but he was surprised that I hadn’t included MapReduce as an important software innovation. Basically, MapReduce makes writing certain kinds of programs that process huge amounts of data, on vast distributed clusters, remarkably easy and efficient. (Wikipedia explains MapReduce, including links to alternative implementations like the open source Hadoop.)
It’s not because I didn’t know about MapReduce; I read about it almost immediately after it got published. I thought it was very promising, and even forwarded the original paper to some co-workers. I think MapReduce is especially promising because, now that we have cheap commodity computers, having a way to easily exploit their capabilities is really valuable. But even with something this promising, I didn’t want to add it to my list of innovations right away - after all, maybe after a little while it turns out to be not so helpful.
Currently, there’s aren’t many who have Google-sized clusters of computers available. But it’s clear that this approach is useful in many other circumstances as well. It’s new, but I think it’s stood the test of time enough that it’s a worthy addition… so I’ve just added it.
One interesting issue is that the MapReduce framework is itself built primarily on the “map” and “reduce” functions, which are far, far older. So, is MapReduce really a new idea, or is it just a high-quality implementation of an old idea? I’ll accept that it’s a new idea, but that can be difficult to judge. This judgment doesn’t really matter unless you think software patents are a good idea (since every software patents in theory prevents progress for 20 years). But I think it’s quite clear that software patents are a foolish idea, and it’s clear that others have come to the same conclusion. Eric S. Maskin, an economist who has long criticised the patenting of software, recently received the 2007 Nobel Prize for Economics. Here’s a nice quote: “… when patent protection was extended to software in the 1980s, […] standard arguments would predict that R&D intensity and productivity should have increased among patenting firms. Consistent with our model, however, these increases did not occur.” Someone who correctly predicted that software patents were harmful to innovation just received a Nobel prize. I hope to someday see people receive other prizes because they ended software patenting in the United States.
path: /misc | Current Weblog | permanent link to this entry