In connection with your comments about writing papers using wikis, I’ve collaborated on several papers (and a book) using version control systems. I was initially skeptical, but found this incredibly useful. I’ve never written a paper with a wiki, but they share a lot in common with version control systems, and I expect would have many of the same advantages.

Over a year ago, I edited a sentence in the Wikipedia entry for “invertible matrix” when one of my students was confused by the phrase “as a rule of thumb, almost all square matrices are invertible.” The Wikipedia article went on to describe how the set of non-invertible matrices has Lebesgue measure zero. On the discussion page, I explained the “bug” and pointed out that the fact about Lebesgue measure is also true in the sense of Baire category. Months later, I returned to the page and noticed that the old bugs had been fixed. Moreover, someone had noticed my remark on Baire category and included it in the article.

This type of interaction seems to be close to what you’re describing. The discussion
http://en.wikipedia.org/wiki/Talk:Invertible_matrix
is similar to bugtracking, but it is far less task-oriented. Perhaps Polymath1Wiki would benefit from making better use of its discuss page. This seems better organized than the “100 block” method. I can definitely imagine a paper written through a multitude of individual contributions and edits, as well as contributions made many small collaborations taking place behind the scenes in the discussion page.

Although closer to the ideal, the Wikipedia model is not a perfect fit. For example, there appear to be major issues with large scale edits and maintaining any kind of consistency. (See
http://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment/Maths,_science,_and_technology) This is a problem in open source software too, but there is always option of creating a version (n+1).0 when necessary. Perhaps this is a mandatory price to pay in order to reap the benefits of massive collaboration.

Certainly, wikis are very useful for this kind of project. The reason I don’t mention them in this article is because they were already extensively used in the project. See http://michaelnielsen.org/polymath1

I do think issue trackers have some interesting features not mirrored in wikis. In particular, their workflow is organized around lists of issues in a way that wikis seldom are. Wikis can, of course, be used in a way that mimics that, but the issue tracking software really makes it very easy to fall into that pattern, and moreover provides other features (like dependency tracking) useful in developing ideas.

Perhaps you already had this model in mind as it has many of the features you describe. The interaction of the three components — article, discussion, edit history — separates what is known (article) from how it is known (discussion/history). Also, the discussion often takes a form similar to bugtracking. Moreover, the “narrative structure” is much closer to that of a mathematical collaboration.