Parallel Cons
Best viewed with ANY browser!
Valid HTML 4.0!



This page is a central location for information about past (and future) attempts at creating a version of the Cons software construction utility, that can build multiple targets in parallel (the sort of functionality supported by the make -j option). This includes copies of the relevant code, email exchanges, and editorial comments (see the bottom of the page). I hope that sharing this information will provide the right insight or inspiration for someone to complete the job.

There have been (to my knowledge) four significant attempts at creating a parallel version of Cons. None of these versions has yet provided a complete solution. So that we have some common terminology, I'm calling them, in order, multi, cons-p, pcons, and pcons-2.3.0:

multi

This is Bob Sidebotham's original attempt at parallelizing Cons. Bob did most of this work in 1996, and then set it aside. Bob created job::serial and job::parallel classes to manage job creation, and tries to enumerate all the dependencies of a target before starting any builds by doing a separate recursive descent. This is a nice idea, but can't accomodate dynamic dependencies (that is, it loses if you have to generate a .c file before you can scan it for #include lines). multi.tar.gz Bob's original tar file he sent me (and another person on the cons-discuss mailing list) containing all his work on parallelizing Cons.
multi.email Bob's original email explaining what's in the multi.tar.gz file.
cons.multi Bob's parallel version of Cons, already extracted from the tar file for easy viewing. (This is slightly different from the version in multi.tar.gz; I pasted back in an enumerate subroutine that was left out.)
multi.diff My diff -c of cons.multi and the original version of Cons on which it was based, so you can see the changes Bob made to implement his parallelism.

cons-p

This is my attempt at picking up and finishing Bob's work on multi. This took place in early 1999 (February through April). I took a lot of wrong turns, among them: merging Bob's two job classes into one (I thought it would be clever to handle serial jobs as an N=1 case of parallelism, but it's much more efficient to keep them separate); breaking signature handling; spending too much energy on a complicated scheme to enumerate dependencies up-front and still handle dynamic dependencies... I added some documentation, though, and extended Bob's job classes to handle multi-line commands. cons-p.tar.gz An archive of the last three versions of cons-p, plus the test cases (shell scripts in the manner of versions 2.X of the cons-test suite) for which I was trying to get cons-p to work.
cons-p.email Some fairly lengthy email exchanges among Bob, Rajesh and myself as I struggled to get cons-p working.
cons-p.diff diff -c output of the baseline version of Cons I used (1.6b1) and the last version of cons-p I was working on before setting it aside.

pcons

This is John Erickson's parallel Cons from August 1999. John took a different approach from Bob's, spawning parallel jobs within the recursive dependency descent for a target. This leads to an elegantly concise set of changes that don't give ideal parallelism (you can't keep N compiles running simultaneously if they're spread across multiple targets), but do an excellent job of keeping each target as parallel as possible. The only other drawback is that John didn't keep multi-line commands working; you had to separate commands with && instead of \n. pcons.announce John's original announcement to the Cons mailing list about pcons, with his attached patch.
pcons.diff John's original patch, as attached to his announcement mail.
pcons John's patch applied to Cons 1.6b1 (the current version at the time).
pcons.email An email exchange between John and myself about pcons, differences between his and Bob's approaches, and how we might take it to the next step. We also discussed the idea of using threads.

pcons-2.3.0

This is Thomas Gleerup's merging and updating of John Erickson's pcons change with Cons version 2.3.0. This has the same minor drawback as Erickson's (very useful but slightly imperfect parallelism), and half-supports multi-line commands by automatically converting newline separators into &&. Note that the Cons 2.3.0 version on which this is based has known problems with multi-target dependencies, and the Install, InstallAs, and Install-Local methods. pcons-2.3.0.announce Thomas's original announcement to the Cons mailing list about his work.
pcons-2.3.0 Thomas' version of pcons, based on Cons 2.3.0. Note that this is actually slightly updated from the version in his announcement, adding the automatic conversion of \n command separators to && in multi-line commands.
pcons-2.3.0.diff diff -c output of pcons-2.3.0 against the 2.3.0 version of Cons that on which it was based.

EDITORIAL COMMENTS:

After re-familiarizing myself with these attempts, I think each has some good ideas that could be pulled together into a good implementation:





Feel free to send me something at knight@baldmt.com.
Last modified: 8 November 2001