subdirmk - ergonomic preprocessing assistant for non-recursive make

Problem

Peter Miller's 1997 essay Recursive Make Considered Harmful persuasively argues that it is better to arrange to have a single make invocation with the project's complete dependency tree, rather than the currently conventional $(MAKE) -C subdirectory approach.

These problems are not theoretical for me. In the Xen Project we use recursive make and sadly suffer from occasional concurrency bugs. In my work on secnet (the currently rather unproductised greenend.org.uk vpn program) I have been frustrated by unreliability of the build system (which is developing fairly rapidly, as I overhaul secnet) .

However, actually writing a project's build system in a non-recursive style is not very ergonomic. I was complaining about this in the pub a week or two ago. Accepting my challenge, Mark Wooding demonstrated a proof of concept showing that it was possible to do better. I thought I had a better approach so I took his code and I ran with it.

Solution

I stole the sigil & to mean "directory local".

The result is subdirmk, which is an arrangment for preprocessing your per-subdirectory makefiles (really, makefile fragments) to prefix file and variable names with the name of the fragment's directory.

It's designed to be incorporated into your project via git  subtree (which is excellent and not to be confused with the awful git submodule). subdirmk is implemented in (very conservative) perl, which is ubiquitious nowadays. It expects you to be using autoconf, but not automake (I am not much of a fan of automake; I don't think its extra complexity buys much, especially now that make dist can be replaced with git archive).

I'm very pleased with the result. It's small and pleasant to use. Although it's not stabilised, I have already adopted it for secnet. But if you want to see an example of its use you probably want to look at the included toy example project (which is also used for subdirmk's self-tests).

There are a variety of &-escapes. The complete syntax etc. is documented in the current README.

Status

As I say, this is not yet stabilised. If you adopt it now, you should be prepared to make systematic changes to your project when you merge new versions of subdirmk. (git-subtree makes this fairly easy and I would be happy to advise eg with git-fu.)

Questions

I have a few things you might like to bikeshed:

Is my set of substitution syntaxes the best set? See "Tables of file reference syntaxes" and "Substitution Syntax" in the README. I'm particularly unsure about &=_ and &=/ (but there has to be a way to ask for these as they are the most primitive things the filter can provide) and the choice of ! in &!.

Please comment on my choices of filenames and extensions. (The input files to subdirmk must end in .mk, rather than being something like .mk.in.sd, so that editors choose makefile syntax mode.)

Should I introduce &$VARIABLE as a shorthand for $(&VARIABLE) ? If so I guess characters matching \w are part of the variable name.

The &TARGETS_foo thing is a bit of a wrinkle, especially the way that mentioning it in a #-comment has a side effect. (See line 22 in the provided clean.sd.mk for an example.)

Makefile fragment inclusion order is often relevant. Should I add a Perdir-top.sd.mk which works like Perdir.sd.mk but is included first rather than last?

I do not currently have a convenient way to write a reference to the build or install toplevel which is relative to one of the build subdirectories. This is sometimes needed for symlinks, or if a command wants to cd to the subdirectory and then refer to files elsewhere. autoconf's @top_builddir@ and @abs_top_srcdir@ and so on can be used. This seems rare enough that this is OK.

GNU make's metaprogramming facilities are rather awkward. You end up expanding macros with $(eval $(call ... )) and you have to $-double everything in the macro. (Just using $(call ...) often doesn't work because that construct cannot generate multiple separate rules etc.) I am tempted to add something to subdirmk to make that easier. It's not so simple because subdirmk's filtering engine is purely textual and does not understand make syntax, and runs before make so there is no access to make variables. (And I certainly don't want to add a third variable namespace to all the makefiles.) Also any facility of this kind would be available only in .sd.mk files and not .mk.in ones (but maybe that doesn't matter).

Edited 2019-11-25 15:19 to fix a few typos and make a minor clarification