In the makefiles branch of my OpenSSL
fork, I’m experimenting with
refactoring the existing recursive make structure into a
Makefile-with-includes structure. I believe this is a critical first step
towards improving automated test coverage and instilling a testing culture.
Motivation
I started down this path when writing a small test in my encode-test
branch. Not the most
important test to write, but it was something to sink my teeth into to get
better acquainted with the code and try out more of my testing ideas. When I
tried to change crypto/evp/encode.c to make the test fail—and it didn’t—I
realized that make test didn’t rebuild the crypto library on which the test
depends. The paper Recursive Make Considered
Harmful explains why: The
top-level Makefile recursively invokes make in the test/ directory. The
make process parsing test/Makefile doesn’t have the full dependency graph
information, therefore it doesn’t know that it needs to rebuild libcrypto when
rebuilding a test.
The first, most obvious workaround is to always invoke make && make test. The
second, less-obvious way is to use the top-level GitConfigure and GitMake
scripts instead; GitConfigure will compile a single Makefile out of all of
the sub-Makefiles with the correct dependency info (mostly).
When trying to encourage testing as a regular habit, any friction in the build
process works against that goal. The make && make test solution is
inefficient, increasing the length of the feedback loop due to unnecessary work
being done (i.e. tons of unnecessary recursive make processes), and
error-prone. The GitConfigure/@GitMake@ solution works, but in both cases, the
need to resort to anything other than a straightforward make test violates
the principle of least
surprise. This
is especially problematic right now, as I’ve got a good number of volunteers
ready and willing to help improve OpenSSL’s testing, but who are struggling to
understand the code and how they can best contribute. I feel like they’re
waiting for me to clear a path, and the first obstacle isn’t the code itself,
but the recursive make structure.
Refactoring Steps
The end result I’m aiming for is to have a top-level Makefile include all of
the other Makefiles, as well as dependency files for each object
file generated as a side-effect of compilation, so that rebuilding and executing
a test is fast and correct. It will remain compatible with GitConfigure and
GitMake for those who prefer to use them.
I’m not going to go into excruciating detail here, but suffice it to say this
will happen over a long series of changes, and I’ll constantly rebase them on
updates to the master branch. Each change will maintain compatibility with the
existing recursive make structure until the final switch is flipped. I’ll take
care to support both GNU and BSD make, and ensure that everything works on Mac
OS X, Linux, and FreeBSD at the very least. Now that I’m also running Windows
8.1, I’ll be able to check my changes for Windows compatibility, too.
I wrote a Python script to automate the changes now evident in my makefiles
branch. I developed it to apply the changes one step at a time, and made sure
that, after each step, running the script multiple times didn’t alter the
Makefiles after the first run. As I’m now going back to redo many
of those changes in a different order, I’m going to be building the script back
up again, this time checking it into my personal Google Code repository as
update_makefiles.py.
Since this is a one-off script that only I’m using, I don’t imagine it’ll go
into OpenSSL proper—plus, I’m breaking tradition by using Python instead of Perl
for this—but this way folks can take a look at how I’m going about the process
if they’re curious.
Side Effects
Even if the core team ultimately decides to reject my experiment, it’s already paid off enormously. First of all, now I know many of the problems that could frustrate people helping with the testing effort. If we can’t fix these issues, we can at least make sure they’re clearly documented. Second, as a result of all this digging into the build system, I’ve already issued a pull request to fix several build issues, particularly those that prevented successful OS X builds, since that’s my home platform. These fixes are worth committing independently of all the other experimentation I’ve been doing; they’re small, noncontroversial, and eliminate a great deal of friction in and of themselves.
Third, I’m having way too much fun with this.