In the makefiles branch of my OpenSSL fork, I’m experimenting with refactoring the existing recursive
make structure into a
Makefile-with-includes structure. I believe this is a critical first step towards improving automated test coverage and instilling a testing culture.
I started down this path when writing a small test in my encode-test branch. Not the most important test to write, but it was something to sink my teeth into to get better acquainted with the code and try out more of my testing ideas. When I tried to change
crypto/evp/encode.c to make the test fail—and it didn’t—I realized that
make test didn’t rebuild the
crypto library on which the test depends. The paper Recursive Make Considered Harmful explains why: The top-level
Makefile recursively invokes
make in the
test/ directory. The
make process parsing
test/Makefile doesn’t have the full dependency graph information, therefore it doesn’t know that it needs to rebuild
libcrypto when rebuilding a test.
The first, most obvious workaround is to always invoke
make && make test. The second, less-obvious way is to use the top-level
GitMake scripts instead;
GitConfigure will compile a single
Makefile out of all of the sub-
Makefiles with the correct dependency info (mostly).
When trying to encourage testing as a regular habit, any friction in the build process works against that goal. The
make && make test solution is inefficient, increasing the length of the feedback loop due to unnecessary work being done (i.e. tons of unnecessary recursive
make processes), and error-prone. The
GitMake solution works, but in both cases, the need to resort to anything other than a straightforward
make test violates the principle of least surprise. This is especially problematic right now, as I’ve got a good number of volunteers ready and willing to help improve OpenSSL’s testing, but who are struggling to understand the code and how they can best contribute. I feel like they’re waiting for me to clear a path, and the first obstacle isn’t the code itself, but the recursive
The end result I’m aiming for is to have a top-level
Makefile include all of the other
Makefiles, as well as dependency files for each object file generated as a side-effect of compilation, so that rebuilding and executing a test is fast and correct. It will remain compatible with
GitMake for those who prefer to use them.
I’m not going to go into excruciating detail here, but suffice it to say this will happen over a long series of changes, and I’ll constantly rebase them on updates to the master branch. Each change will maintain compatibility with the existing recursive
make structure until the final switch is flipped. I’ll take care to support both GNU and BSD
make, and ensure that everything works on Mac OS X, Linux, and FreeBSD at the very least. Now that I’m also running Windows 8.1, I’ll be able to check my changes for Windows compatibility, too.
I wrote a Python script to automate the changes now evident in my
makefiles branch. I developed it to apply the changes one step at a time, and made sure that, after each step, running the script multiple times didn’t alter the
Makefiles after the first run. As I’m now going back to redo many of those changes in a different order, I’m going to be building the script back up again, this time checking it into my personal Google Code repository as update_makefiles.py. Since this is a one-off script that only I’m using, I don’t imagine it’ll go into OpenSSL proper—plus, I’m breaking tradition by using Python instead of Perl for this—but this way folks can take a look at how I’m going about the process if they’re curious.
Even if the core team ultimately decides to reject my experiment, it’s already paid off enormously. First of all, now I know many of the problems that could frustrate people helping with the testing effort. If we can’t fix these issues, we can at least make sure they’re clearly documented. Second, as a result of all this digging into the build system, I’ve already issued a pull request to fix several build issues, particularly those that prevented successful OS X builds, since that’s my home platform. These fixes are worth committing independently of all the other experimentation I’ve been doing; they’re small, noncontroversial, and eliminate a great deal of friction in and of themselves.
Third, I’m having way too much fun with this.