These are a few examples of my coding and other technical work. Most of these examples are from projects in the past couple of years, since most of my work before then was closed-source development for Northrop Grumman and Google. The ordering is roughly reverse chronological order, though some more prominent projects are bumped up a little.
Many of the works listed below also appear on my Open Source profile pages:
This a framework I began developing in the summer of 2016 for writing modular, well-documented, and well-tested Bash script applications—in other words, "to make the right thing the easy thing," even for Bash scripts! It uses the Bash Automated Testing System (Bats) and contains an efficient and powerful library of Bats test assertions and other Bats helper functions.
The experience I've gained working with Bash and Bats as part of this project enabled me to improve Bats performance by more than 2x on UNIX, and by 10x-20x on Windows, as well as inspired me to submit my request to become a maintainer of Bats.
At the request of 18F Handbook designer and curator Andrew Maier, I wrote this plugin for team's server. Whenever a user flags important information shared in a Slack discussion with a specific emoji reaction, Hubot will file a GitHub issue with a pointer to the information. Afterwards, whoever is responsible for cultivating the Handbook can triage the issues and make sure important information is not lost to the sands of Slack history.
This plugin makes it easy for anyone, not just technical people, to make material contributions to the organizational knowledge base. Though originally developed for the 18F Handbook, it's configurable to support any Slack team or set of GitHub repositories. It was also a great example of writing a small, thoroughly-tested distributed system that made for a great instructional example, inspiring the Unit testing in Node.js tutorial, described below.
It also provided the opportunity to make an upstream contribution to
hubot-slack integration package (written in
CoffeeScript) to better support responding to
events. This support first appeared in version
4.1.0 via pull request #360, and was
improved in version
4.2.0 via pull request #363. I also
hubot-slack-reaction-example program to illustrate
its basic usage.
After completing the slack-github-issues Hubot plugin, I felt it made for a great, real-world example upon which to base a unit testing tutorial, since the tests comprised the gamut from small (unit) to medium (integration) to large (system).
The result was this tutorial, which builds up from testing straightforward (yet detailed) configuration logic, to external service API wrappers, to distributed event orchestration, to a small set of end-to-end tests. The instructional website is based on the 18F Guides Template described below.
I've only attempted to give this workshop once to date, to an audience of mostly Windows developers from the Tennessee Valley Authority in Chattanooga. The experience made it clear I have a lot to learn about designing and leading workshops, but the difficulty the developers had installing Ruby on Windows also led me to start my go-script-bash project shortly thereafter.
The 18F Handbook is the team's central repository of organizational knowledge. The descendant of my first 18F project, The Hub, it benefits from the expertise of actual frontend designers and developers and sports many of the Jekyll features I developed during the course of my work on The Hub and the 18F Guides. The cultivation of its material is greatly augmented by the slack-github-issues Hubot plugin that I developed specifically for the purpose.
Its primary researchers/designers/content architects were Andrew Maier and Melody Kramer, building on previous work led by Colin MacArthur and Ethan Heppner as part of the Documentation Working Group efforts to improve the 18F onboarding process.
After my colleague Aidan Feldman integrated the
engine into The Hub, using a document corpus
generated by his
jekyll_pages_api Jekyll plugin, I eventually
set about learning to improve its performance. Eventually I came up with the
jekyll_pages_api_search plugin, which produces the search
corpus as the site is generated on the server, rather than in the user's
browser, and provides Liquid and Sass directives to include a standard
To address Andrew Maier's accessibility concerns, I created a default search results page instead of relying on a drop-down list of results. As a life-long Vim user, I made sure the widget and results page were completely navigable by keyboard shortcuts. This plugin became a standard feature of both the 18F Guides style and the 18F Handbook.
This documents a set of principles, practices, idioms, and strategies pertaining to automated software testing and its adoption.
This was the main artifact of the 18F Testing Grouplet, which I co-led with Alison Rowland, and also featured Josh Carp, Moncef Belyamani, and Shawn Allen.
This is a guide to organizing grassroots initiatives for organization-wide improvement and enablement, a.k.a. "Grouplets".
This was the main artifact of the 18F Working Group Working Group, which I co-led with Gray Brooks, and also featured Nick Bristow and Nick Brethauer.
A core initiative of the Documentation Working Group that I led, the Guides are a series of documents curated by Grouplets and other groups to cultivate specialized domain knowledge and disseminate it to the rest of the organization. Most of the Guides were also available to the general public, as part of our mission to work in the open, share knowledge, and provide tools and practices for other agencies to adapt to their needs.
The Guides Template is both a template and a tutorial to writing a Guide that leads the reader through the entire process, from cloning the Guides Template repository, to replacing its content, to publishing a new Guide.
The Guides concept was so compelling and Guides Template product so effective that dozens of Guides proliferated thanks to word-of-mouth. The underlying publishing tools are Jekyll, a static site generator already in use by much of the team (and the Web); GitHub, which the team used exclusively for source control; 18F Pages, a very lightweight server inspired by GitHub Pages; and the Nginx web server.
A core part of the Guides Template is the guides_style_18f Jekyll plugin, described below.
See the image from jekyll_pages_api_search to see the Guides template and its search mechanism in action.
After being notified that our organization couldn't use GitHub Pages to publish canonical U.S. federal government information anymore, I wrote and launched the first version of this lightweight server that listens for GitHub webhooks and rebuilds Jekyll sites just over a day later.
Running on a single Amazon EC2 instance behind an Nginx web server, with no live database component, it's as secure as it is cheap as it is easy to use to publish content. It allowed for the proliferation of 18F Guides, was the original platform for the 18F Handbook, and was used to initial development and launch of the U.S. Web Design Standards (USWDS).
Alison Rowland helped me greatly by reviewing many
pages-server pull requests and introducing me to the Mocha
testing framework and Chai assertion library. Julia Elman from the USWDS
team provided tons of great feedback, bug reports, and feature requests.
Promises to structure
the asynchronous logic, after I'd originally used a style resembling
ControlFlow class, only having just begun
During the course of expanding the capabilities of the server, I also made a small contribution to the Jekyll project (PR #3823).
A core part of the Guides Template is the
Jekyll plugin, which packages style, structure, and data management tools so
that users can focus exclusively on their content rather than data massaging
and presentation mechanisms.
The original style was taken from the 18F Agile Guide, which was built upon the DOCter template produced by the Consumer Financial Protection Bureau (CFPB). Nick Bristow did a lot to ensure the style met Section 508 accessibility standards.
See the image from jekyll_pages_api_search to see the plugin's style and its search mechanism in action.
I discovered this project, originally called
while implementing authentication for the Hub. As a
static site generated using Jekyll, it contained no server-side component
to control authentication; but using the
conjunction with the Nginx web server fit the bill.
Thanks to help from the maintainer, Jehiah Czebotar, I was able to
introduce a new
Provider interface into the code so that I
could add support for 18F's (now defunct) MyUSA auth provider. Since then,
others have added support for GitHub, LinkedIn, and more, prompting Jehiah
to change the name to
oauth2_proxy. I added a number of other
features as well, and even managed to test it on Plan 9.
The predecessor of 18F Guides, the Team API, and the 18F Handbook, The Hub was intended to be a prototype to cultivate and disseminate important team-wide information, as well as show the connection between people, projects, and skill sets. It also tried to introduce the "snippets" practice to the team, though it didn't stick.
Aidan Feldman was my closest collaborator for most of the project's life. Michelle Hertzfeld took the first pass at the frontend design, and Jen Thibault took the second. The Hub was the first central artifact of the Documentation Working Group, prior to the introduction of 18F Pages/18F Guides.
The Team API Ruby gem contains the team metadata graph-building logic extracted from the Hub.
The Team API server is a Node.js program that listens for GitHub webhooks indicating updates to .about.yml files, then processes the updates to generate a graph of people, projects, and skill sets exposed as a set of JSON endpoints. It was a little fragile, but worked as a proof-of-concept, and even supplied live updates to the 18F Dashboard for a while.
.about.yml mechanism allows a project to publish and
maintain YAML metadata that can be easily maintained by project owners, that
is visible and accessible to interested parties, and that can be harvested
and processed by tools and automated systems.
The Team API was the original consumer of this information, which in turn provided data to The Hub and the 18F Dashboard. The hope was that by making it easy for projects and grouplets to maintain this data, it would be easy anyone inside or outside the organization to have visibility into it people, structure, projects, and activity, and how we worked.
Recently, the TODO Group, a consortium of tech companies helping develop
Open Source standards, expressed an interest in the
format, inspiring my ".about.yml background" blog post.
.about.yml was spearheaded by both the 18F Testing Grouplet
(in pursuit of building a "code health" dashboard, never entirely
realized) and the Working Group Working Group (to facilitate transparency
and discovery of Grouplets). Alison Rowland got the code started and
did most of the heavy lifting in the early days.
This was the blueprint for the 18F Pages/18F Guides server. I'd originally attempted to use Docker for this, but at the last minute was instructed not to use it. Fortunately it was a relatively straightforward process to migrate the Dockerfiles to Bash scripts in short order.
I include this project for two reasons. One, it ties together many of the servers listed above, and then some. This was the beginning of the "tools" package I'd hoped to package with our practices (via the Guides) to promote knowledge sharing and enablement across federal agencies. Two, I haven't completely abandoned this project, and intend to revisit and productionize it one day, as I believe it could be useful not only to the federal government, but to many organizations that need a kick start to their knowledge sharing practices.
Signs and authenticates HTTP requests based on a shared-secret HMAC signature. Developed and tested in parallel across languages to ensure interoperatability.
A server that routes authenticated requests to multiple authentication servers based on the presence of specific headers or cookies in a request.
Proxy server that signs and authenticates HTTP requests using an HMAC signature; uses the hmacauth Go package.
heartbleed_test.c in response to those who claimed
that the OpenSSL "Heartbleed" bug was essentially untestable and
build.sh script that accompanies it
automatically downloads the source for OpenSSL-1.0.1-beta1 (in which the
bug was introduced) and OpenSSL-1.0.1g (in which the bug was fixed),
patches the test into each copy of the source, builds each version and runs
the test in each. The fixed version produces no output, indicating success;
the "bleeding" version prints:
TestDtls1Heartbleed failed: expected payload len: 0 received: 1024 sent 26 characters "HEARTBLEED " received 1024 characters "HEARTBLEED \xde\xad\xbe\xef..." ** TestDtls1Heartbleed failed **
The structure of the test code follows what I call the "Pseudo x-Unit
Pattern", which I originally stumbled upon when writing the goto fail unit/regression test. This test was
eventually adapted to the OpenSSL coding style (instead of the
Google coding style used
in the original) and submitted as
I wrote several blog posts about Heartbleed, Testing on the Toilet Episode 330: "While My Heart Gently Bleeds", and the "Goto Fail, Heartbleed, and Unit Testing Culture" article for Martin Fowler.
For a brief time, I offered to help the OpenSSL team improve its automated testing, and they accepted. However, after joining the government in November 2014, I became consumed with my new job and never made the time to revisit my OpenSSL efforts. Since then, heartbeat support has been removed, along with the test. (See pull requests #1669 and #1928.)
Before the Heartbleed bug was discovered, I wrote
tls_digest_test.c and an accompanying patch for Apple's Secure
Transport library in response to those who claimed that the
fail bug was essentially untestable and unavoidable. The
Security-55471-bugfix-and-test.tar.gz bundle contains a
build.sh script that downloads the buggy version of the Secure
Transport code from Apple, applies the patch, builds just the affected code
and runs the test without requiring the full set of dependencies needed to
build the entire library. It requires OS X and Xcode. Changing the code to
reintroduce the extra
goto fail statement into the handshake
algorithm triggers a test failure:
Executing Security-55471/libsecurity_ssl/build/libsecurity_ssl.build/Debug/libsecurity_ssl.build/Objects-normal/x86_64/tls_digest_test TestHandshakeFinalFailure failed: expected FINAL_FAILURE, received SUCCESS 1 test failed Test failed
This was the first test I wrote in which I stumbled upon what I call the
"Pseudo x-Unit Pattern", in which the code resembles a test using an
xUnit-based framework without actually using a framework at all. I also
wrote a guest blog post for the (now-defunct) AutoTest Central, a flurry of
goto fail posts on this site, gave a lightning talk at an
Automated Testing Boston Meetup event, wrote Testing on the Toilet Episode
327: "Finding More Than One Worm in the Apple", and compiled the lightning
talk into the "Finding More Than One Worm in the Apple" article for ACM
Queue. The Queue editors then submitted the article for
publication in the July 2014 issue of Communications of the ACM,
the full text of which you can access via the "Finding More Than One Worm in
the Apple" page on this site.
I worked on the Instant Indexing Tier, a distributed system that indexes all new content on the World Wide Web within minutes. Developed features for removing obsolete content, spam blocking, taking inventory of signals imported from other indexing systems, and reindexing to take advantage of rapid signal changes. Refactored a core indexing component to achieve greater-than-expected code reuse across multiple Google websearch systems. Deployed binaries, monitored and resolved issues in close collaboration with our Site Reliability Engineers—I refer to this as "doing DevOps before realizing it had a name".
Pyfakefs is a fake file system module for Python that I started back in 2006, announcing it internally at Google in an early Testing on the Toilet episode. My motivation was less to isolate code under test from interacting with the filesystem than to avoid having a plethora of little test files littering the code base. The isolation aspect was a nice side-effect, however; and having the test "files" defined in the test case itself made it easier to write, read, and maintain large numbers of file system manipulation tests. (Our overloaded Perforce server was also happy to manage just one test file, too.)
Over time, a number of contributors expanded the module far beyond what I'd
intended; but thanks to rigorous review and unit testing, it both grew to
fulfill many people's needs and remained stable such that existing users
didn't break. Well, one specific breakage comes to mind, and that was
because the code's interface became more strict; the client only had to add
"w" parameter to an
open() call and all
was well. (Every other client opening a file for writing already had this
By the time I left, Pyfakefs was used in over 900 tests inside Google; the current GitHub page (linked above) reports over 2,000. It's also now available on the Python Package Index (PyPI), thanks to the current maintainers who stepped up after I abandoned the project upon leaving Google.
A while back, I added the
with statement to the Python language
in version 2.6. Being primarily a C++ programmer at the time, and a firm
believer in the Resource Acquisition Is Initialization (RAII) idiom, the
idea of adding a mechanism to support the idiom in Python appealed to me.
With hints from Neal Norwitz, I produced the patch to add the
with statement, and Guido van Rossum cleaned it up and
submitted it. Then I had to write a bunch of Python code before Google ever
upgraded to Python 2.6.
In 2005 Google's code was largely untested, untestable, and lacking tools to fix this. I helped lead the successful effort to drive automated testing adoption, as illustrated in many of my presentations and quoted in Chapters 10 and 21 of The DevOps Handbook by Gene Kim et al. Specific initiatives included:
Jun 2007 – Jan 2009: Test Mercenaries. Joined, then managed a team of developers dedicated to improving design and testing techniques while embedded in other teams for months at a time.
Sep 2005 – Sep 2007: Testing Grouplet. Joined, then led a grassroots, 20%-time volunteer effort to improve testing techniques and tools. Drove initiatives such as the weekly Testing on the Toilet newsletter and Test Certified (a testing plan for individual teams). Partnered with tools teams, culminating in the Test Automation Platform, which tests every committed change within minutes.
Sep 2007 – Nov 2008: Fixit Grouplet. Recruited all of Google engineering for one-day, intensive sprints of code reform and tool adoption. Organized four company-wide Fixits, the last involving 100+ volunteers in 20+ offices in 13 countries. Led the grouplet to document its collective experience in order to make Fixits of all sizes easier to run and more effective.
Contributed to the backend of SrcFS, distributed file system used to scale Google's source repository. Worked on efforts to improve build speed and scale Perforce usage, and Testing Grouplet-driven tasks.
Worked on rendering of radar data and Vector Product Format (VPF) nautical chart data in support of shipboard navigation and port monitoring systems for the U.S. Coast guard. Designed and implemented a stable, efficient replacement module for a port monitoring system to render data from multiple radars to multiple X-based clients. Developed a spatial query engine, algorithms, and data structures to improve rendering of charts (90s to 5s) and add features.