Mike Bland

Music student, semi-retired programmer, and former Googler

Fixits, or I Am the Walrus

Fixits, when engineers throughout Google focus on back-burner issues for a day--and, more importantly, Fixit schwag

- New York
Tags: Fixit Grouplet, Fixits, Google, grouplets, schwag, TAP, Testing Grouplet, The Beatles
Discuss: Discuss "Fixits, or I Am the Walrus" on Google+

Traditionally, Fixits are days when ordinary engineers with an idea and a sense of mission recruit all of Google engineering for one-day, intensive sprints of code reform and tool adoption. Nowadays Fixits come in many shapes and sizes, from single-team “fixlets” to company-wide events, from one-day to one week to one quarter. When done right, rather than being a distraction from normal work, Fixits are incredibly powerful vehicles not just for fixing a problem, but for generating and spreading knowledge, driving the adoption of new tools, and making Google engineering feel like a cohesive culture despite its size.

I organized four company-wide Fixits, two pure Testing Fixits and two that were more tools-related, the last involving 100+ volunteers in 20+ offices in 13 countries. I also led the Fixit Grouplet from Fall 2007 until Fall 2008, inheriting it from David Kramer and passing it on to Lisa Carey and Jonas Klink. Today the grouplet is essentially David Plass and Cristina Tcheyan. We did a fantastic job documenting our collective experience and making ourselves available for schedule coordination, funding, tools and advice, in order to make Fixits of all sizes easier to run and more effective with seemingly no hint of “fixit fatigue”, whereby poor scheduling and organization leads to a lack of participation and general cynicism about the tradition.

There’s a whole bunch more I need to say about individual Fixits and the Fixit process in general, but I need to tackle the most important topic first: Schwag, aka “swag” (Stuff We All Get). People can buy into your mission, and may even participate out of self-interest, but the schwag is what will send them over the moon. Folks dig kitsch.

For the first two Testing Fixits I ran, we produced thousands of T-shirts. If you ever run a Fixit yourself, do not do this! A future post will expound upon the perils of T-shirt procurement and distribution (or lack thereof). Just remember: T-shirts are bad, mkay?

For the first Testing Fixit (August 3, 2006), we also gave out little plastic eggs filled with green, glow-in-the-dark putty. (So your tests will be passing, and the little results display will show green instead of red—get it?) The Testing Grouplet folks had at least one putty-filling party, which was a blast: Highly trained engineers crammed into a room filling little plastic eggs with stupid putty for the sake of one day convincing their fellow engineers to just c’mon, try writing a test for that code, try to fix that broken one no one pays attention to. Ah, what we do for love!

When the Fixit came, people ate this stuff up! But not so much because they really wanted to play with the putty themselves; surprisingly many of them wanted it to give to their kids or other tiny relatives. That’s an interesting nerve we struck on there without intending to—not only did we give people a fun prize to entice them to participate in our event, but the choice of schwag, in turn, made the whole experience into something that the participants could share with folks they cared about.

For the next Testing Fixit (March 8, 2007), we only had T-shirts. Participation was good, but not ostensibly fantastic. At least, we didn’t measure it so well at the time. There’s much I could say here about the horror of T-shirts, the apparent necessity for specific Fixit “roles”, and the introduction of tools to track-and-simultaneously-encourage participation, but I’ll save that for later. This post is about the schwag.

The next Fixit I ran was originally billed as a Testing Fixit, but it focused specifically on rolling out a completely new set of build tools, and eventually it became apparent that we’d be better off not pushing on folks to worry too much about before-and-after test results, but to just get their stuff onto the new tools with the same results as before. Stuck for a new name and theme for the Fixit, I fell back on my Beatlemanic tendencies, which wound up striking the perfect chord: This Fixit would be “The Revolution”.

I went crazy nuts with the idea, lacing Beatles references throughout every aspect of the production, and all of my fellow groupleteers fell into the same hysteria. For the first time, I’d divided up clear responsiblities amongst volunteers via specific “roles”, giving each an “official” name and a more popular “fun” name. (Never, ever underestimate the importance and power of fun when it comes to doing great work.) Being the official “Organizer”, I reserved the fun title of “The Walrus” for myself. This time we had a scoring system, and tiers of schwag and prizes, all of which were Beatles-themed in one way or another. Just for performing a single task, you got a little plastic helicopter blade thingy—dubbed a “Revolver” by Simon Pyle. We handed out Billy Shears Awards to folks who were able to get by quite well with a little help from their friends. We gave away framed John Lennon lithographs to those who performed the most tasks, since they clearly could “Imagine” what impact the new tools would have. And so on.

But the Revolvers were actually a late substitute for the intended standard participation prize: small plush walruses (source link below). Our original announcement to Google engineering had the walruses listed as the official schwag, but when it became apparent that our demand outstripped the immediate supply, we had to switch to the Revolvers.

The Revolution happened on January 31, 2008. People snapped up the new tools, and talked everywhere about “revolutionizing” their projects’ builds, long before and after the Fixit. More folks participated than we had the ability to measure, even with the new Fixit dashboard I created for the task (which evolved into a project known as Gladwell by David Plass, which I’ll post about later), and eventually, within a quarter or two, everybody had switched from the old tools to the new because they were so damn good. But you wouldn’t believe how many people picked up on the walrus schwag after the original announcement and were so genuinely heartbroken that they couldn’t snag the walruses after all. Again, we’d hit on a raw nerve that had made the Fixit experience one that folks were looking forward to on an emotional level. The Revolution was a great success in many ways, but the walruses would’ve put everyone right over the top, if only we’d had them.

As fate would have it, there was one more Fixit in my future: The TAP Fixit, which rolled out the Test Automation Platform. Again, I was The Walrus, but I divided a huge amount of the organizational responsibilities amongst pairs of regional Eggmen and Eggwomen—Eggpeople, really. And this time, we were able to make good on an old promise: We got our walruses. Hundreds of them, if not thousands. Around the world.

There were walruses everywhere. People were already nuts for TAP because of what it promised, and the TAP engineers had successfully migrated a ton of projects automatically, leaving it up to everyone else to fix whatever was broken and retire their old Chris/Jay Continuous Build machines (again, more later). But it was the little plushie guy that won minds and hearts everywhere, that made the event really, really special for a lot of participants and volunteers. Some offices ran out, and we had to issue IOUs and order more. One volunteer in London, Mike Davis, mused something along the lines of: “I sent out an announcement about the Fixit and no one responded. I sent out another about the walruses, and eighteen people replied. I feel I’ve learned something about Google engineering today.”

The TAP Fixit was on March 4, 2010. It concluded a long arc of Fixits and other Build Tools, Testing Technology, and Testing Grouplet activity and strategy starting from 2005, when Google’s code base was largely untested, untestable, and lacking tools to help anyone fix the situation. Today Google is a place where thorough unit testing is the expected cultural norm. (Thanks to esteemed colleague and fellow Intergrouplet veteran David Agraz for letting me plagiarize here.) Fixits continue to happen everywhere, all the time, at scales large and small. But what tickles me as much as all of that is that love for the walrus continues to spread to this day.