Mike Bland

As summarized in my first post on Grouplets, Fixits are days when we ask developers to put aside normal project work for a day (or part of a day) and focus on fixing lower-priority problems that tend to linger—i.e. “important but not urgent” tasks—as a concentrated effort to fix such problems or at least make significant progress towards fixing them. After running two company-wide testing Fixits as a member of the Testing Grouplet, I assumed leadership of the Fixit Grouplet in fall 2007—eventually running two more company-wide Fixits, while closely advising several others.

Since running a Fixit wasn’t something explicitly taught at that point, in early 2008 I worked with my fellow Fixit organizers who comprised the Fixit Grouplet to organize our collective knowledge and experience and make it universally accessible and useful. We did a great job: By the time I left Google, Fixits happened all the time, from team-level “Fixlets” to company-wide events, with no one complaining anymore of “Fixit fatigue” due to inefficient organization and scheduling.

This first post covers the details of the first company-wide Fixit I ran, the Testing Grouplet’s August 3, 2006 Testing Fixit.

• The Testing/Tools Fixit Arc
• Square One
• Flanking Mountain View
• Crack for Googlers
• The Testing Grouplet Logo
• Carrots, not sticks
• Mamie Rheingold, Secret Weapon
• Fixit Fever
• Fingerspitzengefühl
• Fixit Day
• The Filers of Jericho
• Foreshadowing
• The Grand Prize
• The Reward
• Epilogue
• Footnotes

The Testing/Tools Fixit Arc

Before getting into the details, let me outline the overall arc that the collection of Testing and Tools Fixits ended up taking, pointing out what appeared to me to be their most salient achievements. The idea of an “arc” didn’t start taking shape until the Revolution Fixit was underway, and didn’t take complete shape until the Forgeability and TAP Fixits appeared on the horizon. I was the organizer of the two Testing Fixits, Revolution, and TAP Fixit.

The thing to keep in mind about Fixits and a Fixit “arc” is that these are big events that punctuate a much longer, much more incremental, extended effort. A Fixit in and of itself won’t produce a lasting change, usually. But they’re great rallying points for groups dedicated to a large-scale culture change, which infuse the overall effort with a tremendous sense of energy and help the group make a quantum leap to the next level of their efforts, whether that’s a leap in insight or a leap in actual change.

As a quick-yet-relevant aside, a huge part of the Testing Grouplet’s effort was focused on identifying sources of flakiness and common forms of code complexity, then educating developers about how to manage them, as well as collaborating with Testing Technology, Build Tools, and others to develop tools to help make controlling dependencies and writing tests easier. A big part of something being “too hard to test” was a problem of education and tooling, not an inherent problem of a program’s function or architecture.

August 3, 2006: Testing Fixit: The Testing Grouplet’s first big “Man in Orbit” effort, highly-visible and successful thanks to the new Testing on the Toilet publication and “flanking” Mountain View by fully engaging “remote” development offices; first Testing Grouplet T-shirts, sporting the brand new Testing Grouplet “lightbulb” logo

May 8, 2007: Testing Fixit: Successful, yet stressful; awareness of the need to define specific “roles” and to work with a global schedule; first “Gladwell” Fixit application prototype; extreme, long-lasting T-shirt pain

January 31, 2008: Revolution: Breakthrough in adoption of new tools to solve the “I don’t have time to test" issue, namely Blaze (a.k.a. Bazel), SrcFS, and Forge; successful application of Fixit “roles”; second Gladwell prototype; fostered Mark Striebeck’s vision of what would become the Test Automation Platform (TAP)

Mid-2008: Test Certified Challenge (Matt Vail and Tayeb Karim, organizers): Breakthrough in getting a huge number of projects to participate in the Test Certified program, via a months-long Fixit that harnessed the dedicated energy and enthusiasm of Software Engineers in Test in particular; leaderboards and inter-office competition; repurposing of second Gladwell prototype to great effect

October 2009: Forgeability Fixit (Rachel Potvin, organizer): Last big push by the Build Tools team to adapt all projects to the distributed compilation and test execution infrastructure introduced during the Revolution; introduction of “team spreadsheet” idea, whereby individual teams would maintain their own task lists to coordinate participation in the larger Fixit; perfect setup for the TAP Fixit

March 4, 2010: TAP Fixit: Completion of the “Man on the Moon” mission with the company-wide rollout of the Test Automation Platform, a centralized continuous integration and test service; division of Fixit organization into distinct, largely autonomous regions; “emergence” of effort leads to enormous impact and a successful company-wide rollout, retiring thousands of Chris/Jay Continuous Build systems; workflow of build cops permanently changed

Square One

Organizing my first Testing Fixit wasn’t my idea. Bharat Mediratta and Nick Lesiecki, the leaders of the Testing Grouplet at the time, decided it was time to run another one; the last one had been sometime in 2005, before I joined Google that August. For some reason, they decided I should be the one to run it.

The environment at the time was that described in the first section of Coding and Testing at Google, 2006 vs. 2011. My morale was at rock bottom due to an unsuccessful Build Tools project I’d just abandoned (prelinking), and I had no idea how to even begin organizing a Fixit. Bharat, Nick, newly-hired Intergroup Program Manager Mamie Rheingold and I had a small meeting to help me figure out how to start. Bharat asked, “What’s the purpose of a Fixit? It’s to fix something. So what needs to be fixed?” (I may not have the exact words right, but it was something like that.)

On the one hand, I was deep in my I’m-the-stupidest-Googler imposter syndrome funk, and my inability to realize and ask this question for myself seemed to confirm this self-image; on the other, from that point, we got down to concrete goals: Get people to fix broken and flaky tests. Get people to write new tests for untested (i.e. “uncovered”) code. Get people to improve code coverage metrics (i.e. the quantity of production code executed by at least one test). Give every participant a small prize, and give the top participants a shot at a big prize.

The next step was to contact the Fixit Grouplet for advice, which at the time consisted of David Kramer. David was a very easygoing and supportive mentor; given the state of the art of Fixits at the time, David’s mentorship was the only “documentation” provided by the Fixit Grouplet. Fixits were infrequent, straightforward, relatively uncomplicated, and cheap. David provided us with some budget, and also helped me pick a date for the Fixit that wouldn’t interfere with any other Fixits: August 3, 2006. That was about a month away.

Flanking Mountain View

With the small set of directives and a date in my pocket—and nobody telling me what to do, or what not to do—I figured I had carte blanche to start improvising. First, I started developing the Fixit’s internal home page, yanking the raw HTML from the previous Testing Fixit, which looked very official-like for internal Google business at the time. I edited it to explain the goals of the Fixit, the high-level of how it would be run, that there’d be prizes, and where the volunteers were in each office to help out. Of course, I needed to actually recruit volunteers before filling out that section of the page.

By this time, Ana Ulin and I had become fast friends, and she was already helping establish the Testing on the Toilet institution while based in Zürich. She was in; it was good to know that I had an agent in EMEA’s largest development office. Bharat was already planning on being in New York at that time, and I also recruited NYC Googler Michael Chastain, one of Google’s most respected C++ developers, if not the most respected. But still, Google had many more development offices than Mountain View, New York, and Zürich. What’s more, I wasn’t confident that I and the Testing Groupleteers at the time could make sufficient headway in engaging all the developers in Mountain View, which was home to over 50% of the developers at the time. Even by then, working in MTV was becoming an object lesson in shutting out distractions, and that’s certainly how unit testing and its advocates were perceived to a large extent, at that time in Google’s history.

Somehow I got the idea that we could have a greater impact on MTV if we took an indirect approach. Through my friendship with Ana, I became acutely aware of how Googlers in so-called “remote” (i.e. non-MTV) offices viewed their role in the company as somewhat subservient to MTV. It’s definitely a first-world problem to consider oneself a lesser citizen of Google, but the sense of resentment and lack of empowerment was very real in these offices. It’s not that they didn’t have microkitchens, or on-site cafes, or lavish holiday parties and ski trips; the issue was that they felt like they weren’t having an impact at Google, that their work and their achievements were invisible to the company at large—and at Google, impact is everything.

The flip side of this arrangement is that practically every development project outside of MTV, at least at the time, required direct contact with someone or some team in MTV. With those channels of communication open, and shared code and other technical dependencies, I figured that we could get developers in “remote” offices excited enough about the Testing Fixit that they would communicate the importance of the Fixit and testing in general to their counterparts in MTV. Since the MTV folks would be getting the message from folks they dealt with daily, despite the distance, rather that just through random development-wide emails and flyers from folks they’ve never had occasion to encounter at all, they would prioritize participating in the Testing Fixit more highly than they might otherwise.

The problem then was that I didn’t actually know that many folks across the different development offices. But, with this strategy in mind, and with the official-looking home page in place, I was able to scour the internal employee directory, pick a few developers at random from each development office, and shoot them an email out of the blue that read something like:

Hi [INSERT DEVELOPER’S NAME HERE],

My name’s Mike Bland, and I’m organizing a Google-wide Testing Fixit on August 3, 2006 on behalf of the Unittest Grouplet. We’d really love to have the [INSERT OFFICE LOCATION HERE] office be a part of it, and we need volunteers to make that happen. We need people to post flyers ahead of time, and to be available to answer questions from other developers in the office and hand out T-shirts on Fixit Day, possibly even to set up a war room for that purpose.

You can find out more information about the Testing Fixit here: [YOU CAN IMAGINE A GOOGLE-INTERNAL DEAD LINK HERE]

Would you be able to act as a volunteer? If not, I’d really appreciate it if you could forward this information to someone else in your office who you believe might be interested.

Thanks,

Mike

That’s it. Though I cut-and-pasted some boilerplate, I sent every email by hand, and responded to each and every reply. Most of my requests met with enthusiastic acceptance of the offer, and in the future, I found I could largely rely on many of the same volunteers from one Fixit to the next. Though the request was coming from Mountain View, this was not an order from on high to comply with some new corporate initiative. This was a chance to change MTV, and Google as a whole, from the ground up. People dug that.

Crack for Googlers

Oh, right, there was one other detail, one other lever that I used to nudge random unsuspecting Googlers over to the dark side: T-shirts. Googlers looooove T-shirts, as I’ve mentioned before. Why is that? Identification with a worthy cause? Identification with a popular employer, a winning team? Materialistic hoarding instincts? Mate bait? Who knows, who cares? Googlers lose nearly all sense of rational thought and civil consideration when you wave a T-shirt under their noses. It’s like catnip, just like catnip.

The Testing Fixit 2006 T-shirt designed by Mamie Rheingold, with the Testing Grouplet “lightbulb” logo by Johannes Henkel (with a small contribution from yours truly). Click for a larger image. Notice the clever use of a bandage in the word “Fixit” on the back.

Having had my first desk right around the corner from “the” T-shirt closet in Building 41, second floor, I both witnessed and succumbed to the power of this force on many occasions. When I noticed the closet had just been filled, I’d grab a couple of shirts for myself, then silently walk back to my cube and whisper to my teammates that there were T-shirts today, at which point they’d drop whatever they were in the middle of like a hot potato and head straight, silently, for the T-shirt closet, lest they attract the attention of the inevitable hordes prematurely.

If you are or you know a programmer, you should know how unusual it is for a programmer to drop immediately out of his/her flow for any reason. Fire alarms don’t get that swift and focused of a response from a Google developer. It’s doubtful an actual fire in the vicinity, close enough to smell the smoke and feel the heat, would, either. They’d at least save their files and lock their screens first. If Requiem for a Dream were filmed at Google, the T-shirt closet would jump off of the wall and trundle over to torment J. Random Developer at his/her desk. If Winnie the Pooh were a Googler, it would be T-shirts, not honey or Satan, that conquered his soul:

But oh, the perils of such addiction…not so much for the addicted, actually, as for the dealers. And I just started down the dealer path, much to my later chagrin. Still, it did the job at the time. Boy howdy, did it ever.

The Testing Grouplet Logo

As noted in the photo caption above, and in several other articles in this blog, the Testing Grouplet “lightbulb” logo was designed by Johannes Henkel, when he and I both were on the Build Tools team and shared an office, not long before the 2006 Testing Fixit. (Actually, he might’ve designed them just in time for the Fixit, given my recollection about the T-shirt design below.) The two colors most associated with automated testing are green, for passing tests, and red, for failing tests. Johannes made the bulbs point in opposite directions to imply Yin and Yang, and came up with the motto—Debugging sucks. Testing rocks.—all on his own. My lone contribution, after he pointed out the Yin/Yang thing to me, was suggesting the frowny/smiley thing, hinting at the tragedy and comedy theatre masks.¹

Johannes’s original graphic had the green bulb with more mischievious-looking eyes. I think it was when we were ordering T-shirts for the Testing Fixit that the eyes changed. Since the T-shirts were going to also have the Google logo on them, we had to get official corporate approval, and they didn’t like the look of the green lightbulb’s eyes; they thought they looked “too evil” or something. Whatever. But, honestly, I like the more neutral-looking eyes better after all.

After the Fixit, the Testing Grouplet started using this logo to identify its different activities, such as Testing on the Toilet, the cheatsheet sticker below, and in Tech Talks. A derivative logo, the “Pirate Bulb”, became the unofficial second logo of the Test Certified program (the official one being the “T” shield), and appeared on T-shirts and coffee mugs. The Testing Grouplet NY’s logo was the Statue of Liberty holding the green bulb in place of her torch, and an Apple PowerBook in place of her tablet, the Apple logo where “July IV MDCCLXXVI” should be.²

Click for a larger image.
The Testing Grouplet gave this sticker to Nooglers during the weekly unit testing lecture. It contained the most common commands for checking out and building code at the time, and was intended for folks to stick on their monitors or laptops. We’d hoped they’d remember the Testing Grouplet, perhaps checking out the links, maybe even joining us. After the Revolution, the stickers became obsolete, and we never produced new ones.

However, while people recognized the logo, mainly from Testing on the Toilet, precious few actually realized that there was such a thing as a “Testing Grouplet” behind it, or that Test Certified was a Testing Grouplet program, or that TAP was largely a consequence of Grouplet-instigated developments, and so on. The day I returned from my sabbatical, I attended a tech talk by a researcher who had surveyed the Google population in my absence, and it was sad to hear how everyone was aware of these by-now deeply-ingrained elements of Google development culture, but only a very small percentage of them had ever heard of Grouplets at all, let alone the Testing or Fixit Grouplets.

Back to our story…

Carrots, not sticks

Remember, the Testing Grouplet just decided it wanted to have a Testing Fixit in which it wanted all of Google development to participate. Nobody higher up the food chain asked us to do this—but neither did they stop us from doing it. Vice Presidents were more than happy to send out emails advertising these Fixits—which we always drafted ourselves, and often were sent with very few edits—but otherwise, it was completely up to us to persuade our fellow developers that playing along and writing tests was worth their time. The short-game view would consider this a handicap, that we might’ve gotten more visibility and higher participation if we had more vigorous support from highly-visible executives, possibly even a mandate.

However, as I mentioned in my earlier post about the Test Certified program, Patrick Copeland, Director of Engineering Productivity when I met him in mid-2007, explained to me that a lot of our execs had come from big-name tech companies, and had seen even well-meaning corporate mandates fall flat precisely because they were shipped to the development organizations as mandates, rather than voluntary programs for which persuasive, rational arguments could be made. Over the course of four or five years, the Testing Grouplet’s Fixits and other efforts—most notably Testing on the Toilet and Test Certified—succeeded because we were forced to be creative in our efforts, appealing to reason, improving education and documentation and tooling as needed, and keeping it up for the long-haul. We had no idea if we’d ever succeed, and in fact, I stepped out of the Grouplet community not knowing for sure if all the trouble I’d caused and resources I’d spent as a Groupleteer were at all worth it. Fortunately, history bore out the fact that, yeah, actually, we kicked ass at the long game—and it shows.

So, back to this particular Testing Fixit: All we had to beat Googlers over the head with were carrots, not sticks. We couldn’t tell anyone they had to participate in the Fixit. Hence, the T-shirts, the flyers, the TotT articles, the projections behind the B43 lobby—and the glow-in-the-dark Silly Putty that we gave to all participants as an extra bit of schwag, since we anticipated that we might run out of the right sizes of T-shirts in the right places too fast. As a grand prize, we offered an Aeron chair as part of a random drawing from all of the Fixit participants.

Why an Aeron chair? Because Bill Coughran, one of the Vice Presidents, told Bharat that he couldn’t guarantee a reserved parking spot at the Googleplex for a month, which was the original idea. Besides, what if the winner wasn’t in MTV? (Hard to believe that could happen, though, right?)

Mamie Rheingold, Secret Weapon

As I mentioned above, Craig Silverstein had just hired Mamie Rheingold as the new Intergroup Program Manager. Her job was to work with all of the individual Grouplets to help them better organize and develop their efforts, and to share ideas and innovations across the different Grouplets, which were all based on 20%-volunteer time, so they’d all have a bigger, broader impact on Google development as a whole. However, by that time, the Testing Grouplet—then still known as the Unittest Grouplet—was already starting to roll along, so she spent a large amount of her time working with us, to find out what made us tick, and to find ways to improve our dynamic, even.

Talk about luck! I honestly don’t know what would’ve become of my first Testing Fixit—or the second, or the Revolution, or the Test Certified program, or the Fixit Grouplet—without her enthusiastic and determined support, combined with an acute design sense that served the Testing and Fixit Grouplets in particular very well. While I was starting to put together the Testing Fixit home page and recruit volunteers, Mamie was hard at work designing the T-shirts and promotional flyers, managing the logistics of distributing the same, and helping schedule war rooms in various offices. She also arranged to have a series of Testing Fixit flyer images projected behind the desk of the main lobby in Building 43 for the week leading up to the Fixit Day.

It’s true, Mamie’s role in the early days, and particularly during this Testing Fixit, was largely logistical. But in less than a year, she was a full-fledged strategic partner. She used her time handling all of these tasks to get to know people in the community, and begin engaging them in more and more strategic discussions as time went on. Having internalized the importance of Test Certified in particular, during the time that TC got rolling before Bella Kazwell (née Voldman) assumed leadership of the program, Mamie single-handedly carried the burden of its management while the rest of us were focused on other aspects of the Testing Grouplet’s operations. Quite simply, there would be no Test Certified today if it weren’t for Mamie keeping it alive in its darkest hour.

Also, the Fixit Grouplet, at the time I led it, wouldn’t have flashed in and out of existence were it not for Mamie helping put together a fantastic team in the time between when I assumed leadership in September 2007³, already eyeballs-deep in organizing what would become the Revolution Fixit, and the time the Revolution took place on January 31, 2008. This team included Jonas Klink from the Accessibility Grouplet, Lisa Carey and Chris Holstrom from the Documentation Grouplet, as well as Logan Eldridge, Won Chun, and David Plass. A year or so later, the Fixit Grouplet was essentially David Plass and Cristina Tcheyan, Intergroup Program Manager after Mamie, but their role was essentially to point prospective Fixit/Fixlet organizers to the Fixit Grouplet docs and tools, help pick a good date, and offer moral support. That Fixit Grouplet team that Mamie, essentially, put together did so much in that one year that it essentially “finished”—possibly the only Grouplet that could say that.

Back to the time of the Testing Fixit, I felt like I was on my way out of Google, right when she’d just made her way in. We both had something to prove, I suppose. We became close partners-in-crime for years, until I drifted away from mainline Grouplet activity in November 2008, when I resigned from the Fixit Grouplet. I’ll be talking about her more throughout this series of posts.

Fixit Fever

At first, the Testing Fixit organization involved just me and Mamie. As we started picking up steam, recruiting volunteers and designing flyers, etc., and as we shared more and more of our progress in the weekly Unittest Grouplet meetings, the rest of the Grouplet began to get more excited and invested in the Testing Fixit process. People wanted to start stepping up to post flyers, sign up for Fixit Day volunteer/war room duty, etc. The Groupleteers who were also from Testing Tech, John Penix in particular, started enhancing the test coverage-related Fixit goals, slicing and dicing numbers to reward improvement along several dimensions, at a time when coverage data were more-or-less exclusively generated by Testing Tech.

After about two weeks of this, Nick Lesiecki proposed to the group at the end of one of my Fixit reports that the Grouplet agree to dedicate the remaining meetings before the Fixit to Fixit organization. “That would be awesome!” was my response, and thus everyone agreed.

To illustrate this fever in more concrete and amusing terms, let’s go back to the glow-in-the-dark Silly Putty eggs. There was just one catch: The putty didn’t come in the eggs ahead of time; we had to order them separately, then put the putty in the eggs by hand. This was not a fun job for one person, perhaps not even two. But for seven or eight in the course of a couple hours one afternoon—with beer? That was fun as hell.

Erik Kline, who had previously worked on coming up with unit testing stats for the monthly unit testing awards of yore, before I’d even joined the company, signed on to work on our stats dashboard for the Testing Fixit. He was killing himself with his regular work on top of tweaking our dashboard on what seemed like an hourly basis, but his enthusiasm and dedication was inspiring, and the resulting near-real-time view of Testing Fixit activity was exhilirating. For Googlers who can’t get their crack fix from T-shirts, good data is a more-than-satisfactory substitute. T-shirts or data feeds are two of the fastest and best ways to a Googler’s heart—or other parts.

Fingerspitzengefühl

As the Testing Fixit operations developed, as the storm system organized into a hurricane, it became apparent that I was firmly in its eye; I wasn’t micromanaging anyone, but I had complete knowledge of damn near everything that everybody was doing related to the Fixit. I responded to every email from every volunteer and every participant in every development office personally. I developed what I would later discover was known in military operations terminology as fingerspitzengefühl, or “fingertip feel”. I did not tire, and I did not slow down. How could I, when everyone else who surrounded me was getting so revved up? When people I’d never met in countries and continents I’d never visited were also getting visibly excited about all the trouble we were making together?

By the end of my next Testing Fixit, I realized this model could not be replicated reliably—but it sure was nice at the time. I did, however, find ways to keep energy high and maintain fingerspitzengefühl while scaling up to larger operations without killing myself—freeing me to find new ways to kill myself with too much Fixit-related work.

Fixit Day

August 3, 2006 came, and those of us in the MTV Testing Grouplet reported to the war room on the first floor of Building 43. The Fixit was largely done in Zürich, given the nine-hour time difference, and was already well underway in NYC. But MTV was soon rolling in frenetic activity, as the email list and IRC channel lit up with participants asking questions and claiming schwag. A fair number of folks also made the effort to swing by the war room in-person, adding to the upbeat energy that sent us all spinning about like happy little electrons.

Michelle Levesque had already begun recruiting Nooglers—new Googlers—every week at the end of the Unit Testing training lecture as part of [the effort to distribute Testing on the Toilet][tott-everywhere] within Mountain View. When it came time to find volunteers to distribute T-shirts on Fixit Day, Michelle was able to call on this so-called Noogler Army to step up to the task. All day long, as tests were fixed and written, fresh-faced Nooglers would be dispatched from the war room with handfuls of T-shirts to be delivered to the worthy recipients all across the Verse—uh, ’Plex. Michelle even developed the Noogler Dispatch Protocol, as I called it, whereby she’d write a recipient’s name, desk address, and T-shirt size on a sticky note to hand it to a Noogler to take over from there.^4,5

People actually really dug the glow-in-the-dark Silly Putty, too. Not because they cared so much for it, but because they had kids, or had nieces or nephews or friends with kids, and they thought the kids would get a real kick out of the stuff. We’d accidentally tapped into another schwag-related vein other than the genetic predisposition Googlers have towards having severely giddy reactions around T-shirts: People love reliving the experience of childhood, either by acquiring cute schwag, or by acquiring schwag for the purpose of making a child happy. It seems clear in hindsight, but people often love experiences not just for the experience itself, but for the experience of sharing it somehow, especially with children. So part of my advice to you is, if you ever think of running a Fixit: Think of the children!

The Filers of Jericho

With this constant sight of happy-go-lucky Nooglers wandering about the hallways doling out T-shirts to people, Googlers all around the watchtower—err, Googleplex—were increasingly moved to participate and score sweeet T-shirts and glow-in-the-dark silly putty for themselves. As the day grew long in MTV, more and more developers began editing and compiling code, and running tests, and submitting changes for new or fixed tests to Perforce. Right around 5pm or so, we’d started getting reports from many developers that everything on their workstations started getting really, really slow. People wanted credit on the scoreboard, they wanted their schwag, and they wanted us to know they’d earned it since the signal did not appear to be propagating through the normal channels.

Eventually, the Build Tools team revealed what had happened. All Googlers have their personal, “home” directories not on their own workstations, but on a network storage system known as a “filer”. This is so that if an developer’s machine stops working, or that developer moves to a new location or visits another office, that developer’s data remains accessible, and is also easily backed up and restored. Pretty standard operating procedure for most development operations beyond a certain size. Consequently, however, whenever an developer does something that causes files to be read or written within their home directories, that causes network traffic that the filers must respond to. Apparently the end-of-day Testing Fixit activity was so great, the filers couldn’t handle the load. In the words of my Noogler classmate Damian Menscher, Google’s greatest denial-of-service threat is Google itself, and the Testing Fixit helped reinforce that observation that day.⁶

Foreshadowing

Build Tools upgraded the filers shortly after the Fixit. I was on the Build Tools team at the time, as I would be when I organized my second Testing Fixit. When I’d later propose what would become the Revolution Fixit to Build Tools after I’d left to join the Test Mercenaries, my former manager Rob Peterson remembered this episode. This led to a tense situation, not because the filers continued to be a bottleneck—SrcFS was already just about ready to launch by then, eliminating much of that particular problem—but because the Build Tools team was not ready for me to launch the Revolution when I’d decided I wanted to, given the somewhat incomplete state of Blaze and Forge’s one-man’s-20%-project status, and the filer episode was tangible proof that the Testing Grouplet could pull together a very large Fixit regardless of the Build Tools team’s approval.

People were so desperate for improved build tools at the time, they would take what they could get, and I wanted to give them SrcFS and Forge as soon as possible, having experienced the difference they made while setting up a Chris/Jay Build. The sooner people had faster, better tools, the sooner we’d achieve our Man on the Moon mission of getting all of Google development to Test Certified Level Three, and I knew everyone would take SrcFS and Forge even without Blaze. I put all of my old colleagues in the Build Tools team in a very uncomfortable situation, and the resulting tension took years to fade away. I’ll further explain all these dynamics and developments in my future post about the Revolution, but suffice it to say that, fortunately, all turned out remarkably well for all involved, despite the sharp elbows thrown and deep bruises resulting at the time.

The Grand Prize

Fixits, at least one-day Fixits, are traditionally held on Thursdays, somewhat coinciding with the “No-Meeting Thursdays” of the distant past. Thanks to EK’s near-real-time dashboard stats, we were prepared to announce the winner of the grand prize the following day, during the weekly TGIF meeting, where the founders and other execs share news of the company’s progress with all developers willing to attend, as well as host a Q&A directly with developers out in the open. There is food, and beer.⁷

Fixit Day was a really good day, and consequently, TGIF after Fixit Day was a really good day. I celebrated with a couple of beers before I was called up on stage to stand next to Sergey to offer him a funny fuzzy green hat full of slips of paper with participant’s names, so he could be the one to draw the winner of the Aeron chair. The moment came, and he drew the name, and I recognized it—granted I would’ve recognized most of them at that point, given how many people I’d interacted with over the course of the Fixit. But the name he drew was Katya Rogers, one of the volunteers helping out with the Fixit in Kirkland, Washington. I’d never actually met her, neither in person nor via a videoconference, but when I saw her name, I giggled very conspicuously, amused by the coincidence that one of the volunteers had won the prize. Sergey looked at me very strangely. I straightened up as best I could, and exited the stage soon after. Thankfully.⁸

It took us a year to work out the purchase and delivery of that damned chair. Katya was plenty patient and good natured about it, but Nick Lesiecki started feeling really guilty about how we were dragging our feet, and eventually shamed me into getting it done, finally. By all accounts, Katya was very pleased once it happened.

The Reward

At the next Testing Grouplet meeting after the Testing Fixit, the Grouplet rewarded me with two incredibly meaningful tokens of their appreciation: As signed bottle, and bag, of Jägermeister; and an eBay gift certificate for my very own Jägerator.

Click for a larger image.

I never cashed in the gift certificate, as I had no room in my apartment in which to place a Jägerator, and thought the prospect could prove deadly, potentially. The bottle of Jäger, well, that didn’t last long, though I’ve kept the bottle and bag all these years.

Also, the following month, I was honored to assume leadership of the Testing Grouplet along with Neal Norwitz and Michelle Levesque. With Mamie Rheingold as my partner-in-crime in MTV, Ana Ulin as my partner-in-crime in Zürich, and starting in November 2006, David Plass as my partner-in-crime in NYC, we would build a company-wide network of passionate 20%-time volunteers over the next couple of years that would not only spread ideas and tools, but help organize and run several more Fixits that would permanently change the Google developering culture. The biggest reward I could imagine for all this effort was having the freedom to keep doing what we were doing, being creative and taking power into our own hands to change the company for the better. It was exhausting work, but a lovely atmosphere, most of the time.

Epilogue

Two weeks after the TAP Fixit, in March 2010, I was in Kirkland for a team-building exercise between my websearch team in NYC and a counterpart team in Kirkland with which we worked closely. After the first day of activities, I needed to head over to the Seattle office to pay a visit to Nick Lesiecki and Patrick Doyle, who had finished his tenure as Test Certified administrator by then, and I rushed around the office trying to find a printer which I could use to print directions. I found one, but couldn’t totally figure out how to locate it on the network to print from it. I asked for help from an developer sitting nearby, a woman with a slight hint of an accent, who accommodated me. I couldn’t see her nameplate somehow, but I noticed her very shiny chair.

I had to ask, “Excuse me, but are you Katya?”

“Yes…?”

“Oh! Well, hi! I’m Mike Bland…I ran the Fixit a while back when…”

She stuck out her hand for me to shake and smiled. “Hi Mike. Thank you so much for the chair!”

Footnotes