Mike Bland

Copyright 2017 Mike Bland, licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This work is derived from The Convergence of Wills (Surge 2016 Edition), which is Copyright 2016 Mike Bland, licensed under a Creative Commons Attribution 4.0 International License, delivered at the Surge 2016 conference on September 23, 2016.

This presentation, The Rainbow of Death, describes my experience driving automated test adoption at Google, and lays out my perspective on cultural/social challenges in general.

I also invite you to check out my more recent talk, Making Software Quality Visible.

• Artifacts
• Videos
• Abstract
• Who am I? How did I get here?
• State of Google’s automated testing in 2005
• The Google Web Server
• Chasms and rainbows
• The Testing Grouplet
• Fixits
• Commitment model
• Roles
• The strategy emerges
• The Tipping Point
• The Scarlet G
• The real challenge
• Approved and improved by…
• History
• Footnotes

Artifacts

• The Rainbow of Death Google Slides
• The Rainbow of Death Google Doc
• The Rainbow of Death speaker’s notes in Google Drive
• The Rainbow of Death abstract on the DevOpsDays Baltimore 2017 schedule

Videos

Click to watch The Rainbow of Death from the KRK Tech Talks Meetup sponsored by Schibsted Tech Polska. Probably my favorite Rainbow of Death video so far!

Click to watch the video of The Rainbow of Death from the Microscope Barcelona Meetup on March 4, 2017. Has the slides edited in, and a Q&A session afterwards.

Click to watch the first ever The Rainbow of Death presentation from DevOpsDays Baltimore 2017. However, note that it’s backlit and doesn’t have the slides edited in. You can also check out the DevOpsDays Baltimore YouTube channel.

Abstract

Believe it or not, Google wasn’t always the shining portrait of automated testing virtue. It had smart people, a knowledge-sharing culture, unimaginable compute power, and great food; it was flush with cash and positive press; and very few developers wrote tests—why would they need to?

In this talk, we’ll see how a band of upstarts within Google called the Testing Grouplet followed the grain of its culture and gradually, over five years, shifted it from one in which code was “too hard to test” and many “didn’t have time to test” to one in which automated testing has become a nearly indispensable aspect of daily development. We will see how the basic forces of human nature working against that effort map to the same forces evident in other organizations. Finally, we’ll see how a custom-tailored, multi-dimensional approach based on timeless principles—a “Rainbow of Death”—drove the development of tools and processes that ultimately produced a permanent cultural shift that helped avert a potentially significant crisis and that continues to benefit not just the company, but its customers and society at large.

Who am I? How did I get here?

This is a talk about change. It’s about the “Rainbow of Death” model that explains how my fellow Instigators and I crossed the chasm between us and the majority of Googlers to make automated testing a standard development practice. I’m hoping it will inspire you to break from the Standard Narrative and overcome your own cultural challenges.

But first, allow me to share the story of how I earned the privilege to speak to you today. From the beginning, I’ve always felt compelled to push boundaries, assert my own truth. Whenever others suggested I follow a traditional path, I would tilt in a completely different direction. The “Standard Narrative” never applied to me.

Eventually, however, my lack of musical commitment—or talent?—gave way to the creative euphoria of programming. In my first job after college, my colleagues poked fun at me for carrying my “library” of programming and algorithms books with me day in, day out. Then, my team went through a “death march”, a crushing experience of working insane hours to deliver on impossible requirements by an impossible deadline on a fragile, untested code base that barely worked to begin with. The pressure was intense, and I wasn’t my best self at this time. But somehow, we did it. We met the spec by the deadline, and then were given the runway to do whatever we could to make the damn thing faster.

During this period of relative calm and freedom, I discovered unit testing, completely by accident, from an article in C/C++ Users Journal.¹ With my library and my newfound testing practice in hand, I rewrote an entire subsystem from the ground up. I’d apply a design principle, or an algorithm, and exercise it with focused, fast, thorough tests every step of the way, rather than waiting for system integration. I could immediately validate that each part worked as planned, and spend more time developing than debugging. In the end, my new subsystem improved performance by a factor of 18x and saved the project. And when a bug or two cropped up, I could investigate, reproduce, fix, validate, and ship a new version the same day—not the next week, or the next month.

My teammates recognized my abilities, but they never adopted my practices, despite their obvious impact. I couldn’t figure out why. So I put my house on the market, quit my job, and met a woman on an online dating site whose referral led to me joining Google in 2005. In that order.

Google already had smart people, a knowledge-sharing culture, unimaginable compute power, and great food; it was flush with cash and positive press—and at the time, it appeared to have zero of the typical organizational bullshit. It served the people, and the people served the mission.

With all that freedom (and food) came a great deal of responsibility—not only to our users and customers, but to one another. There were already standard practices built into the culture such that everyone was expected to conform to style idioms, submit code for review, review others’ code, disseminate knowledge through talks and documentation, and generally collaborate with and support one another. All of this kept silos from forming, and enabled the company and its developers to increase their capabilities and move very rapidly.

State of Google’s automated testing in 2005

We had the responsibility to help one another improve our code, our products, our country—err, company—and ourselves. Yet even though Google was doing so many things right, very few developers wrote tests. Why not? Unit testing wasn’t part of the MIT or Stanford Ph.D. curriculum. Most had never experienced a death march. Relatively few had any exposure to automated testing concepts, techniques, or tools—or had any experience informing them of their value. On top of that, the company seemed capable of doing little wrong at the time, as its stock price and hiring rate showed no signs of slowing down. How could one claim a lack of testing was hurting it?

It was impossible to prove this objectively, yet Google’s core priority structure was founded on data-driven decision making. This structure, by all accounts (including mine), has served Google very well; but it made it extremely difficult to convince people that an investment in automated testing would pay dividends.²

Even so, some of us felt that the company was at risk of collapsing under its own success. The size and complexity of its code and products exploded with its growth. Though Google tried to hire the best and provide a supportive environment, the development operation was getting so big it was starting to break down.³

The Google Web Server

No project was a better example of this than the Google Web Server (GWS for short, pronounced “gwiss”, like “Swiss”), the program running the google.com home page. (I didn’t work on the GWS team, but was close with many of its members.)

In 2004, GWS was not a glamorous assignment. It was a dumping ground for lots of unrelated changes as search features were developed by different teams. Imagine if you break google.com…or allow it to be broken. Search results might be bad, or unacceptably slow. Thousands of queries per second turn to thousands of unfulfilled promises, and the business loses not only revenue, but trust.

When Bharat Mediratta became tech lead of the GWS team, he believed automated testing would go a long ways towards curing the ills of complexity and fragility. So the team took a hard line: No changes would be accepted into GWS without accompanying tests. They set up a continuous build and religiously kept it passing, and monitored their test coverage to ensure that it went up over time. They wrote a policy and a testing guide, and insisted that contributors both inside and outside the team followed them.

Despite the initial unpopularity this policy, the GWS team held firm and eventually turned a corner. They became one of the most productive teams in the company, integrating large numbers of changes from different teams while maintaining a brisk release schedule. New team members made productive contributions to this complex system quickly, thanks to good test coverage and code health. Ultimately, this radical policy enabled the google.com home page to expand its capabilities rapidly and to thrive in an amazingly fast-moving and competitive technology landscape.

GWS was a model of automated testing effectiveness, but it was still a relatively small team in a large and growing company. Given all the other forces, worldviews, and personalities at Google, and our inability to prove the value of testing using data, its message was drowned out by Google’s Standard Narrative. We had to find ways to amplify the GWS message, and build a bridge across the chasm.

Chasms and rainbows

The “chasm” model comes from Geoffrey A. Moore’s book Crossing the Chasm, and explains how different people adopt change differently. Each of these classes has an important role to play in the change process⁴:

• Innovators and Early Adopters are like-minded seekers that I call “Instigators”. They are the catalysts for change; but their unfiltered barrage of new ideas can be upsetting and costly to the organization.
• The Early Majority is open to change, but needs accessible tools and procedures. They’re the “fad filter”, the sympathetic audience willing to give us the direct, constructive feedback we need to polish and scale the most worthwhile ideas.
• The Late Majority is the stabilizing force, the repository of institutional knowledge that slowly absorbs and productionizes the ideas proven to best serve the organization. They aren’t as eager for change as the Early Majority, but they’re happy to adopt proven practices.
• The Laggards provide challenge the Instigators most directly, questioning or outright denying the value of a new idea, and provide the most vocal and active resistance. However, their direct criticism may inspire the Instigators to find unexpected common ground and more effective solutions than they otherwise might.

Getting the right messaging to the right people in the right order is key, and crossing the “chasm” between the Instigators and the Early Majority is what makes or breaks an initiative.

What I’ve added to this model across the chasm is the titular “Rainbow of Death”. I borrowed this model from fellow Ex-Googler Albert Wong (but the funny name is my idea).⁵ It delineates the steps necessary to carry an initiative from one side of the chasm to the other:

• Intervene: The Instigators with the necessary expertise write tools and resolve critical issues themselves.
• Validate: The Instigators give the Early Majority a means of visualizing and marking progress, to demonstrate value and encourage further progress.
• Inform: The Instigators teach the Early Majority how to do the right thing the right way.
• Inspire: The Instigators articulate a vision and compel the Early Majority to act on it.
• Mentor: The Instigators cultivate relationships with the Early Majority, providing them with insight and moral support, coaching them to acquire new habits.
• Empower: After the Instigators have provided the knowledge and the power to make the right thing the easy thing, the Early Majority exercises their new capability without guidance. They’ve become experts themselves.

The progression of concepts demonstrates the theoretical transition from dependency on experts towards empowerment. However, it’s not a hard ordering requirement. Parts of the Rainbow can be developed in parallel, starting with the resources at hand; and over time, we can see where more work is needed fill in the remaining ones. Even if our efforts seem scattered, random, and disconnected at first, we can begin to understand how they actually reinforce one another.

(By the way, if you’re already getting hung up on the word “Death” in the name, hang on until the end of this ride—all will be revealed.)

It’s the responsibility of the Instigators to build this bridge across the chasm; and we testing Instigators, despite the challenges we faced, still had a lot of advantages by virtue of being at Google.

• Transparency: We had access to information, and tools that facilitated discovery;
• Autonomy: we were empowered to develop and promote a vision; and
• Collaboration: we had the freedom to combine forces with like-minded folks to pursue a common goal.

The Testing Grouplet

Shortly before I joined Google in 2005, Bharat joined forces with Nick Lesiecki to form the Testing Grouplet. Grouplets were teams of volunteers pooling their 20% time—from the famous policy whereby Googlers are free to work on outside projects—to address an issue affecting the entire company. The power of Grouplets comes from the common language that emerges when different parts of the company come together to solve common problems.⁶ (Eventually, I succeeded Bharat and Nick as one of the co-leads of the Grouplet.)

We had very little budget and zero authoritah, but we had the freedom to explore creative solutions to the problems facing automated testing adoption, often relying on the GWS experience as a model.

To better frame the change process and the challenges we faced, let’s apply Kurt Lewin’s theory of social change.⁷ It consists of the “unfreezing” phase, in which existing behaviors must be unlearned; the “changing” phase in which new behaviors are learned; and the “refreezing” phase in which the new behaviors become standard procedure. It also describes “driving forces” that compel change, and “restraining forces” that work against it.

Google developers were growing ever-more frozen in an unsustainable pattern, and we believed automated testing was necessary to break that pattern. At the same time, there was an array of restraining forces pushing against its adoption. People mostly had no experience with testing outside of the slowness and brittleness of the status quo, and were under constant delivery pressure while feeling intimidated by many of their peers. Who could blame them for not testing when they couldn’t afford the time to learn?

Economists call this “temporal discounting”.⁸ Basically, if someone’s presented with an option to push a feature now without tests, or prevent a problem in the future (that may or may not happen) through an investment in testing, they’ll tend to ship and hope for the best. Combined with the fact that the ever-slowing tools made it impossible to reach a state of flow, this combination of immediate pain and slow feedback in pursuit of a distant, unclear benefit made the “right” thing way harder than it needed to be.

Arguing against these excuses would’ve been counterproductive. They signified real problems we needed to solve, and the Standard Narrative from which they emerged would not provide their solution.

We began experimenting with a lot of ideas to start chipping away at these problems, starting with the resources already at our disposal. We worked closely with EngEDU, our in-house training organization, to develop training materials called “Codelabs”, a new hire (a.k.a. “Noogler”) lecture and lab session based on these Codelabs, and tech talks, often with book giveaways. We introduced the small/medium/large test size schema, to dispense with the old “unit” and “regression” test terminology—which came to mean “runs in less than five minutes” and “runs longer than five minutes”—and to define more rigorously the different types of tests and what they were good for. Finally, we worked closely with our tools teams to reduce the friction created by slow tools that amplified the “I don’t have time to test” excuse.

We can see how these pieces began to fit into the Rainbow. Notice the early emphasis on “Inform”; the natural first inclination is drop knowledge into people’s hands and hope it results in widespread adoption. But as we’ve learned time and again, information and facts alone aren’t always sufficient to change people’s worldviews and behaviors. Still, these activities helped the Grouplet itself converge upon concepts and language that helped us articulate our message moving forward.

In early 2006, during a brainstorming session to set quarterly goals, we literally asked ourselves what we could do to take all this good information we’ve been pumping into the ecosystem and make it actually stick with people. After bouncing around ideas to the point of silliness, and unbeknownst to us at the time, we hit upon possibly our greatest breakthrough: Testing on the Toilet, our weekly testing periodical that is still published to this day. By publishing an episode each week in nearly every bathroom in the company worldwide, we were able to gradually raise the degree of testing knowledge and sophistication throughout the company. People would apply new concepts and tools they learned about in their “spare time” to their production code. Eventually TotT episodes became like the official style guides, references to justify approaches or support suggestions during code reviews.

We also recognized that many teams that were interested in testing needed a few hints. So we developed the Test Certified program, a roadmap based on the GWS model designed to hack the measurement-focused priorities of the culture (by providing measurable tasks, levels, and a “ladder” of teams); and to overcome the first, scary obstacle of not knowing where to start. We also provided volunteer “mentors” to each team who would provide advice and ultimately “validate” their progress in climbing the “TC Ladder”.

Eventually some larger teams with big, visible products became interested in Test Certified, but needed more assistance than a TC Mentor could provide. So we pitched an idea to Google executives, got approval and funding, and launched the Test Mercenaries, a team of full-time “internal consultants” hired to help teams improve their testing practices by applying the Testing Grouplet’s tools and techniques to the team’s own code, using Test Certified as a guide and a goal. The Mercs also worked closely with the tools teams, providing a steady stream of feedback that drove innovation as we applied new tools to challenging projects.⁹

We can see more of the rainbow coming together. Test Certified served inform as well as inspire and mentor, but the validation function was its most critical: Teams could take concrete steps towards making a tangible impact on their code quality and productivity. We sweetened the deal by providing participating teams with “build orbs”, glowing information radiators that provided a conspicuous visual indication of the state of a team’s continuous build. Hacking on these orbs and their driver software was a source of fun and inspiration for Grouplet members as well.

Fixits

We also began taking part in the proud Google tradition of organizing Fixits, where volunteers would organize one day events to address “important, but not urgent” issues. These were neither mandated nor approved by the executive hierarchy, though executives often gladly encouraged participation. As it turned out, writing tests, fixing broken tests, or moving up the TC ladder were perfect Fixit tasks. The urgency of Fixits generated a critical mass of both volunteer and participant effort that ratcheted up testing awareness to a new plateau each time.¹⁰

To give you an idea of how much effort and how much fun Fixits generate: I helped instigate a Testing Fixit for a client in Europe on June 22,

1. Twelve offices all around Europe and in South America hosted talks and workshops, wrote and fixed tests, even wrote their own Testing on the Toilet episodes.

One of the most surprising outcomes of this particular Fixit was the interpretation of this very talk in cake form by a couple of folks in the London office, or what immediately came to be known as The Rainbow of Death—By Diabetes. Certainly I’d never seen anything like it before and never would’ve dreamed that I might. This is what I love about building grassroots networks of Instigators and unleashing their creativity—they will never fail to surprise you in delightful ways!

Commitment model

A critical organizational pattern that emerged from Fixits and the other Testing Grouplet initiatives—one that helped my client launch their own Testing Grouplet, Testing Fixit, and related initiatives quite rapidly—is the division of labor I call the “commitment model”. This model explains how to distribute responsibilities effectively across a range of volunteers according to their level of availability.

• Core: Core members define the vision for accomplishing the mission, assume major responsibilities and drive major initiatives, and communicate to other volunteers the roles they can fill to execute on important parts of the operation.
• Frequent: Frequent contributors assume important, if not major roles, to ensure that ongoing tasks are consistently fulfilled over the long-term. Whereas core members function strategically, frequent contributors function tactically, driving regular activities that ensure momentum is created, maintained, and amplified.¹¹
• Occasional: The majority of volunteers will be occasional contributors, offering their time and energy as they can, when they can, to drive the mission forward. While not as committed as the other levels, their actions are critical to creating widespread, tangible, lasting change

Here’s how these commitment levels looked in reality for the Testing Grouplet:

• Core: The core group consisted of the Testing Grouplet leads and active core members; Fixit organizers and core team members recruited to fulfill specific roles; and the coordinators of the Testing on the Toilet and Test Certified programs.
• Frequent: The frequent contributors consisted of regular Testing on the Toilet reviewers and posters, Test Certified mentors, and local Fixit Instigators that acted as Fixit organizers for each of their offices.
• Occasional: Many other volunteers wrote Testing on the Toilet episodes, Codelabs, or other documentation; gave Tech Talks on specific testing concepts, tools, or techniques; or did their part during Fixit events to improve their own team’s tests and practices.

In order for our initiatives to become successful, we needed a balance of all three types of commitment from volunteers across the company. No matter how much time and energy anyone had to give, we were able to make the best use of it.

Roles

Recruiting and managing volunteer contributions across commitment levels took a quantum leap after I stumbled upon an idea that helped channel them in very specific, powerful ways. In the retrospective for my second Testing Fixit, after realizing how much work I had taken upon myself, I defined roles to identify clear responsibilities that must be fulfilled to coordinate an initiative, Fixits being only one example. My experience has confirmed ever since that defining explicit roles is one of the most effective ways to organize and mobilize volunteers very, very quickly.

Also, since tackling cultural change can be a grinding, thankless task, while the work itself is serious, it’s critical to have as much fun as possible with it. Picking funny alternative names for each role inspires people to assume and carry out responsibilities with a positive attitude and lots of energy. To illustrate, here’s a sampling of the roles that I frequently like to use.

• Organizer (The Walrus): The Organizer (or Organizers) is responsible for defining the mission and the vision for achieving it, for ensuring all other roles are filled, and for removing obstacles for the rest of the team. Because I’m a Beatlemaniac, the label I chose to self-apply while organizing Fixits was “The Walrus”. Goo goo goo joop!
• Documentation Coordinator (Minister of Information): The Minister of Information ensures that information is cultivated so that it remains accessible to everyone who needs it. Nothing will sink an initiative faster than people not being able to find the knowledge they need, when they need it—or getting sick of repeating themselves because there’s no publishing platform in place.
• Publicist (Minister of Propaganda): The Minister of Propaganda does exactly what it sounds like: Ensures that people know things are happening, and gets them excited about getting involved.
• Statistician (Damned Liar): The Damned Liar tracks metrics to illustrate the observable impact.
• Recognition Coordinator (Heart and Soul): The Heart and Soul makes sure that core members as well as frequent and occasional contributors are recognized and rewarded for their time and energy.
• Local Instigators: Local Instigators effectively act as mini-Walruses in their offices, and so on.

Hopefully this sample gives you a clear idea of the concept.

• Transparency: It makes explicit who’s who in an organization, and what their responsibilities are.
• Autonomy: It grants autonomy by distributing decision making; the Walrus need not be involved in every conversation.
• Collaboration: This means that the burden of the mission is clearly distributed amongst teammates, rather than falling squarely on the shoulders of a critical few.

Just as any other business or governmental organization needs a degree of social structure, so do grassroots volunteer initiatives.

The strategy emerges

After two years, we really started to roll…and to drift apart. The Grouplet, Testing on the Toilet, the Test Certified program, and the Test Mercenaries were all robust, independent entities, and I felt like I was the sole common thread between them. I prodded folks to articulate a unifying vision, and we eventually hit on the right one: Getting all teams to Test Certified Level Three—whether they were enrolled in the program or not. This not only aligned our efforts; it also enabled the perfect partnership with our testing staff. It provided a framework they could use to encourage their dev teams to test and catch bugs up-front, so the testers could provide higher-value quality feedback.

(And if I may be petty for a moment, I personally sold this concept to the testers in 2007, before a certain author had even joined the company.)

Notice that “Empower” remains a bit light. The tooling innovations weren’t yet in wide use, limiting the impact of our shared knowledge. We needed a way to drive tool adoption company-wide as rapidly as possible.

The Tipping Point

Eventually we realized Fixits were an incredibly powerful means of doing this. The Revolution Fixit in January 2008 made the first huge dent in the “I don’t have time to test” problem by rolling out our new cloud-based build infrastructure and Blaze (recently open-sourced as Bazel), the GNU Make replacement that took advantage of the new infrastructure. The primary cost teams had to pay to adopt these new tools was to clean up their dependencies and to ensure all build inputs were version-controlled—a cost they were all too glad to pay for faster builds. Coincidentally, fixing these problems and adopting faster tools made it far easier to try automated testing.

Not everyone participated in the Revolution, but everyone was impacted by it. In the following weeks, teams would talk about “Revolutionizing” their builds. And then, that September, the financial crisis indirectly demonstrated Testing Grouplet’s impact. We weren’t the sole reason Google was able to survive the crisis; but when the company let go of most of its contractors, including most manual testers and most of the Test Mercenaries, there wasn’t a quality crisis, or a development velocity slowdown. People by that point knew what the right thing to do was, and had the power to do it. The right thing had become easy enough.

After three and a half years of saturating the culture, we had the support and momentum we needed to finish building the Rainbow across the chasm—another year and a half later. The Test Automation Platform, launched during the TAP Fixit in March 2010, was a continuous integration system built upon the “Revolutionized” tools that could test every change and clearly indicate the source of a breakage, often so fast that most breakages were already reported and fixed before most people noticed.

After five years of breaking from the Standard Narrative, with success uncertain for most of it, we succeeded in making automated testing an indispensable practice of the Google development culture. So what was the observable impact five years after that? Rachel Potvin’s @Scale presentation from 2015, “Why Google Stores Billions of Lines of Code in a Single Repository”, contains some astonishing figures (that as of now, are two years out of date). While she enumerates a host of other tools that comprise this ecosystem, she makes a point of saying specifically: “TAP is our automated test infrastructure, without which this model would completely fall apart.” (13m:36s)

The Scarlet G

The reason I keep telling this story is to humanize this aspect of Google and to make it accessible, to provide insight into the strengths, vulnerabilities, and flaws of the humans that comprised it. In understanding and empathizing with these characters and their struggles, we can better see how our own thoughts and behaviors shape the course of events. We can draw from these examples the courage we need to stand up in the arena for ourselves and help create a better future.

Even so, some of you may look at me and only see “The Scarlet G”, the mark of the outsider, whose experiences are unique and unusual and either don’t apply to your Standard Narrative, or threaten it. “Google is special,” you may say. “They have smarter, younger people who are willing to take more risks.” Have you not heard what I’ve been telling you? Google’s a human organization like any other; it got some things right that many other organizations don’t, but it still consists of humans, some of whom are open-minded and brilliant, to be sure; but also some who were (or are) ignorant, overconfident, arrogant, and insecure to a fault, just like anywhere else. There were lots of developers straight out of school, but also a lot of people who’d done a decade or more in Silicon Valley and elsewhere. Remember: it took five years and many bold, parallel efforts before all these “smart, risk-taking people” finally got on the bus. And only a handful of us were behind the wheel from start to finish.

Or, you might say, “Googlers think they’re so smart, that whatever Google did is supposed to be the way things are done everywhere.” It’s true, we picked up a lot of exciting ideas from our incredible experiences as Googlers. But are we supposed to discard all those ideas and experiences when we leave to join a new organization? Is it true that they’ll never work in another environment? There’s no guarantee that specific mechanisms that worked for driving automated testing adoption at Google will work for transforming your organization. But maybe some variation or combination of these ideas are worth a try.

And remember: It took two years of experimentation until we finally understood the problem well enough that we were able to articulate a mission and strategy. We didn’t have the clear vision and specific plan figured out when we began; and we likely never would’ve if we kept waiting for both to snap in place before doing anything.

But this talk isn’t about the flyers we posted in bathrooms, or the certification program we developed, or the specific tools we wrote and rolled out in big, fun events. The real tool I’m offering to you is this framework, the “Rainbow of Death”, for examining why these things worked; a tool to help you develop empathy for the fears and challenges your colleagues face so that you can discover the right solution for your environment—a tool to break free from the Standard Narrative that’s holding you back. I want you to develop your own narrative, not passively listen to mine and comfortably conclude my friends and I were just lucky to get hired by Google.

We enjoyed some advantages, but we worked our asses off for five years, when many of the forces of the company’s culture were working against us. Those numbers of Rachel’s wouldn’t exist if we didn’t spend five years breaking from the Standard Narrative, instead of waiting to be told what to do, or waving our fingers in people’s faces and calling them stupid or dangerous.

The real challenge

While the technological advances were critically important to making the right thing the easy thing, the greater challenge was to create the cultural space necessary for lasting change. We had to be patient with ourselves to work out the nuances of the problem, and patient with our colleagues to adopt our practices once the pieces were in place.

Believe me, most days during those five years, whenever I’d see someone send a code change for review without any tests, and the reviewer approved it without asking for any, I’d want to go shout in their faces, “DON’T YOU KNOW HOW IRRESPONSIBLE YOU’RE BEING? STOP BEING SO CARELESS AND OVERCONFIDENT! DON’T YOU KNOW HOW MUCH YOU’RE HURTING YOUR COUNTRY—YOUR COMPANY?”

I think it’s pretty obvious to most of us that such an approach isn’t likely to persuade people to consider a different viewpoint. Not everyone will adopt new practices with the same open mind or the same degree of resistance, with the same educational background or life experiences—in short, with the same worldview. Everybody feels like their mind is the right balance of open and closed, whether they always seek the truth, to question and reaffirm it, or never do. As Instigators, it’s our responsibility to accept that truth, and to work with it.

For example: Today, I have the privilege of standing before you, and of deciding the path and presentation of this narrative. I’m striving to inspire you, the audience, to go forth with these ideas and make change happen.

But for each of you, this is very personal. Your focus is on the effect my words are having on your thoughts, your perception. You have your own reasons why some of what I say resonates, and why other things don’t. What you think is brilliant, the person back at your office who you may think most needs to hear it might think I’m peddling horseshit at best, or snake oil at worst. Or they may hear it and think, “Cool story, bro! But it won’t work for my special snowflake project because of superficial reasons X, Y, and/or Z.” Or perhaps it does resonate, but they don’t feel like they have the knowledge, power, freedom, or safety to act on it.

And to a vocal and tenacious minority, we shall always remain “the opposition party”. We can do little to fight it, and would gain little in doing so; we must succeed in spite of it.

We can’t rely solely on metrics and logical argument to change minds. Metrics can mark progress and validate improvement, and our logic must be as sound as possible while asserting our case; but they are far from sufficient to inspire action in either the skeptical or the powerless. Metrics and arguments don’t change most minds; experiences do.

The key to creating the necessary experiences is to discover the true pain and address it. Don’t get caught up in whether or not people have the right to feel or to complain about such pain, whether they’re entitled or stupid. It’s not your job to judge the people or the pain, it’s your job to understand it, and to make it go away. The problem you want to solve may not be the problem you have to solve first.

This won’t feel natural. It won’t feel right. It’ll feel contrary to every instinct you have to meet resistance with an opposite and greater force. It’s easier to attack than to engage¹²—because that is the Standard Narrative! And the most important thing to remember about the Standard Narrative is that it’s the very thing that created the situation we hope to change. It’s comprised of the motions we go through to perform business as usual with an incomplete or mistaken awareness of the downstream consequences.

How can we hope to change other people’s worldviews if we are incapable of changing our own? This is where the word “death” in the title comes into play, specifically the death our own ideas, our own ego. If we cling to these from the fear that our grasp on reality will slip and the world will fall apart, there is little hope that our “rainbow” will take shape and carry us across the chasm, to connect with the others, to heal the pain.¹³

The “Rainbow of Death” can help you approach your own problems with clarity and courage, with an open heart as well as an open mind, as you shape plans to break from the Standard Narrative that are the best fit for your culture and your capabilities.

I find Caravaggio’s David with the Head of Goliath¹⁴ particularly compelling because it is a self-portrait of the artist…as Goliath. Where others may find this image and its theme jarringly morbid, I see an inspirational breakthrough moment of self-awareness.

As Caravaggio discovered after a lifetime of battling his own demons, we in the Testing Grouplet discovered that the greatest force resisting us wasn’t external. The real opposition we faced was the country’s—company’s—own identity, our own conditioned instincts, our own culture. Often the greatest obstacle to the change we wish to see in the world is the way we, as individuals, organizations, and society at large, continue to see the world.

So now, I put the question to you: What is the pain that that the people around you feel? Where is it really coming from? How can you provide the knowledge and power necessary to relieve that pain? What resources do you have available? What fellow Instigators from all corners are waiting to join you in crossing the chasm? What can you do to intervene, validate, inform, inspire, mentor, and ultimately empower your people?

And how much longer can you afford to wait before taking action? Will you decide to follow the Standard Narrative—or choose your own?

Approved and improved by…

These are the fellow Instigators that provided feedback on this presentation, either directly via Google Slides comments or offline via email, phone calls, or face-to-face conversations. I am deeply grateful and indebted to them for their generous insights that made this presentation of far superior quality than I ever would’ve been capable of producing on my own.

• Eileen Bland
• Ramez Mourad
• David Plass
• Isaac Truett
• Ana Ulin
• Alan Kraft
• Christian Lilley
• Carlo Costino
• Tony Harper
• Marie-Line Germain
• Chris Cairns
• Robert L. Read
• Simmons Lough
• Pepper Lebeck-Jobe
• Andrew Trenk
• Mamie Rheingold
• Henry Poole
• Sean Johnson
• Luis Peralta
• Jason Ribeiro
• Bryan Kinney
• Michael Torres
• Kenneth Bible
• Andrew Stroup

History

2017-03-08: I first presented this talk on Wednesday, March 8, 2017 at 9:15am Eastern Standard Time (GMT+5) at DevOpsDays Baltimore 2017, the first DevOpsDays event in Baltimore.

2017-06-08: First delivery of the Schibsted Edition at leboncoin in Paris, tailored to demonstrate how the RoD model applies to Schibsted’s automated testing adoption efforts.

2017-07-11: Delivered the USPTO Edition at the DevOps@PTO event, tailored to demonstrate how the United States Patent and Trademark Office’s DevOps efforts fit the RoD model.

2017-07-13: Updated the core talk to include Schibsted Testing Fixit pictures, the commitment model, and the explanation of roles from the Schibsted and USPTO Editions, first presented at the Agile Government Sacremento Meetup.

2018-02-26: Delivered at WeWork at Tower 49 in NYC at the invitation of David Plass. This was the first instance to incorporate the line about “The power of Grouplets comes from the common language…” from my Cvent-internal “Intro to Grouplets” Ignite talk.

2018-05-29: Delivered a version for the Cvent MBA program that replaced the last third of the talk with a description of Grouplet activity at Cvent. It also included proposed “in-class” and “homework” exercises at the end. (These may get incorporated as an appendix to the core talk someday.)

2018-08-08: Updated the “Commitment model” to rename “Partially-committed members” as “Frequent contributors”.

Footnotes