Mike Bland

Copyright 2023 Mike Bland, licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

This presentation, Making Software Quality Visible, is my calling card. It describes my approach to driving software quality improvement efforts, especially automated testing, throughout an organization—applying systems thinking and leadership to influence its culture.

If you’d be interested in my help, please review the slides and script below. If my approach aligns with your sense of what your organization needs, please reach out to me at mbland@acm.org. (Also see my Hire me! page for more information.)

• Artifacts
• Abstract
• Introduction
• Agenda
• Google and Apple Examples
• What Software Quality Is and Why It Matters
• Why Software Quality Is Often Unappreciated and Sacrificed
• The Test Pyramid and Vital Signs
• Building a Software Quality Culture
• Calls to Action
• Acknowledgments
• History
• TODOs
• Footnotes

Artifacts

This Google Drive folder contains a recently reimagined, much shorter version:

• Making Quality Work Visible: 2023 DevOps Enterprise Forum

This is the latest, most complete version of the slides. Click Open in Keynote to view the presenter’s notes and download a copy of the Keynote file:

This Google Drive folder contains the a copy of the original Keynote presentation file as well as a PDF.

• Google Drive folder

Abstract

We’ll discuss why internal software quality matters, why it’s often unappreciated and sacrificed, and what we can do to improve it. More to the point, we’ll discuss the importance of instilling a quality culture to promote the proper mindset first. Only on this foundation will seeking better processes, better tools, better metrics, or AI-generated test cases yield the outcomes we can live with.

Introduction

I’m Mike Bland, I’m a programmer, and I’m going to talk about how making software quality visible…

Software Quality must be visible to minimize suffering.

…will minimize suffering.¹ By “suffering,” I mean: The common experience of software that’s painful—or even dangerous—to work on or to use. By “making software quality visible,” I mean: Providing meaningful insight into quality outcomes and the work necessary to achieve them.

Quality work can be hard to see. It’s hard to value what can’t be seen—or to do much of anything about it.

This is important because it’s often difficult to show quality work, or its impact on processes or products.² How do we show the value of avoiding problems that didn’t happen? This makes it difficult to prioritize or justify investments in quality, since people rarely value work or results they can’t see. Plus, people can’t effectively solve problems they can’t accurately sense or understand.

Agenda

• Google and Apple Examples
  I’ll share high-level examples of what I mean by making quality work visible to minimize suffering, based on my experience at Google and Apple.

• What Software Quality Is and Why It Matters
  We’ll define internal software quality and explain why it’s just as essential as external.

• Why Software Quality Is Often Unappreciated and Sacrificed
  We’ll examine several psychological and cultural factors that are detrimental to software quality.

• The Test Pyramid and Vital Signs
  We’ll use the Test Pyramid model to specify the principles underlying a sound testing strategy. We’ll also discuss the negative effects that its opposite, the Inverted Test Pyramid, imposes upon its unwitting victims. I’ll then describe how to use “Vital Signs” to get a holistic view on software quality for a particular project.

• Building a Software Quality Culture
  We’ll learn how to integrate the Quality Mindset into organizational culture, through individual skill development, team alignment, and making quality work visible.

• Calls to Action
  We’ll finish with specific actions everyone can take to lead themselves and others towards making software quality work and its impact visible.

Google and Apple Examples

Now here are some very brief summaries of my experiences with making quality work and its impact visible.

Google: Testing Grouplet

2005-2010, mike-bland.com/the-rainbow-of-death

At Google I was part of the Testing Grouplet, volunteers dedicated to driving automated testing adoption. I tell the full story in my talk “The Rainbow of Death.” Here are a few highlights:³

• Rapid growth, hiring the “best of the best,” build/test tools not scaling
  When I joined in 2005, the company was growing fast,⁴ and we knew we were “the best of the best.” However, our build and testing tools and infrastructure weren’t keeping up.

• Lack of widespread, effective automated testing and continuous integration; frequent broken builds and “emergency pushes” (deployments)
  Developers weren’t writing nearly enough automated tests, the ones they wrote weren’t that good, and few projects used continuous integration. As a result, code frequently failed to compile, and errors that made it to production would frequently lead to “emergency pushes,” or deployments.

• Training, GWS Story, Test Certified, Test Mercenaries, Testing on the Toilet
  Over time, the Testing Grouplet trained new hires, shared the Google Web Server story, and developed the Test Certified roadmap program. We built the Test Mercenaries internal consulting team, and our biggest hit was our Testing on the Toilet newsletter, appearing weekly in every company bathroom.

• Two Testing Fixits, Revolution (Build Tools) Fixit, Test Auto. Platform Fixit
  We also ran several “Fixits,” companywide events to address “important but not urgent” issues. Our Fixits inspired people to write and fix tests, to adopt new build tools, and finally, to adopt the Test Automation Platform continuous integration system.

• Better tests, practices, build and test times; fewer bugs, more productivity
  These efforts made quality work and its impact more visible than it had been. This helped people write better tests, adopt better testing practices and strategies, drastically improve build and test times, reduce bugs, and increase productivity. But perhaps the most visible result was scalability of the organization.

Google: Testing Grouplet results

2015, R. Potvin, Why Google Stores Bills. of LoC in a Single Repo

Rachel Potvin presented the following results in her presentation from @Scale 2015, “Why Google Stores Billions of Lines of Code in a Single Repository.” They may seem quaint to Googlers today, but they speak to the Testing Grouplet’s enduring impact five years after the TAP Fixit.

• 15 million LoC in 250K files changed by humans per week
• 15K commits by humans, 30K commits by automated systems per day
• 800K/second peak file requests

Of course, the Testing Grouplet isn’t responsible for all of this; Rachel’s talk describes an entire ecosystem of tools and practices. Even so, she states very clearly that:

• “TAP is our automated test infrastructure, without which this model would completely fall apart.” (13m:36s)

Also, it may amuse you to know that Testing on the Toilet, started in 2006, continues to this day!⁵

Apple: Quality Culture Initiative

2018-present

At Apple, I joined forces with a few others⁶ to start the Quality Culture Initiative, another volunteer group inspired by the Testing Grouplet.

• Rapid growth, hiring the “best of the best,” build/test tools not scaling
  When I joined in 2018, the company was growing fast, and we knew we were “the best of the best.” However, our build and testing tools and infrastructure weren’t keeping up.

• Widespread automated and manual testing, but…
  There was a strong testing culture, but not around unit testing.

• “Testing like a user would” often considered most important
  With so much emphasis on the end user experience, many believed that “testing like a user would” was the most important kind of testing.

• Tests often large, UI-driven, expensive, slow, flaky, and ineffective
  As a result, most tests were user interface driven, requiring full application and system builds on real devices. Since writing smaller tests wasn’t often considered, this led to a proliferation of large, expensive, slow, unreliable, and ineffective tests, generating waste and risk.

Apple: Quality Culture Initiative results

QCI activity as of November 2022—internal results confidential

It’s too early for the QCI to declare victory, and specific results to date are confidential. However, I can broadly describe the state of the QCI’s efforts by the time I left Apple in November 2022.

• Training: 16 courses, ~40 volunteer trainers, ~360 sessions, ~6100 check-ins, ~3200 unique individuals
  We launched an ambitious, wildly successful, and thriving training program to spread good coding and testing practices.

• Internal podcast: 45 episodes and 500+ subscribers
  Our internal podcast gave a voice to people of various roles and organizations across Apple to help drive the software quality conversation.

• Quality Quest roadmap: ~80 teams, ~20 volunteer guides
  Our Quality Quest roadmap, directly inspired by Test Certified, is helping teams across Apple improve their quality practices and outcomes.

• QCI Ambassadors: 6 organizations started, 6 on the way
  QCI Ambassadors help their organizations apply QCI principles, practices, and programs to achieve their quality goals.

• QCI Roadshow: over 50 presentations
  The QCI Roadshow helped us introduce QCI concepts and programs directly to groups across the company.

• QCI Summit: ~50 recorded sessions, ~60 presenters, ~850 virtual attendees
  Our QCI Summit event recruited presenters from across Apple to make their quality work and impact visible. We saw how QCI principles applied to operating systems and services, applications, frontends and backends, machine learning, internal IT, and development infrastructure.

What’s in a name?

What we realized three years after choosing it

One nice feature about the name “Quality Culture Initiative” that we didn’t realize for three years was how it encoded the total Software Quality solution:

• Quality is the outcome we’re working to achieve, but as I’ll explain, achieving lasting improvements requires influencing the…

• Culture. Culture, however, is the result of complex interactions between individuals over time. Any effective attempt at influencing culture rests upon systems thinking, followed by taking…

• Initiative to secure widespread buy in for systemic changes. Selling a vision for systemic improvement and supporting people in pursuit of that vision requires leadership.

What Software Quality Is and Why It Matters

As leaders, we need to clearly define what we mean by software quality, and explain why it’s so important.

Is High Quality Software Worth the Cost?

martinfowler.com/articles/is-quality-worth-cost.html

In May 2019, Martin Fowler published “Is High Quality Software Worth the Cost?,” a brief article describing the tradeoffs and benefits of software quality.

He distinguished between:

• External quality, which obviously makes users happy. This, in turn, keeps developers productive, since they don’t need to respond to problems reported by users. Then Martin argues that…

• Internal quality helps keep users happy, by enabling developers to evolve the software easily and to resolve problems quickly. This is because high internal quality makes developers productive, since there’s less cruft and unnecessary complexity slowing them down from making changes.

Effects of quality on productivity over time

Martin also used this hypothetical graph, based on his experience, to illustrate the impact of quality tradeoffs over time.

• With Low internal quality, progress is faster at the beginning, but begins to flatten out quickly.

• With High internal quality, progress is slower at the beginning, but the investment pays off in greater productivity over time.

• The break even point between the two approaches arrives within weeks, not months.

• Though Martin’s original graph didn’t show this, the difference in productivity between low and high internal quality is one way to visualize technical debt.⁷

“Fast, cheap, or good: pick two three”

High quality software is cheaper to produce

Martin’s conclusion is that higher quality makes software cheaper to produce in the long run—that the “cost” of high quality software is negative. “Fast, cheap, or good: pick two” doesn’t hold as the system evolves. It may make sense at first to sacrifice good to get a cheaper product to market quickly. But over time, investing in “good” is necessary to continue delivering a product quickly and at a competitive cost.

Internal quality aids understanding

Clarity increases productivity and resilience, manages risk

Internal quality essentially helps developers continue to understand the system as it changes over time:⁸

• Fosters productivity due to the clarity of the impact of changes
  When they clearly understand the impact of their changes, they can maintain a rapid, productive pace.

• Prevents foreseeable issues, limits recovery time from others
  Understanding helps them prevent many foreseeable issues, and resolve any bugs quickly and effectively.

• Provides a buffer for the unexpected, guards against cascading failures
  These qualities help create a buffer for handling unexpected events,⁹ while also guarding against cascading failures.

• Your Admins/SREs will thank you! It helps them resolve prod issues faster.
  Your system administrators or SREs will be very grateful for building such resilience into your system, as it helps their response times as well.

• Counterexamples: Global supply chain shocks; Southwest Airlines snafu
  For counterexamples, recall the global supply chain shocks resulting from the COVID-19 pandemic, or the December 2022 Southwest Airlines snafu. These systems worked very efficiently in the midst of normal operating conditions. However, their intolerance for deviations from those conditions rendered them vulnerable to cascading failures.

• Quality, clarity, resilience are essential requirements of prod systems
  Consequently, internal software quality, and the clarity and resilience it enables, are essential requirements of any production software system.

Focusing on internal software quality is good for business…because it’s the right thing to do.

As mentioned, Martin Fowler’s argument is that internal software quality is good for business—it’s such a compelling argument that I brought it up first. However, he prefers making only this economic argument for quality. He asserts that appeals to professionalism are moralistic and doomed, as they imply that quality comes at a cost.¹⁰

I disagree that we should sidestep appeals to professionalism entirely, and that they’re incompatible with the economic argument. I think it’s worth exploring why professionalism matters, both because it is moral and because customers increasingly expect high quality software they can trust.

Quality without function is useless—but function without quality is dangerous.

Put more bluntly, high quality may be useless without sufficient functionality, but as we’ll see, functionality without quality can be dangerous. Let’s look at a few examples.

Northrop Grumman Mission Systems

Navigation for US Coast Guard vessels or US Navy submarines

My first professional programming project was a nautical chart renderer for a navigation system used by Coast Guard vessels and Navy nuclear submarines.

• Requirement: Enumerate chart features
  One day our product owner sent us some code to enumerate nautical chart features from a file.

• Assumption: In memory size == on disk size
  The code assumed each record was the same size in memory as it was on disk.

• Reality: 21 bytes on disk, 24 in memory
  However, the records were 21 bytes on disk, but the in memory structs were 24 bytes, thanks to byte padding.

• Outcome: File size/24 == 12.5% data loss
  As a result, this code ignored one eighth of the chart features in the file.

• Impact: Caught before shipping!
  Fortunately I caught this before it shipped to any nuclear submarines.

Apple’s goto fail

Finding More Than One Worm in the Apple, CACM, July 2014

In February 2014, Apple had to update part of its Secure Transport component…

• Requirement: Apply algorithm multiple times
  …which applied the same algorithm in six places.

• Assumption: Short algorithms safe to copy
  The developers apparently assumed that this short, ten line algorithm was safe to copy in its entirety, instead of making it a function.

• Reality: Copies may not stay identical
  One problem with duplication is that the copies may not remain identical.

• Outcome: One of six copies had a bug
  As it so happened, one of the six copies of this algorithm picked up an extra “goto” statement that short circuited a security handshake.

• Impact: Billions of devices
  Once it was discovered and patched, Apple had to push an emergency update to billions of devices. It’s unknown whether it was ever exploited.

My article “Finding More Than One Worm in the Apple” explains how this bug could’ve been caught, or prevented, by a unit test.

OpenSSL’s Heartbleed

Goto Fail, Heartbleed, and Unit Testing Culture, May 2014

In April 2014, OpenSSL had to update its “heartbeat” feature…

• Requirement: Echo message from request
  …which echoed a message supplied by a user request.

• Assumption: User-supplied length is valid
  The code assumed that the user supplied message length matched the actual message length.

• Reality: Actual message may be empty
  In fact, the message could be completely empty.

• Outcome: Server returns arbitrary data
  In that case, the server would hand back however many bytes of its own memory that the user requested, including secret key data.

• Impact: Countless HTTPS servers
  Countless HTTPS servers had to be patched. It’s unknown whether it was ever exploited.

My article “Goto Fail, Heartbleed, and Unit Testing Culture” explains how this bug could’ve been caught, or prevented, by a unit test.

The point is that…

Quality Culture is ultimately Safety Culture

…a culture that values and invests in software quality is a Safety Culture. Society’s dependence on software to automate critical functions is only increasing. It’s our duty to uphold that public trust by cultivating a quality culture.

Why Software Quality Is Often Unappreciated and Sacrificed

Our next step as leaders is to understand, if software quality is so important, why it’s so often unappreciated and sacrificed.

Automated (especially unit) testing

Why hasn’t it caught on everywhere yet?

• Apple, in January 1992, identified the need to make time for training, documentation, code review—and unit testing!
  

At Apple, I found a document from January 1992 specifically identifying the need to make time for training, documentation, code review—and unit testing! That’s not just before Test-Driven Development and the Agile Manifesto, that’s before the World Wide Web!

There are a few reasons why unit testing in particular hasn’t caught on everywhere yet:¹¹

• People think it’s obvious and easy—therefore lower value
  Many developers think it’s obvious and easy, and therefore can’t provide much value.¹²

• Many still haven’t seen it done well—or may have seen it done poorly
  Many others still haven’t seen it done well, or may have seen it done poorly, leading to the belief that it’s actually harmful.

• There’s always a learning curve involved
  For those actually open to the idea, there’s still a learning curve, which they many not have the time to climb.

• Bad perf review if management doesn’t care about testing/internal quality
  They may also fear spending time on testing and internal quality will result in a bad performance review if their management doesn’t care about it.¹³

Have you heard these ones before?

Common excuses for sacrificing unit testing and internal quality

Of course, people give their own reasons for not investing in testing and quality, including the following:

• Tests don’t ship/internal quality isn’t visible to users (i.e., cost, not investment)
  “We don’t ship tests, and users don’t care about internal quality.” Meaning, testing seems like a cost, not an investment.

• “Testing like a user would” is the most important kind of testing
  As mentioned before, “Testing like a user would” is considered most important, so investing in smaller tests and internal quality seems unnecessary.

• Straw man: 100% code coverage is bullshit
  The straw man that “writing tests to get 100% code coverage is bullshit.” This speaks to a fundamental ignorance about how to write good tests or to use coverage the right way.

• Straw man: Testing is a religion (implying: I’m better than those people)
  For some reason, technical people, especially programmers, like to pound their chests as being against so called testing “religion” and those who practice it. It’s a flimsy excuse for trying to score social points by virtue signaling in view of one’s perceived peers. Framing a potentially reasonable discussion of different testing ideas in such a way only serves to shut it down for a superficial, unprofessional ego boost.¹⁴

• “My code is too hard to test.” (The Snowflake Fallacy)
  The common, evidence free “My code is too hard to test” assertion, which I call the Snowflake Fallacy.

• “I don’t have time to test.”
  Finally, “I don’t have time to test.” This could be a brush off, or a genuine indication that they don’t know how and can’t spare the time to learn—and management doesn’t care.

Business as Usual

All these reasons are why Business as Usual persists, as well as the Complexity, Risk, Waste, and Suffering that everyone’s used to. This then allows the Normalization of Deviance to take hold.

Normalization of Deviance (paraphrased)

Coined by Diane Vaughan in The Challenger Launch Decision

Diane Vaughan introduced this term in her book about the Space Shuttle Challenger explosion in January 1986.¹⁵ My paraphrased version of the definition is: A gradual lowering of standards that becomes accepted, and even defended, as the cultural norm.

Space Shuttle Challenger Accident Report

history.nasa.gov/rogersrep/genindex.htm, Chapter VI, pp. 129-131

The key evidence of this phenomenon is articulated in chapter six of the Rogers commission report.¹⁶

• The O-rings didn’t just fail on the Challenger mission on 1986-01-28…
  Many of us may know that the O-rings lost elasticity in the cold weather, allowing gasses to escape which led to the explosion.

• …anomalies occurred in 17 of the 24 (70%) prior Space Shuttle missions…
  However, you may not realize that NASA detected anomalies in O-ring performance in 17 of the previous 24 shuttle missions, a 70 percent failure rate.

• …and in 14 of the previous 17 (82%) since 1984-02-03
  Even scarier, anomalies were detected in 14 of the previous 17 missions, for an 82% failure rate.

• Multiple layers of engineering, management, and safety programs failed
  

This wasn’t only one person’s fault—multiple layers of engineering, management, and safety programs failed.¹⁷ However, Normalization of Deviance isn’t the end of the problem.

NASA: NoD leads to Groupthink

Terry Wilcutt and Hal Bell of NASA delivered their presentation The Cost of Silence: Normalization of Deviance and Groupthink in November 2014.¹⁸ On the Normalization of Deviance, they noted that:

“There’s a natural human tendency to rationalize shortcuts under pressure, especially when nothing bad happens. The lack of bad outcomes can reinforce the ‘rightness’ of trusting past success instead of objectively assessing risk.”

—Terry Wilcutt and Hal Bell, The Cost of Silence: Normalization of Deviance and Groupthink

They go on to cite the definition of Groupthink from Irving Janis:

“[Groupthink is] a quick and easy way to refer to a mode of thinking that persons engage in when they are deeply involved in a cohesive in-group, when concurrence-seeking becomes so dominant that it tends to override critical thinking or realistic appraisal of alternative courses of action.”

—Irving Janis, Groupthink: psychological studies of policy decisions and fiascoes

NASA: Symptoms of Groupthink

They then describe the symptoms of Groupthink:¹⁹

1. Illusion of invulnerability—because we’re the best!
2. Belief in Inherent Morality of the Group—we can do no wrong!
3. Collective Rationalization—it’s gonna be fine!
4. Out-Group Stereotypes—don’t be one of those people!
5. Self-Censorship—don’t rock the boat!
6. Illusion of Unanimity—everyone goes along to get along!
7. Direct Pressure on Dissenters—because everyone else disagrees!
8. Self-Appointed Mindguards—decision makers exclude subject matter experts from the conversation.

Any of these sound familiar? Hopefully from past, not current, experiences.

A common result of Groupthink is the well known systems thinking phenomenon called “The Cobra Effect.”

The Cobra Effect

ourworld.unu.edu/en/systems-thinking-and-the-cobra-effect

• Pay bounty for dead cobras
  This comes from the true story of when the British administration in India offered people a bounty to help reduce the cobra population.

• Cobras disappear, but still paying
  This worked, but the British noticed they kept paying bounties when they didn’t see any more cobras around.

• People were harvesting cobras
  They realized people were raising cobras just to collect the bounty…

• Ended bounty program
  …so they ended the bounty program.

• More cobras in streets than before!
  People then threw their now useless cobras into the streets, making the problem worse than before.

• Fixes that Fail: Simplistic solution, unforeseen outcomes, worse problem
  This is an example of the “Fixes That Fail” archetype. This entails applying an overly simplistic solution to a complex problem, resulting in unforeseen outcomes that eventually make the problem worse.

The Arms Race

Systems thinking should replace brute force in the long run.

In software, I call this “The Arms Race.” This may sound familiar:

• Investment to create capacity for existing practices and processes…
  We invest people, tools, and infrastructure into expanding the capacity of our existing practices and processes.

• Exhaustion of capacity leads to more people, tools, and infrastructure
  Things are better for a while, but as the company and its projects and their complexity grow, that capacity’s eventually exhausted. This leads to further investment of people, tools, and infrastructure.

• Then that capacity’s exhausted…
  Then eventually that capacity’s exhausted…and the cycle continues.

• AI-generated tests may help people get started in some cases…
  There’s a lot of buzz lately about possibly using AI-generated tests to take the automated testing burden off of humans. There may perhaps be room for using AI-generated tests as a starting point.

• …but beware of abdicating professional responsibility to unproven tools.
  However, AI can’t read your mind to know all your expectations—all the requirements you’re trying to satisfy or all the assumptions you’re making.²⁰ Even if it could, AI will never absolve you of your professional responsibility to ensure a high quality and trustworthy product.²¹

• In the end, we can’t win the arms race against growth and complexity.
  We need to realize we can’t win the Arms Race against growth and complexity.

Rogers Report Volume 2: Appendix F

history.nasa.gov/rogersrep/v2appf.htm

Richard Feynman’s final statement on the Challenger disaster is a powerful reminder of our human limitations:²²

“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.”

—Richard Feynman, Personal Observations on Reliability of Shuttle

As professionals, we must resist the Normalization of Deviance, Groupthink, and the Arms Race. We can’t allow them to become, or to remain, Business as Usual.

Don’t shame people for doing Business as Usual. Help them recognize and change it.

But we can’t shame anyone for falling into these common traps, because it’s human nature to fall into them. We need to help everyone recognize them, focus on avoiding them, and remain vigilant against them, and change Business as Usual together.

Challenging & Changing Business as Usual

Changing cultural norms requires understanding them first.

Challenging cultural norms supporting Business as Usual isn’t easy, and it’s honestly frightening. There’s actually a good reason for that:

• Everything in a culture happens for a reason—challenge reasons thoughtfully!
  The existing norms do exist for a reason. The question is whether that reason holds up today. So we need to try to understand those reasons first, then challenge them thoughtfully.²³

• People often feel invested in old ways and fear the cost of new ways.
  This is because many people feel invested in their existing methods. They fear the cost and risk of changing those methods, even if they no longer provide the value they once did. This is common human nature…

• We’re often asked for data to prove a different approach actually helps—before trying it…
  …and is why some will ask for data or other proof that a change will be effective before trying it.

• …while we throw time, money, people, and tools at existing processes.
  At the same time, they’ll continue throwing resources into existing processes as they have for years.

The most productive way to approach such a challenge requires taking the time to gather enough information and build some trust.

• Challenge: Haven’t we proven that the existing ways aren’t (totally) working?
  You can then carefully question whether current methods are effective, or effective enough on their own, given substantial historical evidence to the contrary. On this basis, you may persuade some to reexamine the problem and try a different approach.

Notice that this challenge to the status quo is in the form of a thoughtful question.

The power of asking good questions

“What could we do differently to improve our software quality?”

The ultimate question is “What could we do differently to improve our software quality?”²⁴ However…

• Questions develop shared understanding of the culture before changing it
  …there are many more questions necessary to help everyone understand why things are the way they are, what needs to change—and how.

• Asking encourages thinking about a problem, possible solutions
  Asking good questions includes people in the process of discovery and finding solutions, which develops their own knowledge and problem solving skills.

• Good questions enable new information, perspectives, and ideas to emerge
  Good questions enable people to share information, perspectives, and ideas that wouldn’t otherwise arise if they were only told what to think or do.

• Asking what to do is more engaging than telling what to do—produces buy in
  Taking the time to ask people questions pulls people into the change process, which increases their motivation to buy into any proposed changes that emerge.

This is summed up by a great Senegalese proverb shared by Bono in Chapter 20 of John Doerr’s Measure What Matters:

• “If you want to cut a man’s hair, it is better if he is in the room.”
  

There is one catch, though.

If people don’t know where to begin, are stuck in old ways, or are under stress…direct near term guidance may be necessary.

Your audience may get stuck. They may currently have no idea about what to change or how—because they lack knowledge, experience, or imagination to consider approaches beyond the status quo. Or, as we’ll discuss shortly, they may be under incredible stress and unable to think clearly or creatively.

In that case, you may need to provide more direct guidance, at least in the beginning.²⁵ So here are some tools for providing that guidance.

The Test Pyramid and Vital Signs

Lasting improvements to software quality—and Business as Usual—begin with making quality work and its impact visible at a fundamental level. The Test Pyramid and Vital Signs are concepts accessible to any team committed to this principle. Applying these models is the first step towards ending the suffering caused by the Normalization of Deviance, Groupthink, and the Arms Race.

Before getting into the details, let’s understand the specific problems we need to solve.

Working back from the desired experience

Inspired by Steve Jobs Insult Response

In this famous Steve Jobs video, he explains the need to work backward from the customer experience, not forward from the technology. So let’s compare the experience we want ourselves and others to have with our software to the experience many of us may have today.

What we want	What we have
Delight	Suffering
Efficiency	Waste
Confidence	Risk
Clarity	Complexity

• What we want
  We want to experience Delight from using and working on high quality software,²⁶ which largely results from the Efficiency high quality software enables. Efficiency comes from the Confidence that the software is in good shape, which arises from the Clarity the developers have about system behavior.

• What we have
  However, we often experience Suffering from using or working on low quality software, reflecting a Waste of excess time and energy spent dealing with it. This Waste is the result of unmanaged Risk leading to lots of bugs and unplanned work. Bugs, unplanned work, Risk, and fear take over when the system’s Complexity makes it difficult for developers to fully understand the effect of new changes.

Difficulty in understanding changes produces drag— i.e., Technical debt.

Difficulty in understanding how new changes could affect the system is the telltale sign of low internal quality, which drags down overall quality and productivity. Recall from earlier that the difference between actual and potential productivity correlated to internal quality is what we identified earlier as technical debt.

This contributes to the common scenario of a crisis emerging…

Replace heroics with a Chain Reaction!

…that requires technical heroics and personal sacrifice to avert catastrophe. To get a handle on avoiding such situations, we need to create the conditions for a positive Chain Reaction. By creating and maintaining the right conditions over time, we can achieve our desired outcomes without stress and heroics.

The main obstacle to replacing heroics with a Chain Reaction isn’t technology…

The challenge is belief—not technology

A little awareness goes a long way

…it’s an absence of awareness or belief that a better way exists.^27,28

• Many of these problems have been solved for decades
  Despite the fact that many quality and testing problems have been solved for decades…²⁹

• Many just haven’t seen the solutions, or seen them done well…
  …many still haven’t seen those solutions, or seen them done well.³⁰

• The right way can seem easy and obvious—after someone shows you!
  The good news is that these solutions can seem easy and obvious—after they’ve been clearly explained and demonstrated.³¹

• What does the right way look like?
  So how do we get started showing people what the right way to improve software quality looks like?

The Test Pyramid

A balance of tests of different sizes for different purposes

We’ll start with the Test Pyramid model,^32,33 which represents a balance of tests of different sizes for different purposes. Realizing that tests can come in more than one size is often a major revelation to people who haven’t yet been exposed to the concept.³⁴ It’s not a perfect model—no model is—but it’s an effective tool for pulling people into a productive conversation about testing strategies for the first time.³⁵

The Test Pyramid helps us understand how different kinds of tests give us confidence in different levels and properties of the system.³⁶ It can also help us break the habit of writing large, expensive, flaky tests by default.³⁷

• Small tests are unit tests that validate only a few functions or classes at a time with very few dependencies, if any. They often use test doubles³⁸ in place of production dependencies to control the environment, making the tests very fast, independent, reliable, and cheap to maintain. Their tight feedback loop³⁹ enables developers to detect and repair problems very quickly that would be more difficult and expensive to detect with larger tests. They can also be run in local and virtualized environments and can be parallelized.

• Medium tests are integration tests that validate contracts and interactions with external dependencies or larger internal components of the system. While not as fast or cheap as small tests, by focusing on only a few dependencies, developers or QA can still run them somewhat frequently. They detect specific integration problems and unexpected external changes that small tests can’t, and can do so more quickly and cheaply than large system tests. Paired with good internal design, these test can ensure that test doubles used in small tests remain faithful to production behavior.⁴⁰

• Large tests are full, end to end system tests, often driven through user interface automation or a REST API. They’re the slowest and most expensive tests to write, run, and maintain, and can be notoriously unreliable. For these reasons, writing large tests by default for everything is especially problematic. However, when well designed and balanced with smaller tests, they cover important use cases and user experience factors that aren’t covered by the smaller tests.

Thoughtful, balanced strategy == Reliability, efficiency

Each test size validates different properties that would be difficult or impossible to validate using other kinds of tests. Adopting a balanced testing strategy that incorporates tests of all sizes⁴¹ enables more reliable and efficient development and testing—and higher software quality, inside and out.

Inverted Test Pyramid

Many larger tests, few smaller tests

Of course, many projects have a testing strategy that resembles an inverted Test Pyramid, with too many larger tests and not enough smaller tests. This leads to a number of common problems:

• Tests tend to be larger, slower, less reliable
  The tests are slower and less reliable than they could be compared to relying more on smaller tests.

• Broad scope makes failures difficult to diagnose
  Because large tests execute so much code, it might not be easy to tell what caused a failure.

• Greater context switching cost to diagnose/repair failure
  That means developers have to interrupt their current work to spend significant time and effort diagnosing and fixing any failures.

• Many new changes aren’t specifically tested because “time”
  Since most of the tests are large and slow, this incentivizes developers to possibly skip writing or running them because they “don’t have time.”

• People ignore entire signal due to flakiness…
  Worst of all, since large tests are more prone to be flaky,⁴² people will begin to ignore test failures in general. They won’t believe their changes cause any failures, since the tests were failing before—they might even be flagged as “known failures.”⁴³ And as we’ll recall…

• …fostering the Normalization of Deviance
  …the Space Shuttle Challenger O-rings suffered from “known failures” as well, cultivating the Normalization of Deviance.

Causes

Let’s go over some of the reasons behind this situation, touching on some of the same reasons we covered before.

• Features prioritized over internal quality/tech debt
  People are often pressured to continue working on new features that are “good enough” instead of reducing technical debt. This may be especially true for organizations that set aggressive deadlines and/or demand frequent live demonstrations.⁴⁴

• “Testing like a user would” is more important
  Again, if “testing like a user would” is valued more than other kinds of testing, then most tests will be large and user interface-driven.

• Reliance on more tools, QA, or infrastructure (Arms Race)
  This also tends to instill the mindset that the testing strategy isn’t a problem, but that we always need more tools, infrastructure, or QA headcount. This is the Arms Race mindset we discussed earlier.

• Landing more, larger changes at once because “time”
  Because the existing development and testing process is slow and inefficient, individuals try to optimize their productivity by integrating large changes at once. These changes are unlikely to receive either sufficient testing or sufficient code review, increasing the risk of bugs slipping through. It also increases the chance of large test failures that aren’t understood. The team is inclined to tolerate these failures, because there isn’t “time” to go back and redo the change the right way.

• Lack of exposure to good examples or effective advocates
  As mentioned before, many people haven’t actually witnessed or experienced good testing practices before, and no one is advocating for them. This instills the belief that the current strategy and practices are the best we can come up with.

• We tend to focus on what we directly control—and what management cares about! (Groupthink)
  In such high stress situations, it’s human nature to focus on doing what seems directly within our control in order to cope. Alternatively, we tend to prioritize what our management cares about, since they have leverage over our livelihood and career development. It’s hard to break out of a bad situation when feeling cornered—and too easy to succumb to Groupthink without realizing it.

So how do we break out of this corner—or help others to do so?

Quality work can be hard to see. It’s hard to value what can’t be seen—or to do much of anything about it.

We have to overcome the fundamental challenge of helping people see what internal quality looks like. We have to help developers, QA, managers, and executives care about it and to resist the Normalization of Deviance and Groupthink. We need to better show our quality work to help one another improve internal quality and break free from the Arms Race mindset.

In other words, internal quality work and its impact is a lot like The Matrix…

“Unfortunately, no one can be told what the Matrix is. You have to see it for yourself.”

—Morpheus, The Matrix

One way to start showing people The Matrix is to get buy-in on a set of…

Vital Signs

…“Vital Signs.” Vital Signs are…

A collection of signals designed by a team to reflect quality and productivity and to rapidly diagnose and resolve problems

Intent

• Comprehensive and make sense to the team and all stakeholders.
  They should be comprehensive and make sense at a high level to everyone involved in the project, regardless of role.

• Not merely metrics, goals, or data
  We’re not collecting them for the sake of saying we collect them, or to hit a goal one time and declare victory.⁴⁵

• Information for repeated evaluation
  We’re collecting them because we need to evaluate and understand the state of our quality and productivity over time.

• Inform decisions whether or not to act in response
  These evaluations will inform decisions regarding how to maintain the health of the system at any moment.

Common elements

Some common signals include:

• Pass/fail rate of continuous integration system
  The tests should almost always pass, but failures should be meaningful and fixed immediately.

• Size, build and running time, and stability of small/medium/large test suites
  The faster and more stable the tests, the fewer resources they consume, and the more valuable they are.

• Size of changes submitted for code review and review completion times
  Individual changes should be relatively small, and thus easier and faster to review.

• Code coverage from small to medium-small test suites
  Each small-ish test should cover only a few functions or classes, but the overall coverage of the suite should be as high as possible.⁴⁶

• Passing use cases covered by medium-large to large and manual test suites
  For larger tests, we’re concerned about whether higher level contracts, use cases, or experience factors are clearly defined and satisfied before shipping.

• Number of outstanding software defects and Mean Time to Resolve
  Tracking outstanding bugs is a very common and important Vital Sign. If you want to take it to the next level, you can also begin to track the Mean Time to Resolve⁴⁷ these bugs. The lower the time, the healthier the system.

Other potentially meaningful signals

Some other potentially meaningful signals include…

• Static analysis findings (e.g., complexity, nesting depth, function/class sizes)
  Popular source control platforms, such as GitHub, can incorporate static analysis findings directly into code reviews as well. This encourages developers to address findings before they land in a static analysis platform report.⁴⁸

• Dependency fan-out
  Dependencies contribute to system and test complexity, which contribute to build and test times. Cutting unnecessary dependencies and better managing necessary ones can yield immediate, substantial savings.

• Power, performance, latency
  These user experience signals aren’t caught by traditional automated tests that evaluate logical correctness, but are important to monitor.

• Anything else the team finds useful for its purposes
  As long as it’s a clear signal that’s meaningful to the team, include it in the Vital Signs portfolio.

Use them much like production telemetry

Treat Vital Signs like you would any production telemetry that you might already have.

• Keep them current and make sure the team pays attention to them.
  

• Clearly define acceptable levels—then achieve and maintain them.
  

• Identify and respond to anomalies before urgent issues arise.
  

• Encourage continuous improvement—to increase productivity and resilience.
  

• Use them to tell the story of the system’s health and team culture.
  

To this last point, we’ll return to the importance of storytelling later.

Example usage: issues, potential causes (not exhaustive!)

Here are a few hypothetical examples of how Vital Signs can help your team identify and respond to issues.

• Builds 100% passing, high unit test coverage, but high software defects
  If your builds and code coverage are in good shape, but you’re still finding bugs…

  • Maybe gaps in medium-to-large test coverage, poorly written unit tests
    …it could be that you need more larger tests. Or, it could be your unit tests aren’t as good as you think, executing code for coverage but not rigorously validating the results.

• Low software defects, but schedule slipping anyway
  If you don’t have many bugs, but productivity still seems to be dragging…

  • Large changes, slow reviews, slow builds+tests, high dependency fan out
    …maybe people are still sending huge changes to one another for review. Or maybe your build and test times are too slow, possibly due to excess dependencies.

• Good, stable, fast tests, few software defects, but poor app performance
  Maybe builds and tests are fine, and there are few if any bugs, but the app isn’t passing performance benchmarks.

  • Discover and optimize bottlenecks—easier with great testing already in place!
    In that case, your investment in quality practices has paid off! You can rigorously pursue optimizations, without the fear that you’ll unknowingly break behavior.

Getting started, one small step at a time

Here are a few guidelines for getting started collecting Vital Signs. First and foremost…

• Don’t get hung up on having the perfect tool or automation first.
  Do not get hung up on thinking you need special tools or automation at the beginning. You may need to put some kind of tool in place if you have no way to get a particular signal. But if you can, collect the information manually for now, instead of wasting time flying blind until someone else writes your dream tool.

• Start small, collecting what you can with tools at hand, building up over time.
  You also don’t need to collect everything right now. Start collecting what you can, and plan to collect more over time.

• Focus on one goal at a time: lowest hanging fruit; biggest pain point; etc.
  As for which Vital Signs to start with, that’s totally up to you and your team. You can start with the easiest signals, or the ones focused on your biggest pain points—it doesn’t matter. Decide on a priority and focus on that first.

• Update a spreadsheet or table every week or so—manually, if necessary.
  If you don’t have an automated collection and reporting system handy, then use a humble spreadsheet or wiki table. Spend a few minutes every week updating it.

• Observe the early dynamics between the information and team practice.
  Discuss these updates with your team, and see how it begins to shift the conversation—and the team’s behavior.

• Then make a case for tool/infrastructure investment based on that evidence.
  Once you’ve got evidence of the value of these signals, then you can justify and secure an investment in automation.⁴⁹

Building a Software Quality Culture

By asking good questions, spreading awareness of the Test Pyramid, and making internal quality visible via Vital Signs, you’re shifting the culture of your team. The next step is to change the culture of your organization.

First, let’s define specifically what we mean by culture.⁵⁰ One possible definition is that:

Culture is the shared lifestyle of a team or organization.

This lifestyle is what you see people doing together day to day, and the way they do it. For our purposes, we need to understand the essence of lifestyle, where it comes from and what shapes it. So here’s an expanded definition of “culture”:

Culture is the emergent result of a shared mindset manifest through concrete behaviors.

In order to influence lifestyle, which is the result, we have to influence concrete behaviors. In order to influence those, we need to influence people’s mindset. The most effective way to influence mindsets is to…

Sell—don’t tell!

People don’t like being told to change their behaviors, because it’s like being told to change their minds. If you know anything about people, you know they hate changing their minds unless they are doing the changing, by their own choice.

This is why we’ve emphasized asking questions, raising awareness, and working together to make quality visible—instead of imposing process changes or technical solutions through force. People need to understand and buy into changes in order to embrace them fully. We can’t force them to make changes they don’t perceive as necessary or valuable if we want the change to be successful.

Of course, not everyone’s going to change their mindset at once—some may never come around at all. However, our ultimate goal should be to…

Make the right thing the easy thing!⁵¹

As we continue working to improve quality and make it visible, it will get easier and easier to do both. Practices and their results will become more accessible, encouraging wider and wider adoption. Eventually, we want to make it harder not to do the right thing, because the right thing will happen by default.

This will be challenging, and take time. It’s important to identify the right people to engage directly in the beginning, when we’re starting the process, and who to put off until later.

Focus on the Early Majority/Total Product

Geoffrey A. Moore, Crossing the Chasm, 3rd Edition

The “Crossing the Chasm” model from Geoffrey Moore’s book of the same name can help us make that identification.⁵² There are many nuances to it, but at a high level, it illustrates how different segments of a population respond to a particular innovation.

• Innovators and Early Adopters are like-minded seekers, enthusiasts and visionaries who together bring an innovation to the market and lead people to adopt it. I like to lump them together and call them Instigators.

• The Early Majority are pragmatists who are open to the new innovation, but require that it be accessible and ready to use before adopting it.

• The Late Majority are followers waiting to see whether or not the innovation works for the Early Majority before adopting it.

• Laggards are the resisters who feel threatened by the innovation in some way and complain about it the most. They may potentially raise valid concerns, but often they only bluster to rationalize sticking with the status quo.⁵³

The Instigators face the challenge of bringing an innovation across The Chasm separating them from the Early Majority, developing what Moore calls The Total Product. Developing the Total Product requires that the Instigators identify and fulfill several needs the Early Majority have in order to facilitate adoption.

As Instigators leading people to improve software quality and make it visible, focus your energy on connecting with other Instigators and the early Early Majority. Don’t worry so much about the rest—focus on delivering the Total Product, and it will take care of the other groups.

The Rainbow of Death

mike-bland.com/the-rainbow-of-death

This connection across the chasm isn’t part of the original Chasm model, but one I borrowed from a friend⁵⁴ and called “The Rainbow of Death.” It helps illustrate those Early Majority needs the Instigators must satisfy. Doing so transforms the Early Majority from being dependent on the Instigators’ expertise into independent experts themselves.

My talk “The Rainbow of Death,” linked here, uses this model to tell the story of Google’s Testing Grouplet. It extracts order and clarity from five years of chaos, evolution, and eventually revolution.⁵⁵ However, I’ve realized that it’s a great storytelling device, but too complicated a device to apply at the beginning of the change process.⁵⁶

Instigating Culture Change

Essential needs an internal community must support

Instead, once you’ve started forming an internal community of like-minded fellow Instigators, it’s best to focus on these essential needs and simplify your initial efforts:⁵⁷

• Individual Skill Acquisition
• Team/Organizational Alignment
• Quality Work/Results Visibility

Supporting each of these needs also helps support the other two, creating a virtuous cycle. No matter where you decide to focus as a starting point, the cycle gains momentum with every contribution of effort.⁵⁸ However, it is important to focus on getting one effort fully in motion before trying to launch the next one.⁵⁹

I’m going to give an overview of each of these needs and some ideas of how to address them. You can use these suggestions to determine a starting point, and to start percolating ideas for future efforts.

Individual Skill Acquisition

Help individuals incorporate principles, practices, language

Quality begins with the choices each of us make as individuals throughout our day. Awareness of sound quality principles and practices improves the quality of these choices. Developing a common language makes these principles and practices visible, so we can show them to one another, helping raise everyone’s quality game.

• Training, documentation, internal media, mentorship, sharing examples
  We can offer training, documentation, and other internal media to spread awareness. We can also offer direct mentorship or share examples from our own experience to help others learn.

Now here’s a quick, high level summary of a few key principles and practices to help developers write better code and tests.

• Testable code/architecture is maintainable—tests add design pressure, enable continuous refactoring; use code coverage as a tool, not a goal
  Designing code for testability, given proper guidance on principles and techniques, adds design pressure that yields higher quality code in general. Having good tests then enables constant improvements to code quality through continuous refactoring, instead of stopping the world for complex, risky overhauls or rewrites.⁶⁰

  Good tests also enable developers to use code coverage as a tool while refactoring, helping ensure new and improved code replaces the previous code.⁶¹

• Stop copy/pasting code; send several small code reviews, not one big one
  Two common habits that contribute to worse code quality are duplicating code⁶² and submitting large changes for review.⁶³ These changes make code difficult to read, test, review, and understand, which hides bugs and makes them difficult to find and fix after they’ve shipped. Helping people write testable code also helps people break these costly bad habits.

• Tests should be designed to fail: naming and organization can clarify intent and cause of failure; use Arrange-Act-Assert (a.k.a. Given-When-Then)
  The goal of testing isn’t to make sure tests always pass no matter what. The goal is to write tests that let us know, reliably and accurately, when our expectations of the code’s behavior differ from reality. Therefore, we should apply as much care to the design, naming, organization of our tests as we do to our production code.⁶⁴

  Merely showing people the immediately graspable Arrange-Act-Assert (or Given-When-Then) pattern can be a profound revelation that changes their perspective forever.

• Interfaces/seams enable composition, dependency breaking w/ test doubles
  Of course, many of us start out in legacy code bases with few tests, if any.⁶⁵ So we also need to teach how to make safe changes to existing code that enable us to begin improving code quality and adding tests. Michael Feathers’s Working Effectively with Legacy Code is the seminal tome on this subject, showing how to gently break dependencies to introduce seams. “Seams” are points at which we introduce abstract interfaces that enable test doubles to stand in for our dependencies, making tests faster and more reliable.⁶⁶

Speaking of interfaces, Scott Meyers, of Effective C++ fame, gave perhaps the best design advice of all for writing testable, maintainable, understandable code in general:

“Make interfaces easy to use correctly and hard to use incorrectly.”

—Scott Meyers, The Most Important Design Guideline?

To propose a slight update to make it more concrete:

“Make interfaces easy to use correctly and hard to use incorrectly—like an electrical outlet.”

—With apologies to Scott Meyers, The Most Important Design Guideline?

Of course, it’s not impossible to misuse an electrical outlet, but it’s a common, wildly successful example that people use correctly most of the time.⁶⁷ Making software that easy to use or change correctly and as hard to do so incorrectly may not always be possible—but we can always try.

Team/Organizational Alignment

Get everyone speaking the same language

Living up to that standard is a lot easier when the people you work with also consider it a priority.⁶⁸ Here’s how to create the cultural space necessary for people to apply successfully the new insights, skills, and language we’ve discussed.

• Internal media, roadmap programs, presentations, advocates
  Use internal media like blogs and newsletters to start a conversation around software quality and to start developing a common language. Roadmap programs create a framework for that conversation by outlining specific improvements teams can adopt. Team and organizational presentations can rely on the quality language and roadmap to inspire an audience and make the concepts more memorable. Software quality advocates can then use all these mechanisms to drive progress.

• Understanding between devs, QA, project managers/owners, executives
  Articulate how everyone plays a role in improving software quality, and get them all communicating with one another! An executive sponsor or project manager may not need to understand the fine details of dependency injection and test doubles. However, if they understand the Test Pyramid, they can hold developers and QA accountable for improving quality by implementing a balanced, reliable, efficient testing strategy.

• Focus and simplify! (Don’t swallow the elephant—leave some for later!)
  This is a lot of work that will take a long time. Rather than getting overwhelmed or spreading oneself too thin trying to swallow the entire elephant, focus and simplify by delivering one piece at a time.

• Every earlier success lays a foundation and creates space for future effort
  Delivering the first piece creates more space to deliver the second piece, then the third, and so on.

• Be agile—make plans, but recalibrate often
  Of course, this process need not be strictly linear. It’s important to be clear about priorities and delivering pieces over time, but make adjustments as everyone gains experience and the conversation unfolds.⁶⁹

• Absorb influences like a musician/band—then create your own voice/style
  Ultimately the process is a lot like helping one another grow as a musicians. It’s not about everyone doing exactly as they’re told. Everyone should be absorbing ideas, trying them out, gaining experience with them, and ultimately making them their own, part of their individual style. Then everyone can share what they’ve learned and discovered, enriching all of us further by adding their own voice to the ongoing conversation.

Roadmap Programs

Guidelines, language, conversation framework, examples

Let’s examine the value of roadmap programs more closely. They can provide the language and conversational framework for the entire program, as well as other powerful features.

• Define beginning, middle, and end—break mental barrier of where to start
  One of the most important features a roadmap provides is helping teams focus on getting started. It can help overcome the mental barrier of not knowing where to begin by helping to visualize the beginning, middle, and end of the journey. This helps break “analysis paralysis” by narrowing the options at each stage, and providing a rough order in which to implement them.

• Align dev, QA, project management, management, executives
  A roadmap can help produce alignment across the various stakeholders in a project, making clear what will be done, by who, and in what order. Shared language and collective visibility encourage common understanding, open communication, and accountability.

• Recommend common solutions, but don’t force a prescription
  A good roadmap won’t force specific solutions on every team, but will provide clear guidelines and concrete recommendations. Teams are free to find their own way to satisfy roadmap requirements, but most teams stand to benefit from recommendations based on others’ experience.

• Give space for conversation and experience to shape the way
  The point of a roadmap isn’t to guarantee a certain outcome, or to constrain variations or growth. It’s to help teams communicate their intentions, coordinate their efforts, and adjust as necessary based on what they learn together throughout the process.

• Provide a framework to make effort and results visible, including Vital Signs
  A roadmap helps teams focus, align, communicate, learn, and accomplish shared goals by making software quality work and its impact visible. It helps people talk about quality by giving it a local habitation and a name. Then it provides guidelines and recommendations on implementing Vital Signs that make quality efforts and outcomes as tangible as developing and shipping features.

• Can help other teams learn by example and follow the same path
  Finally, a roadmap that makes software quality work and its results visible to one team can also make it visible to others. This visibility can inspire other teams to follow the same roadmap and learn from one another’s example. Once a critical mass of teams adopts a common roadmap, though the details may differ from team to team, the broader organizational culture evolves.

To continue the musical metaphor, roadmaps should act mainly as lead sheets that outline a tune’s melody and chords, not as note for note transcriptions. They provide a structure for exploring a creative space in harmony with other players, but leave a lot of room for interpretation and creativity. At the same time, studying transcriptions and recordings to learn the details of what worked for others is important to developing one’s own creativity.

Quality Work and Results Visibility

Storytelling is essential to spreading language, leading change.

Finally, great storytelling is essential to providing meaningful insight into quality outcomes and the work necessary to achieve them.⁷⁰ This can happen throughout the process, but sharing outcomes, methods, and lessons learned is critical to driving adoption of improved practices and making them stick.

• Media, roadmaps, presentations, events
  In fact, good stories can drive alignment via the same media we discussed earlier. Organizing a special event every so often can generate a critical mass of focus and energy towards sharing stories from across the company. Such events can raise the company wide software quality improvement effort to a new plateau. They also help prove that common principles and practices apply across projects, no matter the tech stack or domain, refuting the Snowflake Fallacy.

• Make a strong point with a strong narrative arc
  The key to telling a good story is adhering a strong narrative arc.⁷¹ Here are three essential elements:

  • Show the results up front—share your Vital Signs!
    First, don’t bury the lede!⁷² We’re not trying to hook people on solving a mystery, we’re trying to hook people on the value of what we’re about to share. This holds whether you’ve already achieved compelling outcomes or if you’re still in the middle of the story and haven’t yet achieved your goals. In the latter case, you can still paint a compelling picture of what you’re trying to achieve, and why.

    Either way, having meaningful Vital Signs in place can make telling this part of the story relatively straightforward.

  • Describe the work done to achieve them, and why
    Next, tell them what you had to do (or are trying to do now) to achieve these outcomes and what you learned while doing it. Don’t just give a laundry list of details, however.

    • Practices need principles! The mindset is more portable than the details.
      Practices need principles.⁷³ Help people understand why you applied specific practices—show how they demonstrate the mindset⁷⁴ that’s ultimately necessary to improve software quality. Technical details can be useful to make the principles concrete, but they’re ultimately of secondary importance to having the right mindset regardless of the technology.

  • Make a call to action to apply the information
    Finally, give people something to do with all the information you just shared. Tell them how they can follow up with you or others, via email or Slack or whatever. Provide links to documentation or other resources where they can learn more about how to apply the same tools and methods on their own.

Building a Software Quality Culture

Cultivating resources to support buy-in

Here we can see how different resources serve to fulfill one or more of the essential needs for organizational change. There’s no specific order in which to build up these resources—it’s up to you to decide where to focus at each stage in your journey.

Mapping the Testing Grouplet and Quality Culture Initiative activities onto this table reveals how the same basic resources apply across vastly different company cultures.

Google Testing Grouplet

2005-2010

At Google, we provided introductory unit testing training and Codelab sessions to new employees, or “Nooglers.” We made extensive use of the internal wiki, and of course Testing on the Toilet was our breakthrough documentation hit. TotT helped people to participate in the Test Certified program, which was based on the experience of the Google Web Server team. Then we scaled up our efforts by building the Test Mercenaries team and hosting four companywide Fixits over the years.

Apple Quality Culture Initiative

2018-present

At Apple, we knew that posting flyers in Apple Park bathrooms wouldn’t fly, but our extensive Training curriculum was wildly successful. We also made extensive use of our internal Confluence wiki, maintained a Quality Blog, and had a ton of fun producing our own internal podcast. Quality Quest was directly inspired by Test Certified, but adapted by the QCI community to better serve Apple’s needs.⁷⁵ We promoted our resources via dozens of QCI Roadshow presentations for specific teams and groups, as well as a few official, high visibility internal presentations. We recruited QCI Ambassadors from different organizations to help translate general QCI resources and principles to fit the needs of specific orgs. Finally, we organized a QCI Summit promote software quality stories from across the company, demonstrating how the Quality Mindset applies regardless of domain.

This comparison raises an important point that I’ve made in response to a common question:

“What are the [important] differences between companies?”

“What are the differences between companies?” with the assumption or implication that the differences are of key importance. Reflecting upon this just before leaving Apple, I realized…

The superficial details may differ…

…of course there are obvious differences. Google’s internal culture was much more open by default, and people back in the day had twenty percent of their time to experiment internally. Apple’s internal culture isn’t quite as open, and people are held accountable to tight deadlines. Even so…

…but they’re more alike than different.

…the companies are more alike than they might first seem. Both are large organizations composed of the same stuff, namely humans striving for both individual and collective achievement. Much like code from different projects, at the molecular level, they’re more alike than they are different.

Over time, I’ve come to appreciate these similarities as being ultimately more important than the differences. The same essential issues emerge, and the same essential solutions apply, differing only in their surface appearances. So no matter what project you’re on, or what company you work for, everybody everywhere is dealing with the same core problems. Not even the biggest of companies is immune, or otherwise special or perfect.

On the bright side, no matter what project you’re on, or what company you work for…

Calls to Action

…you can do something about these problems.⁷⁶ The resources for dealing with them are just as available to you as they are to any other company, given the right mindset.

I’ve already listed several technical and organizational concepts comprising the substance of the changes you may need to make. These are important, but relatively straightforward to grasp. The harder part of the problem isn’t getting people to pay attention to these concepts and to understand them, but to act on them. The most important skill you’ll need to make that happen…

Learn about leadership!

The biggest challenge isn’t technical—it’s changing the mindset.

…is leadership.

Many technical people think they’re above this fuzzy “people stuff,” that driving improvements are all about data and logic and meritocracy. But look where that’s gotten us as an industry with regard to software quality and avoidable damage done to society. The purely technological mindset, and its lack of appreciation for “people stuff,” is why good practices like those we’ve discussed so often fail to spread.

Leadership is also an eminently transferable skill, highly valuable and useful no matter where you find yourself during your career. It has nothing to do with the title you happen to hold, but with how you conduct yourself to achieve alignment with others. It’s a vast topic that you can study for life, but here are a few of my favorite starting points at the moment:

• John “Add Value to People” Maxwell, The 5 Levels of Leadership
  John Maxwell’s personal mission is to “add value to people,” which is my favorite short definition of “leadership.” His book The 5 Levels of Leadership clearly illustrates how leadership transcends title and decision making authoritah, and what’s required to realize outstanding leadership potential.

• L. David Marquet, Leadership is Language
  David Marquet’s Leadership is Language is a playbook highlighting how to replace Industrial Revolution era communication habits with more empowering and productive modern habits. It illustrates how to stop merely telling people what to think and do, and how to encourage everyone to grow as decision makers and leaders. The results can literally make the difference between life and death.

• Liz Wiseman, Multipliers
  In a similar vein, Liz Wiseman’s Multipliers illustrates the distinction between “Diminishers” that drain intelligence and energy from organizations an “Multipliers” that amplify people’s capabilities. It catalogs and contrasts the behaviors of each, encouraging explicit awareness of our own tendencies, weaknesses, and strengths.

• Scott Miller and David Morey, The Leadership Campaign
  Miller and Morey’s The Leadership Campaign is a guide to the dynamics of stepping forward to lead a movement, modeled directly on political campaigning. The focus is on clarity of messaging and organization, building momentum, and taking advantage of opportunities.

• Michael Bungay Stanier, The Coaching Habit
  Michael Bungay Stanier’s The Coaching Habit is another book focused on language, specifically when it comes to leading individuals to think through their own challenges. It talks about how to “tame your Advice Monster” and focusing on helping people develop their own solutions and capabilities.

• Ken Blanchard, et. al., Leadership and the One-Minute Manager
  Leadership and the One-Minute Manager describes Blanchard’s “Situational Leadership” model. This model describes the need to adapt one’s leadership style to each individual over time based on their current capabilities.

Now remember from Crossing the Chasm that it falls on the Instigators to lead adoption of technologies and practices across an organization. As an Instigator, one of the lessons you’ll learn is that, sometimes…

Instigator Theory

It’s easier to change the rest of the world than your own team.

…it’s easier to change the rest of the world than your own team. I call this phenomenon Instigator Theory. However, as frustrating as this is, and as long as it takes to overcome, the basic outline of what you need to do is straightforward:

• Phase One: Connect with—or build—a community of fellow Instigators
  First, find your people. Put your feelers out. Invite folks to coffee, then start organizing informal meetings, and send out open invitations. See who really cares about software quality and is willing to show up to do something about it.

• Phase Two: Develop resources and do the work
  Next, employ your leadership skills and challenge the community to develop resources for helping individuals acquire new skills and teams align on quality practices.

  • Focus, simplify, and take your time in the beginning
    As mentioned before, there’s no need to rush, and take care not to spread yourselves too thin.

  • Every earlier success lays a foundation and creates space for future effort
    In time, every win you deliver will draw more people into the community, which then creates the capacity to deliver the next win.

• Phase Three: Share the results—make the work and its impact visible
  As your community begins delivering resources, and people put those resources to good use, make all that work and its impact visible early and often. Radiate the good work your doing and its results into the environment as much as you can. And as part of generating that radiation…

  • Recognize the value of one another’s contributions!
    …make sure to recognize the value that each member of the community adds to the effort! This is long term work that’s often thankless, as the focus for most of the organization remains on doing business as usual. Recognizing everyone’s value is a big part of keeping up morale and momentum, and makes that value visible to others in the organization as well.

Finally, I’d like to leave you with a concrete list of things you and your fellow Instigators can work to change.

Where we are	Where we’d like to go
Slow, unreliable, expensive processes	Fast, reliable, efficient feedback loops
Lots of duplicated, complex code	Well-factored, readable, testable code
Large, complex, monolithic code reviews	Small, digestible, easily reviewable changes
Large, complex, flaky test suites	Balanced, stable Test Pyramid-based suites
Expensive metrics people can’t act upon	Meaningful, useful Vital Signs taken seriously
Reinventing the wheel in wasteful silos	Sharing stories, language, useful examples
Complexity, risk, waste, and suffering	Clarity, confidence, efficiency, and delight
Testing (only) like a user would	Testing like a user’s life depends on it!

Let’s compare where many of us are today, without good quality practices in place, to where we’d like to get everyone to go.⁷⁷

• Ultimately, we want to replace painful, expensive processes with fast, reliable, and efficient feedback loops. We can start to do that by…

• …rejecting duplication and excess complexity in our code, and by writing readable, testable code instead.

• We can reject large, monolithic code reviews hiding lots of bugs and insist upon smaller, more reviewable changes.

• We can reduce the size, complexity, and unreliability of existing test suites by evolving towards a balanced, reliable suite based on Test Pyramid concepts.

• We can throw out meaningless metrics that are expensive and painful to collect in favor of meaningful, actionable, and relatively cheap Vital Signs.

• We can stop wasting resources on having teams wrestle with common quality problems separately, and help one another by sharing stories, language, and working examples.

• Improving our software quality can minimize complexity, risk, waste, and suffering, and the increased understanding it affords will yield clarity, confidence, efficiency, and delight.

• Once freed from the mental trap of testing only like a user would, we can begin testing like a user’s life depends on it.

Ultimately, creating great, high quality software shouldn’t require heroics, sacrifice, or endless pursuit of technologies or resources.

Ensuring that everyone can see what high software quality work looks like helps create the conditions for a positive Chain Reaction. As principles and practices spread, and priorities align around quality, we’ll see suffering subside as we keep making the right thing easier! Then maybe one day, it’ll be so easy everyone can’t help but to do the right thing by default.

Thank you!

Acknowledgments

I appreciates all the folks who’ve contributed to this presentation!

History

2023-01-12: Presented to Aetion at the invitation of John Turek, a former Google colleague.

2023-01-17: Presented to Microsoft at the invitation of Ono Vaticone, a former Apple colleague and Quality Culture Initiative member.

TODOs

Here are some items I’m still thinking about adding to the script, most likely as footnotes:

• Do Hard Things: The typical view of suffering as building “mental toughness” has the exact opposite effect.

• Connect Kurt Lewin’s model of change (from The Rainbow of Death) to “focus and simplify” and the “quality culture cycle” as well.

  • Frontiers in Group Dynamics: Concept, Method and Reality in Social Science; Social Equilibria and Social Change
  • Change Theory by Kurt Lewin
  • Lewin, Stage Model of Change Unfreezing Changing Refreezing AnimatedPart 5

• Mention “glue work” and the need to discuss, prioritize, and recognize leadership generally.

• Draw a parallel between market incentives for security as well as those for quality—which are often intertwined.

  • Perspectives on the SolarWinds Incident

• Add one or more references to the “Mousetrap” sequence in Hamlet:

  • Instructions to the players to “hold a mirror, as ‘twere, up to nature.”
  • The unsettling effect of seeing one’s own misdeed presented back to you in some form.

Footnotes