Mike Bland


Most recent posts

127 posts total. See Filtering and Navigation for tips on how to find the bits in which you're interested.

I'll be keynoting DevOpsDays Baltimore on March 8, I've made updates to my portfolio page, and I've some updates on go-script-bash v1.4.0 and more.

- Alexandria
Tags: Bash, dev tools, go script, Google, programming, Test Mercenaries, testing, Testing Grouplet, Testing on the Toilet
Discuss: Discuss "DevOpsDays Baltimore and other announcements" on Google+

I love deadlines. I love the whooshing sound they make as they fly by.—Douglas Adams, The Salmon of Doubt

Like Douglas Adams, my announced deadline for go-script-bash v1.4.0 is flew by last week. It’s not that I’ve stalled development on it at all; quite the contrary! While I had 634 tests as of a month ago, now there are 789 pieces of living proof that Bash scripts aren’t "too hard to test".

But let me get back to that. A few updates are in order.

DevOpsDays Baltimore Keynote

First and foremost, I’m a keynote speaker at DevOpsDays Baltimore on March 8. This time the talk will be called The Rainbow of Death, and will focus primarily on how all of the experiments and practices from the Testing Grouplet line up with the Three Ways of Devops (as outlined in The DevOps Handbook) in order to make the right thing the easy thing.

Portfolio updates

Next, after over two years of neglect, I’ve updated my portfolio page to include all the Open Source work I’ve been doing since late 2014. I forced myself to learn a little about flexboxen to make the layout a bit more palatable and responsive, and under the hood, used YAML files to make the collection and presentation of the data more uniform. By making the right thing easy for myself, hopefully I’ll do a better job of keeping it current (and readable!).

go-script-bash and…

Finally, no, I haven’t abandoned the HPKP with Let’s Encrypt series; but in the process of looking to migrate parts from certbot-webroot-setup back into go-script-bash, along with other minor enhancements, I had a stroke of inspiration that led to three outcomes:

Like I mention in that issue comment, assuming maintainership of such a popular Open Source project would be a new experience for me, but I really believe I’m up to it.

Outside of that, still in talks with a bunch of folks to get some independent contracting gigs rolling. Prospects are various and exciting, but I’m anxious to get to work.

I should have go-script-bash v1.4.0 out in a couple more days. Most of the features currently still posted (JSON logs, user prompts for configuration, etc.) will have to get bumped to a later version, but there’s too much good new stuff not to release sooner than later!

This blog now features a Let's Encrypt certificate with HPKP, and I hope to share insights and tooling that may be of use to others.

- Alexandria
Tags: Bash, go script, HPKP, Let's Encrypt, programming, security, ssl, technical
Discuss: Discuss "Switching to Let's Encrypt with HPKP, Part 0" on Google+

The SSL/TLS certificate for this blog was set to expire today. As you can see in your browser’s security information, I’ve installed a new certificate from Let’s Encrypt. However, my prior adoption of HTTP Public Key Pinning (HPKP) made implementing this less than straightforward—or, at least, it wasn’t straightforward at first.

This is the first in a series of posts in which I’ll describe how I did it, along with the insights I gained that informed my approach. And yes, my go-script-bash framework plays a central role, via my new certbot-webroot-setup tool.

Clouds form
Fronts meet
If you will it, it is no dream.
Next steps

Clouds form

As a matter of technical curiosity and good citizenship, I try to maintain an A+ rating on the Qualsys SSL Labs SSL Server Test. I first set up this blog for HTTPS back in mid-July 2014 when I got my first digital certificate from StartSSL. At the time, StartSSL was widely recommended as a Certificate Authority (CA) given its relatively low-cost and easy means of acquiring a certificate. Since my original certificate was issued with a SHA-1 signature, and hence not terribly secure and soon-to-be rejected by browsers, in February 2015 I paid for a new certificate signed with a SHA-256 signature.

Since then, a number of factors converged to produce a perfect storm that inspired me to spend three days from this past Friday, February 3 until yesterday, February 6 figuring out how to deploy a Let’s Encrypt certificate while implementing HPKP. The first factor was Let’s Encrypt’s emergence as a free CA with automated renewals. The standout features of Let’s Encrypt (in addition to the price) are that:

  • certificates are issued using an automated tool called certbot (formerly letsencrypt);
  • they may be automatically renewed using the same tool; and
  • Let’s Encrypt certificates are valid for only ninety days, limiting the scope of any potential compromise and encouraging automated renewals via certbot or another compatible tool.

Independently, there’ve been a number of mechanisms introduced into web servers and browsers to improve the security of HTTPS. I’ve implemented a number of them, but the three most relevant to this story are:

  • HTTP Strict Transport Security (HSTS), which signals to a user’s browser after visiting your site that it should only be accessed via HTTPS for some period of time, never plain HTTP;
  • HSTS Preloading, whereby you register your site such that the major browsers are hardcoded to only access your site via HTTPS, removing the possibility that users ever access your site via plain HTTP in the first place; and
  • HTTP Public Key Pinning (HPKP), which signals to a user’s browser that it should only accept certificates that have been signed with at least one of a specific set of digital signatures for some period of time.

I implemented HSTS and HSTS Preloading sometime in 2015; I first implemented HPKP on October 20, 2016, after reading Scott Helme’s Hardening your HTTP response headers blog post, which linked to his other post HPKP: HTTP Public Key Pinning. Confident I was doing the right thing the right way for the right reasons, I followed Scott’s instructions to generate my three public key/digital signature pins, and set my max-age parameter for 31,536,000 seconds, i.e. one year.

The implication of that step is that for anyone visiting my site between October 20, 2016, when I first implemented HPKP, until a few days ago when I temporarily disabled it, their browser will reject future connections to my site if its digital certificate isn’t signed with one of those three signatures, for one year since their last visit within that time frame.

Fronts meet

When my Google Calendar reminded me a couple weeks ago that my blog’s certificate was about to expire, I began entertaining the notion of switching to Let’s Encrypt. However, I remembered my year-long HPKP setting, and knew something wasn’t quite compatible there. While searching for information regarding how the two might work together, I came across Ivan Ristic’s Is HTTP Public Key Pinning Dead?, and immediately facepalmed. I was similarly dismayed to read Mathias Biilmann Christensen’s Be Afraid Of HTTP Public Key Pinning (HPKP).

Finally understanding how easy it was to get HPKP wrong, as a precautionary measure, I disabled HPKP for a few days last week. Sure, in Chrome it’s possible to clear the pins for a site by visiting chrome://net-internals/#hsts and deleting the domain; but in the case of this site, it’s HSTS Preloaded into the browser, so that wouldn’t work. And even if it did, how would most folks know to try?

Remember at this point that Let’s Encrypt certificates last for ninety days. As part of the default renewal mechanism, those certificates are issued with new digital signatures using keys that are automatically generated as well. So on the surface of things, after setting HPKP headers that last for a year using signatures that I generated myself, switching to Let’s Encrypt certificates seemed like it might risk locking people out of my site until a year after their last visit.

Consequently, this past Friday, February 3 I looked into returning to StartSSL to purchase a new SSL certificate. That’s when I learned about a rather unfortunate development.


I logged into the StartSSL console, which appears to have gotten an update since I last visited. Now it looks like a 2017 website, as opposed to something from the early 2000’s. Nice. However, I noticed a curious disclaimer at the bottom of the page, something about Chrome and Firefox not accepting their certificates for some reason.

Alarmed, I discovered the Mozilla Security Blog post Distrusting new WoSign and StartCom Certificates. In short, StartCom, the parent company for StartSSL, was acquired by WoSign, and then some funny stuff went down that led to Mozilla’s investigation of WoSign and decision to distrust certificates from WoSign and StartSSL after a certain date.

If you will it, it is no dream.

At this point I decided that I wouldn’t get another certificate from StartSSL. I also didn’t want to find another commercial CA and pay them a chunk of money if I could help it. Even so, I needed to live with the HPKP pins I’ve already published. So I began searching more earnestly for information about Let’s Encrypt and HPKP in tandem, just to see if anyone had found a way to make the two play along.

Turns out Scott Helme had another blog post, Getting started with Let’s Encrypt!, wherein he did just that. I also stumbled upon Jens Krämer’s blog post Let’s Encrypt SSL Certificates With HAProxy and Stable Keys.

Between these two posts and the Certbot user documentation, I was able to ascertain the steps necessary to update my blog’s setup to make it compatible with both Let’s Encrypt and HPKP. However, it took jumping back and forth between all three and experimenting repeatedly to come up with the right series of steps, as neither Scott’s nor Jens’s use cases were that closely matched with my own. All in all, it probably could’ve taken me three hours—but it took three days. Why?


Again, it comes down to my innate commitment to making the right thing the easy thing—especially for myself! Remember that I started writing the go-script-bash framework so that it would be easy to write modular, discoverable, easy-to-use, well-documented, portable, and testable Bash programs—as opposed to writing messy and arcane one-off scripts, or piling commands into a README or some other document that contains implicit assumptions or rapidly gets stale. The feedback loop from adding solid new features to the framework itself that in turn helped me develop further, ever more powerful features has borne out the validity of this core principle.

Regarding Let’s Encrypt and HPKP, helpful as the aforementioned documents were, I was concerned that if I only learned how to mutter the incantation well enough to get the job done now, I might find myself in a messy bind in three months when the renewal comes due. So I set out to capture the steps necessary from the start, to make them far more repeatable, as well as obvious to my future self.

Hence my new certbot-webroot-setup tool, developed using the go-script-bash framework. Admittedly, it’s currently very much a product of the Exploration phase, so it’s still a bit rough to look at and has no automated tests. That said, the framework definitely made it a joy to develop and experiment with, and helped me write a bundle of scripts that I can easily add documentation, tests, and other features to over time. In fact, I’ve already filed issues for improvements to certbot-webroot-setup (#1-#5), and the experience has also inspired a series of issues for improvements to go-script-bash (#143-#148).

Next steps

I’ll continue adding posts to this series to share the details of what I learned throughout the process of configuring Let’s Encrypt and HPKP for my system, and how those insights were codified in certbot-webroot-setup. In tandem, I’ll make updates to the code and documentation, as well as add tests now that certbot-webroot-setup is moving into the Settlement phase.

Those three days of investment in trying to do the right thing the right way for the right reasons will certainly pay themselves back by making the right thing easy for me from here on out. If others can benefit too, even better!

Staying busy as a coping mechanism has given way to frustration and outrage that I'm struggling to put to good use.

- Alexandria
Tags: go script, personal, philosophy, politics
Discuss: Discuss "So now what?" on Google+

This post is a bit of a blur between the personal, political, and technical. It may have little something for everyone, or prove repellent to everyone. So be it.

The Announcement
The Horror
The Insight
The Next First Step

The Announcement

Though I’ve kept quite busy lately, you may’ve noticed I haven’t announced that I’m working for anyone yet. That’s because after nearly eleven months off, and after exploring several avenues and entertaining a few offers, I’ve finally decided to try going into business for myself as an independent contractor. I haven’t really figured out how to do it yet, per se, but as with anything and everything, sometimes the first step to figuring something out is to flail and fail as early and often as possible.

In terms of my mission and vision, it really hasn’t changed at all: Making the right thing the easy thing. Whereas others focus on bringing the next big product to market, I seem relentlessly focused on making it easy to build that product the right way and keep it (and the team producing it) healthy in the long run. My ./go script framework is part of that vision, my Slack-to-GitHub Issues plugin is part of it, as are many of the other little projects I’ve spun up over the past couple years. I’m still working to stitch them all into a lightweight package of tools and practices to make it easier for individuals and organizations to do the right thing by default—even if no one’s paying me to do it at the moment.

Speaking to the very last point above, and perhaps most importantly, my attitude is a big part of my value proposition. I believe organizations should serve the people, not the other way around—because the organization is all the people, not a privileged, self-important subset. I believe in eliminating friction and pain, instead of enduring it until it’s codified as an institutional rite of passage. And although my specialty may be automated testing and sound coding practices, my real passion and focus is fixing whatever is wrong with an organization, and doing whatever it takes, however long it takes.

I’m in talks with a few folks now to do some contract gigs, but nothing yet is firmly booked. So if anyone reading has any interesting opportunities to share, I’m all ears.

And I’m not limiting myself to tech industry opportunities. Like many other technical folks I know, I’m growing ever eager to use my skills, vision, energy, and time to resist and rectify the damage to the United States being inflicted upon it by its own new federal administration.

The Horror

Honestly I’ve never been more shocked and upset by my own country’s actions, including the appointment of unqualified agency heads, many of whom actively seek to undermine the mission of the agencies they’ve been appointed to lead–if they’re even aware of what that mission is–as well as the incredibly discriminatory and damaging travel ban targeting people from nations that haven’t even harmed us, while conveniently omitting those that have. Our election appears to have been deeply influenced by a not-entirely-friendly state, and the response from both the administration and the electorate that voted it into office has been alarmingly muted. Most frighteningly of all, we now live in a universe where the person holding the highest office in the country, one of the highest in human history, can’t hold back from Twitter feuds and whose staff are actively undermining faith in logic, reason, and the media by asserting their own “alternative facts”.

In a way, the furious intensity with which I’ve been developing my ./go script framework isn’t entirely attributable to my coding addiction. It’s been a coping mechanism, an escape. While hopefully it’ll prove a relatively productive method of escape, as far as such things go, ultimately it’s been a bit of a security blanket, a means to shut out the horror and the noise that portends the end of the American experiment in democratic governance.

I feel like Peter from Office Space when he describes every day since he started working as the worst day of his life, replacing “since I started working” with “practically every day since the election, and especially since the inauguration”. And that’s me speaking as a white guy, a full citizen, someone who isn’t even in the crosshairs of this administration’s narrow-minded, hare-brained, alt-right (more like all-wrong) protectionist agenda.

That said, as a white guy, a full citizen, who is deeply appalled by the actions of his elected and appointed leaders, what can I do to resolve this? I’m kind of a borderline introvert who can’t stand being in crowds, so marching in the streets isn’t exactly my thing. And while I’m very sympathetic to all the demonstrators across the country and around the globe, I’m not entirely sanguine that protests will make much of a difference. By that same token, phone call and letter writing campaigns strike me as not entirely worthwhile either–not saying they aren’t, and I could be totally wrong about that, but they don’t feel like the right kind of activities to me, or at least not for me.

So what can I do to help? What is it, I would say, I could do here? (Can’t believe IMDB doesn’t have that Office Space quote listed.)

The Insight

As with my career, wherein I realized that marching up to people, wagging my finger in their faces, and voicing as strongly as possible that I found their lack of tests disturbing wasn’t the true path to solving the problem, I don’t believe that mere vocalization of opposition will solve the underlying problems that led to this situation to begin with. I believe I’m beginning to see parallel threads that I, if no one else, might use to understand the forces at play. From that understanding I hope to arrive at meaningful action that might produce a lasting impact.

These threads have to do with communication and empathy, which I believe are inextricable from one another. It’s been well-documented that many of the votes that flipped from Democratic to Republican in this last election were from people who felt like the statistically-sound assertions of an improved economy were at odds with their own day-to-day reality. By the same token, while the dense coastal cities have erupted in protest over the travel ban, for example, many of the heartland voters are generally supportive—despite the fact that most of them don’t live anywhere near where any previous or any imagined future attacks might occur.

What I’m seeing is that while everyone in the United States has a stake in the policies and the laws of the land, different segments experience starkly different realities that foment hostility and resentment instead of balanced discussion. The problem isn’t access to information; there’s way more than enough of that these days, thanks to technology. And the problem isn’t just knowing what to make of it and what to do with it, though that is a challenge we need to address. To put it in terms like I use in my talks, the people actually have plenty of power to do the right thing, but they don’t yet really know what the right thing to do is, or how that power can help them do it.

To me, the biggest part of the problem is the fear and resentment that springs up whenever people on either side are confronted with ideas and perspectives that run counter to their own lived experiences–especially when those strange ideas and perspectives are framed as though they’re the works of the devil.

Granted, the conservative side has been far, far guiltier of promoting misinformation and mistrust in our society. Far guiltier, by a long shot; after all, they actively cultivated the environment that produced the nightmare we’re living today. But as liberals, we must recognize we’re susceptible to the same human frailties that led the conservatives astray, do our best to avoid falling into the same traps, and work with them to pull ourselves out of this mess, together.

The Next First Step

How to do that is the tricky part that I personally haven’t figured out yet, but as I’ve always thrived more as a team player than a solo act (my independent contracting venture notwithstanding–or maybe withstanding, we’ll see), I’m putting my thoughts and intentions out there in the hope it will inspire some meaningful engagement with like-minded seekers, fellow Instigators who are also struggling to find a way to contribute their skills and energy to a worthwhile solution.

If you’re keen, let’s figure out how to flail and fail together. Shoot me an email, even encrypt it using my PGP key if you’re paranoid. Maybe, like ye olde Testing Grouplet, we’ll stumble upon a few crazy ideas that will converge to produce a lasting difference. Maybe one or more of us’ll even end up taking some kind of public office or other one day. I might even be keen to give it a shot eventually. In terms of qualifications, at least it’s an established (as opposed to alternative) fact that I’m not one to get sucked into Twitbook feuds. That alone should qualify me for the office of President of the United States heads and shoulders above the incumbent.