Mike Bland

The SSL/TLS certificate for this blog was set to expire today. As you can see in your browser’s security information, I’ve installed a new certificate from Let’s Encrypt. However, my prior adoption of HTTP Public Key Pinning (HPKP) made implementing this less than straightforward—or, at least, it wasn’t straightforward at first.

This is the first in a series of posts in which I’ll describe how I did it, along with the insights I gained that informed my approach. And yes, my go-script-bash framework plays a central role, via my new certbot-webroot-setup tool.

• Clouds form
• Fronts meet
• Uh-oh
• If you will it, it is no dream.
• certbot-webroot-setup
• Next steps

Clouds form

As a matter of technical curiosity and good citizenship, I try to maintain an A+ rating on the Qualsys SSL Labs SSL Server Test. I first set up this blog for HTTPS back in mid-July 2014 when I got my first digital certificate from StartSSL. At the time, StartSSL was widely recommended as a Certificate Authority (CA) given its relatively low-cost and easy means of acquiring a certificate. Since my original certificate was issued with a SHA-1 signature, and hence not terribly secure and soon-to-be rejected by browsers, in February 2015 I paid for a new certificate signed with a SHA-256 signature.

Since then, a number of factors converged to produce a perfect storm that inspired me to spend three days from this past Friday, February 3 until yesterday, February 6 figuring out how to deploy a Let’s Encrypt certificate while implementing HPKP. The first factor was Let’s Encrypt’s emergence as a free CA with automated renewals. The standout features of Let’s Encrypt (in addition to the price) are that:

• certificates are issued using an automated tool called certbot (formerly letsencrypt);
• they may be automatically renewed using the same tool; and
• Let’s Encrypt certificates are valid for only ninety days, limiting the scope of any potential compromise and encouraging automated renewals via certbot or another compatible tool.

Independently, there’ve been a number of mechanisms introduced into web servers and browsers to improve the security of HTTPS. I’ve implemented a number of them, but the three most relevant to this story are:

• HTTP Strict Transport Security (HSTS), which signals to a user’s browser after visiting your site that it should only be accessed via HTTPS for some period of time, never plain HTTP;
• HSTS Preloading, whereby you register your site such that the major browsers are hardcoded to only access your site via HTTPS, removing the possibility that users ever access your site via plain HTTP in the first place; and
• HTTP Public Key Pinning (HPKP), which signals to a user’s browser that it should only accept certificates that have been signed with at least one of a specific set of digital signatures for some period of time.

I implemented HSTS and HSTS Preloading sometime in 2015; I first implemented HPKP on October 20, 2016, after reading Scott Helme’s Hardening your HTTP response headers blog post, which linked to his other post HPKP: HTTP Public Key Pinning. Confident I was doing the right thing the right way for the right reasons, I followed Scott’s instructions to generate my three public key/digital signature pins, and set my max-age parameter for 31,536,000 seconds, i.e. one year.

The implication of that step is that for anyone visiting my site between October 20, 2016, when I first implemented HPKP, until a few days ago when I temporarily disabled it, their browser will reject future connections to my site if its digital certificate isn’t signed with one of those three signatures, for one year since their last visit within that time frame.

Fronts meet

When my Google Calendar reminded me a couple weeks ago that my blog’s certificate was about to expire, I began entertaining the notion of switching to Let’s Encrypt. However, I remembered my year-long HPKP setting, and knew something wasn’t quite compatible there. While searching for information regarding how the two might work together, I came across Ivan Ristic’s Is HTTP Public Key Pinning Dead?, and immediately facepalmed. I was similarly dismayed to read Mathias Biilmann Christensen’s Be Afraid Of HTTP Public Key Pinning.

Finally understanding how easy it was to get HPKP wrong, as a precautionary measure, I disabled HPKP for a few days last week. Sure, in Chrome it’s possible to clear the pins for a site by visiting chrome://net-internals/#hsts and deleting the domain; but in the case of this site, it’s HSTS Preloaded into the browser, so that wouldn’t work. And even if it did, how would most folks know to try?

Remember at this point that Let’s Encrypt certificates last for ninety days. As part of the default renewal mechanism, those certificates are issued with new digital signatures using keys that are automatically generated as well. So on the surface of things, after setting HPKP headers that last for a year using signatures that I generated myself, switching to Let’s Encrypt certificates seemed like it might risk locking people out of my site until a year after their last visit.

Consequently, this past Friday, February 3 I looked into returning to StartSSL to purchase a new SSL certificate. That’s when I learned about a rather unfortunate development.

Uh-oh

I logged into the StartSSL console, which appears to have gotten an update since I last visited. Now it looks like a 2017 website, as opposed to something from the early 2000’s. Nice. However, I noticed a curious disclaimer at the bottom of the page, something about Chrome and Firefox not accepting their certificates for some reason.

Alarmed, I discovered the Mozilla Security Blog post Distrusting new WoSign and StartCom Certificates. In short, StartCom, the parent company for StartSSL, was acquired by WoSign, and then some funny stuff went down that led to Mozilla’s investigation of WoSign and decision to distrust certificates from WoSign and StartSSL after a certain date.

If you will it, it is no dream.

At this point I decided that I wouldn’t get another certificate from StartSSL. I also didn’t want to find another commercial CA and pay them a chunk of money if I could help it. Even so, I needed to live with the HPKP pins I’ve already published. So I began searching more earnestly for information about Let’s Encrypt and HPKP in tandem, just to see if anyone had found a way to make the two play along.

Turns out Scott Helme had another blog post, Getting started with Let’s Encrypt!, wherein he did just that. I also stumbled upon Jens Krämer’s blog post Let’s Encrypt SSL Certificates With HAProxy and Stable Keys.

Between these two posts and the Certbot user documentation, I was able to ascertain the steps necessary to update my blog’s setup to make it compatible with both Let’s Encrypt and HPKP. However, it took jumping back and forth between all three and experimenting repeatedly to come up with the right series of steps, as neither Scott’s nor Jens’s use cases were that closely matched with my own. All in all, it probably could’ve taken me three hours—but it took three days. Why?

certbot-webroot-setup

Again, it comes down to my innate commitment to making the right thing the easy thing—especially for myself! Remember that I started writing the go-script-bash framework so that it would be easy to write modular, discoverable, easy-to-use, well-documented, portable, and testable Bash programs—as opposed to writing messy and arcane one-off scripts, or piling commands into a README or some other document that contains implicit assumptions or rapidly gets stale. The feedback loop from adding solid new features to the framework itself that in turn helped me develop further, ever more powerful features has borne out the validity of this core principle.

Regarding Let’s Encrypt and HPKP, helpful as the aforementioned documents were, I was concerned that if I only learned how to mutter the incantation well enough to get the job done now, I might find myself in a messy bind in three months when the renewal comes due. So I set out to capture the steps necessary from the start, to make them far more repeatable, as well as obvious to my future self.

Hence my new certbot-webroot-setup tool, developed using the go-script-bash framework. Admittedly, it’s currently very much a product of the Exploration phase, so it’s still a bit rough to look at and has no automated tests. That said, the framework definitely made it a joy to develop and experiment with, and helped me write a bundle of scripts that I can easily add documentation, tests, and other features to over time. In fact, I’ve already filed issues for improvements to certbot-webroot-setup (#1-#5), and the experience has also inspired a series of issues for improvements to go-script-bash (#143-#148).

Next steps

I’ll continue adding posts to this series to share the details of what I learned throughout the process of configuring Let’s Encrypt and HPKP for my system, and how those insights were codified in certbot-webroot-setup. In tandem, I’ll make updates to the code and documentation, as well as add tests now that certbot-webroot-setup is moving into the Settlement phase.

Those three days of investment in trying to do the right thing the right way for the right reasons will certainly pay themselves back by making the right thing easy for me from here on out. If others can benefit too, even better!