Mike Bland

If you received this post via email without ever subscribing to this blog, please follow the Unsubscribe link at the very bottom. I deeply apologize for the bother, and you won’t hear from me again.

That said, if you care to keep reading, you’ll see I’ve put more safeguards in place to drastically reduce the possibility of future bogus subscriptions.

For everyone else who’s deliberately subscribed to my email updates, or are reading this by other means, I’ll explain what’s happening. It’s related to why I’ve been silent the past few months.

In one word: EListMan.

First, a shoutout!

Before I get into all of that, I’d like to point folks towards a blog post by my erstwhile Google and Apple colleague Simon Stewart. In the ingeniously titled Your Roots Are Showing, he gives some interesting background on why some aspects of Bazel are the way they are. He also generously referenced my Tools blog post from ages ago, which I very much appreciates!

Simon’s also the reason why I now explicitly mention fast feedback loops so frequently, such as in Making Software Quality Visible. He emphasized their fundamental importance during our regular video chats at Apple, inspiring me to embed the concept firmly into the Quality Culture Initiative program. Check out his excellent Dopamine Driven Development presentation to become similarly inspired.

Where I’ve been

I’ve been heads down since March 16 working on a passion project for this blog called EListMan. It’s a mailing list system providing address validation and unsubscribe URIs. It’s written in Go using a serverless architecture built on Amazon Web Services.

My vanity prevented me from wanting to post to my blog again until the system was ready. If you’re already subscribed to my emails, it should be immediately apparent why, when you compare this post’s email to emails in the past.

My obsessive compulsive hyperfocus while in code mode prevented me from thinking about posting until the system was done. I allowed myself very little bandwidth to do anything else.

Why

I jumped into this rabbit hole and ended up chasing a white whale through a maze of twisty little passages, all alike for several reasons.

I’ve had a subscription page available for many years. It previously contained a <form> to submit email subscription requests to an email announcement system run by my previous web hosting provider. I used to send plain text versions of my blog posts to this system.

However, I invested substantially in improving the overall look and feel of the blog earlier this year. This upgrade was inspired by writing and publishing Making Software Quality Visible. (Which was inspired by my friend and colleague Ono Vaticone inviting me to speak to his team at Microsoft after I left Apple.) While I don’t expect anyone to read more than a fraction of that entire page, I wanted to make reading it more pleasant nonetheless.

So then, after writing the Making Software Quality Visible announcement, I decided maybe it was time to give HTML emails a try as well.

I tried copying and pasting that post’s HTML into the hosting provider’s system to see what happened—and it didn’t go well. I waited hours for it to show up before realizing it went to my spam folder. It also looked horrible—not just formatting wise, but the encoding also wasn’t right, which made apostrophes and other characters look like garbage (’, etc.).

First, I started learning about how to properly format HTML emails, which was a rabbit hole in itself. If you thought cross browser compatibility was a challenge, check out what cross email client compatibility requires. (Or don’t—sometimes, ignorance is bliss.) It took me a week or two to update my script that generates the text, and now HTML, versions of my blog posts. That seemed like a long time at the time, but I did eventually generate HTML that looked good when I emailed it to myself.

Then I figured, what if I wrote my own email delivery system to replace the other one? I could guarantee proper MIME encoding, as well as sharpen my chops and pick up some new experience with Go, AWS, and serverless system architecture. I figured I’d give Visual Studio Code a solid try as well (with Vim keybindings, of course).

Sure, I could use a commercial product and be done with it instead…but how hard could it be?

This way lies madness…

Well, while it wasn’t impossible, it definitely took a lot of time and dedication to get right. Those commercial systems exist for a reason. And working so hard on it by myself for so long definitely started to take a toll on my sanity. I began to wonder whether I was nobly committed to seeing the project through, or falling prey to the sunk cost fallacy.

Either way, as of today, EListMan is officially in production. I plan to improve the README and code documentation, and to continue tweaking the system and command line interface, but that’s all polish. The system is deployed, fully operational, and working as intended.

…though not as much as there might’ve been

EListMan is also, of course, completely covered by small tests, plus a few medium/contract tests. I created an end-to-end smoke test very early on in the project to ensure that the API Gateway and Lambda were deployed successfully and reachable. (I later heard this referred to as a Walking Skeleton methodology; the smoke test exposed minor logical gaps for which I then added unit tests.) I also set up the mbland/elistman CI/CD pipeline early on using GitHub Actions, thanks to the instructions from The Complete AWS SAM Workshop.

The badges at the top of the github.com/mbland/elistman README attest to all of this—or help “make the software quality visible,” as one might say. Without those tests, and the fast feedback loops they enabled, I might’ve fallen completely off the mental and emotional cliff and into the abyss. They enabled me to make steady forward progress; even though that progress took longer than expected, at least it wasn’t fraught with bug induced despair.

I’d still like to try adding properly automated large/end-to-end tests at some point; however, my manual E2E testing isn’t burdensome at the moment. I’ve also taken pains to propagate and log decent error messages throughout, as is the Go idiom.¹

I’m not saying the code or the tests are perfect, of course. I am saying that both are healthy enough to make it relatively painless to keep improving them both over time. The commit history shows that I’ve already made some substantial improvements, and other significant changes, fairly quickly. Most of the time I did this without breaking anything—though when the mbland/elistman CI/CD pipeline did break, I rolled back or fixed the problem immediately.

So what was that about unsubscribing?

Oh yeah, so about that…

If you visit the subscription page, you’ll notice the actual email subscription <form> is now tucked away behind a CAPTCHA puzzle. But I didn’t set that up until after some opportunistic spam bot(s) pumped dozens of addresses into the original form for a few days.

So if you’re one of the ones who was signed up by a spam bot, and you’re still reading this, I am so, so sorry. I totally understand if you unsubscribe—I even understand if you register a spam complaint instead. That said, I hope you’ll have a little mercy and just unsubscribe, now that you know I’m legit.

Plus, you’ll see below what I’ve done to prevent ever putting anyone in the same position again. (Practically never, at least, modulo actual humans who think signing others up to receive spam is a worthy use of their lives. That, or modulo the rise of capable AI being put up to the task—or of it developing an adolescent, antisocial sense of humor.)

What happened

Large portions of this explanation and the following section are adapted from the EListMan README.

The EListMan system tries to validate email addresses through its own up front analysis and by sending validation links to subscribers. However, opportunistic spam bots can still—and will—submit many valid email addresses without either the knowledge or consent of the actual owner.

Fortunately, the validation link mechanism prevents most bogus subscriptions, and DynamoDB’s Time To Live feature cleans them from the database automatically. A bounce or complaint also notifies the EListMan Lambda to remove the address and add it to the account-level suppression list. The suppression list ensures the system won’t send to that address again, even if someone attempts to resubmit it.

This means most bogus subscriptions will not pollute the verified subscriber list, and such recipients will not receive further emails. However, generating these bogus subscriptions still consumes resources, and their verification emails can yield bounces and complaints that will harm your SES reputation metrics.

Consequently, I ended up learning the hard (naïve) way that I needed to add a CAPTCHA to the subscription page to prevent spam bot abuse.

I originally updated the <form> on that page to point to my production EListMan instance instead of the previous hosting provider’s system:

<form method="post" action="https://api.mike-bland.com/email/subscribe">
  <input name="email" type="email"
   placeholder="Please enter your email address."/>
  <button type="submit">Subscribe</button>
</form>

Shortly after, my production EListMan instance received dozens of bogus subscription requests a day—before I’d even announced it here. I noticed this activity while showing the system to Wolfgang Trumler a few days after publishing the updated <form>.

Clearly some opportunistic spam bot found my email <form> and started pumping addresses into it. I saw about five new addresses that actually passed verification when I first looked—I know which ones they are, and their legitimacy is dubious. However, the good news was none of the other dozens of addresses passed verification after that, and I could see the system working as intended. Unverified addresses were expiring out of DynamoDB, and bounces and verification fails were appearing in the suppression list.

I let this go for a few days, occasionally updating the address verifier with new domains and address patterns to reject. But the spam kept coming—and eventually, an actual complaint came in.

AWS WAF CAPTCHA FTW²

That first complaint finally prompted me to use the AWS Web Application Firewall to set up a CAPTCHA. I also felt silly for having told my friend David Singley the evening before that I hadn’t really thought I needed one.

The setup, configuration, and new subscription <form> is described in detail in the EListMan README. But in short, I removed the <form> from the raw HTML, and now build it programmatically using JavaScript—after the user solves the CAPTCHA. From the subscription page:

<div class="subscribe-captcha"><button>Show CAPTCHA puzzle</button></div>

From /scripts/subscribe-captcha.js:

"use strict";
document.addEventListener("DOMContentLoaded", () => {
  var container = document.querySelector(".subscribe-captcha")

  var showForm = () => {
    var f = document.createElement("form")
    f.className = "subscription"
    f.method = "post"
    f.action = [
      "https:", "",
      ["api", "mike-bland", "com"].join("."),
      "email", "subscribe",
    ].join("/")

    var i = document.createElement("input")
    i.type = "email"
    i.name = "email"
    i.className = "email"
    i.placeholder = "Please enter your email address."
    f.appendChild(i)

    var s = document.createElement("button")
    s.type = "submit"
    s.appendChild(document.createTextNode("Subscribe"))
    f.appendChild(s)

    container.parentNode.replaceChild(f, container)
    input.focus()
  }

  container.querySelector("button").addEventListener('click', () => {
    AwsWafCaptcha.renderCaptcha(container, {
      apiKey: "[...snip...]",
      onSuccess: showForm,
      dynamicWidth: true,
      skipTitle: true
    });
  })
})

Only building the <form> programmatically might’ve done the trick by itself. My suspicion is that the bots are lazily looking for <form> elements, not necessarily executing JavaScript to see what pops up. But it’s possible that they are, and after the sting of that first complaint, I didn’t want to take any chances.

After deploying the CAPTCHA, the number of bogus subscriptions instantly dropped to zero.

But I really hope I hadn’t inadvertently been generating subscription verification spam via the previous system for all those years….

More to come—and please subscribe!

Rather than cram the whole story of the system into this one post, I’ll try to spread it out over the coming weeks. It’ll give me more opportunities to actually use this thing I’ve worked so hard on, for so long, that nobody asked me to build.

At the same time, if you’re at all inclined, please visit the subscription page and sign up to receive my blog posts via email! (Or, as that page also explains, you can subscribe to the Atom feed, if you prefer. Or follow my LinkedIn profile.)

Just be warned that while many of my posts are technical in nature, this is still my personal blog. Sometimes I’ll just post whatever the hell I want.

Speaking of which, if you’ll ‘scuse me, I’m going to go play with my new PRS HDRX 20 that I recently got from Melodee Music. I’ve earned it.

Footnotes