Mike Bland

Like I mentioned in my previous post, I’ve been consumed with a passion project as of late. Not go-script-bash this time, though I did have to investigate why my coverage builds started failing due to some sort of deadlock—which led me to submit SimonKagstrom/kcov#211 to fix it, naturally. (Thanks to Simon Kagstrom for providing such a useful tool, and for merging my fix so quickly!)

Mostly, though, I’ve been working on a new project I call Custom Links, which is an application that for creating and accessing custom URLs. It allows authenticated users to create short URLs with meaningful names that redirect to longer URLs, or to easily create canonical URLs that you can redirect to different target URLs over time.

• A wheel by any other name
• So much for serverless
• The hard part
• The end(-ish) result
• Future development
• Footnotes

A wheel by any other name

Why did I write this? Doesn’t something similar already exist?

First, I got the idea after the Schibsted Eng Prod team decided—before I arrived and independent of my direct involvement—to publish their own Schibsted Testing on the Toilet series. Clearly you can’t expect “users” to “click” on URLs from a paper flyer; and while QR codes are possible now (which we did use), there’s still a lot of value in using short, memorable URLs like go/tott, as we used so often on TotT episodes at Google.

So I started looking around for an open source solution—and couldn’t find one! The closest I found was YOURLS, and at least when I tried setting it up in mid-April, I couldn’t figure out how to get it to create actual custom URLs instead of automatically-generated short URLs. (To be fair, the current home page does make it look like it can do that, but I haven’t deeply investigated it since.) Plus, it requires setting up PHP and MySQL, and uses its own user account system.

In the meanwhile, Luis asked Kenneth to set up a bit.ly account to create custom links. The URLs all begin with some form of bit.ly/, which isn’t as nice as go, but it was enough to start publishing TotT episodes.

However, as Kenneth found out, Bitly apparently won’t let you change the target of a custom short link after you create it. Hence, the use case of providing a canonical URL that can be updated as needed isn’t met using this solution.

So much for serverless

When I couldn’t find any other open source alternative, I figured writing such a server shouldn’t be too hard, and set out to do so. At first, sensing an opportunity to put what I learned from Serverless Single Page Apps into practice, I spent a couple days trying to think through how to implement it in a serverless fashion. However, trying to design a general-purpose application raised one gigantic problem I couldn’t devise an answer for: redirects. Yes, people have used S3 buckets to set up their own redirect “databases”, but for seemingly custom internal services that require some degree of specialized maintenance, rather than being mostly self-serve.

Confronted with this, I fell back on Ken Thompson’s old bit of programming wisdom: When in doubt, use brute force. Being a backend developer, and one who generally avoids SQL databases, I decided to bang out the backend using Node.js, using Redis as a backing store, and using the Passport authentication framework to support user authentication via Google OAuth 2.0. Learning the Redis API was easy, and getting things set up with Passport took a bit of archaeology, but all in all the backend largely came together over a weekend or so—tested six ways from Sunday, of course.

The hard part

“Users are a frontend problem.”—Written on the whiteboard of former Google colleague Matthew Simmons

While I had a functional server that would efficiently provide short link redirections, and I could easily update the Redis database directly, I accepted the reality that most users, even technical ones, expect a more user-friendly interface. Also, since I’ve done very little frontend development—and even less frontend testing—I decided I’d use this project to teach myself a few things about browser-side JavaScript and browser-driven end-to-end testing.

The first thing to decide upon was a framework. Influenced by Serverless Single Page Apps, I decided not to use one, opting to embed template elements in my single index.html page, and to use JavaScript to instantiate and update them. On top of that, I went one step beyond the book and eschewed JQuery in favor of Vanilla JS.¹ I’m not sure I’d do this again for every project—as much as I hate to accept it, React and GraphQL appear to be the new hotness²—but I think it was a great exercise to go through before resorting to higher-level abstractions in the future.

Of course, me being me, I made setting up my environment to “make the right thing the easy thing” (i.e. automated testing) my top priority before going full-bore into development. Since Schibsted was using bit.ly, and this was a personal project, I could afford to force myself to learn a lot about JavaScript testing tools and frameworks rather than rush to deliver a product.

And man, did I learn. Rather than enumerate the details here, check out the Custom Links listing on my portfolio page if so inclined. The punch line, however, is thanks to creative use of go-script-bash, I managed to make it really easy to run browser tests that automatically run after a file save, to run them on the command line via Phantom JS, to collect coverage reports that can automatically open in the browser or get shipped to Coveralls—and, finally, after all of these years, to run Selenium-WebDriver tests.

The end(-ish) result

I wound up setting up these environment pieces and debugging tooling issues—such as the lack of nonblocking connect crashing PhantomJS on the Windows Subsystem for Linux (also enumerated in the portfolio listing)—all the way until my second trip to Europe. After that, I didn’t write a line of code for the entire rest of June and for the first couple weeks of July.

Finally, after fixing the aforementioned kcov issue, I dove back into Custom Links on July 21, and finally completed what I hope is a reasonably functional user interface this past Tuesday, August 8. In terms of tests (all times for my 2015 MacBook Pro with a 2.9GHz Intel Core i5 and 8GB RAM):

• The server test suite (./go test server) contains 143 test cases and runs in two seconds.
• The browser test suite (./go test browser) contains 123 test cases and runs in about 700 milliseconds.
• Running the browser test suite across Chrome, Firefox, Safari, and PhantomJS simultaneously using Karma (karma start) takes about 4.5 seconds.
• The end-to-end test suite (./go test end-to-end) contains nine test cases and runs in about twelve seconds.
• Running all of the above at once (./go test) takes about 30 seconds.

I updated the README with basic installation and configuration instructions this past Wednesday, August 9, and now finally consider the basic product ready to try out by other humans.

Future development

There’s lots of room for improvement, of course. For starters, I may try switching from PhantomJS to Headless Chrome. I could implement other auth providers, and perhaps add a landing page to select which one to use. The code could probably be slightly reorganized, with function, class, and module comments. (I’ve leaned heavily on descriptive function, object, and test case names up until now). Perhaps I could clean up and test my scripts. Certain components and test helpers could be extracted into npm packages. The README can almost certainly be improved.

And of course, of course, I should look into Docker-izing it.

Still, even now, I think it could be useful to folks. So if you’re inclined to give it a try, I’d love to help you get set up if needed, and get your feedback on how it could be improved!

Even if no one uses it, at any rate, it really challenged me to shore up my skills in an area that I’d neglected for the better part of my career. That experience alone made the whole project worthwhile!

Footnotes