Mike Bland

Kinda late notice, but better than never: I’ll be delivering "The Convergence of Wills" at the Surge 2016 conference this Friday, September 23 at 11am.

Also, it’s a day late, but R.I.P. Jimi!

I’m happy to announce v1.0.0 of my go-script-bash framework, available as Open Source software under the ISC License.

What’s a ./go script?
Is this related to the Go programming language?
Why write a framework?
Why Bash?
Will this work on Windows?
Why not use tool X instead?
Where can I run it?
How is it tested?
So what’s next?
Postscript

What’s a ./go script?

The ./go script idea came from Pete Hodgson’s blog posts In Praise of the ./go Script: Part I and Part II. To paraphrase Pete’s original idea, rather than dump project setup, development, testing, and installation/deployment commands into a README that tends to get stale, or rely on oral tradition to transmit project maintenance knowledge, automate these tasks by encapsulating them all inside a single script in the root directory of your project source tree, conventionally named "go". Then the interface to these tasks becomes something like ./go setup, ./go test, and ./go deploy. Not only would this script save time for people already familiar with the project, but it smooths the learning curve, prevents common mistakes, and lowers friction for new contributors. This is as desirable a state for Open Source projects as it is for internal ones.

Is this related to the Go programming language?

No. The ./go script convention in general and this framework in particular are completely unrelated to the Go programming language. In fact, the actual ./go script can be named anything. However, the go command from the Go language distribution encapsulates many common project functions in a similar fashion.

Why write a framework?

Of course, the danger is that this ./go script may become as unwieldy as the README it’s intended to replace, depending on the project’s complexity. Even if it’s heavily used and kept up-to-date, maintenance may become an intensive, frightening chore, especially if not covered by automated tests. Knowing what the script does, why it does it, and how to run it may become more and more challenging—resulting in the same friction, confusion, and fear the script was trying to avoid.

The ./go script framework makes it easy to provide a uniform and easy-to-use project maintenance interface that fits your project perfectly regardless of the mix of tools and languages, then it gets out of the way as fast as possible. The hope is that by making the right thing the easy thing, scripts using the framework will evolve and stay healthy along with the rest of your project sources, which makes everyone working with the code less frustrated and more productive all-around.

This framework accomplishes this by:

• encouraging modular, composable ./go commands implemented as individual scripts—in the language of your choice!
• providing a set of builtin utility commands and shell command aliases—see ./go help builtins and ./go help aliases
• supporting automatic tab-completion of commands and arguments through a lightweight API—see ./go help env and ./go help complete
• implementing a quick, flexible, robust, and convenient documentation system—document your script in the header, and help shows up automatically as ./go help my-command! See ./go help help.

Plus, its own tests serve as a model for testing command scripts of all shapes and sizes.

The inspiration for this model (and initial implementation hints) came from Sam Stephenson’s rbenv Ruby version manager.

Why Bash?

It’s the ultimate backstage pass! It’s the default shell for most mainstream UNIX-based operating systems, easily installed on other UNIX-based operating systems, and is readily available even on Windows.

Will this work on Windows?

Yes. It is an explicit goal to make it as easy to use the framework on Windows as possible. Since Git for Windows in particular ships with Bash as part of its environment, and Bash is available within Windows 10 as part of the Windows Subsystem for Linux (Ubuntu on Windows), it’s more likely than not that Bash is already available on a Windows developer’s system. It’s also available from the MSYS2 and Cygwin environments.

Why not use tool X instead?

Of course there are many common tools that may be used for managing project tasks. For example: Make, Rake, npm, Gulp, Grunt, Bazel, and the Go programming language’s go tool. There are certainly more powerful scripting languages: Perl, Python, Ruby, and even Node.js is a possibility. There are even more powerful shells, such as the Z-Shell and the fish shell.

The ./go script framework isn’t intended to replace all those other tools and languages, but to make it easier to use each of them for what they’re good for. It makes it easier to write good, testable, maintainable, and extensible shell scripts so you don’t have to push any of those other tools beyond their natural limits.

Bash scripting is really good for automating a lot of traditional command line tasks, and it can be pretty awkward to achieve the same effect using other tools—especially if your project uses a mix of languages, where using a tool common to one language environment to automate tasks in another can get weird. (Which is part of the reason why there are so many build tools tailored to different languages in the first place, to say nothing of the different languages themselves.)

If you want to incorporate different scripting languages or shells into your project maintenance, this framework makes it easy to do so. However, by starting with Bash, you can implement a ./go init command to check that these other languages or shells are installed and either install them automatically or prompt the user on how to proceed. Since Bash is (almost certainly) already present, users can run your ./go script right away and get the setup or hints that they need, rather than wading through system requirements and documentation before being able to do anything.

Even if ./go init tells the user “go to this website and install this other thing”, that’s still an immediate, tactile experience that triggers a reward response and invites further exploration. (Think of Zork and the first "open mailbox" command.)

Where can I run it?

The real question is: Where can’t you run it?

The core framework is written 100% in Bash and it’s been tested under Bash 3.2, 4.2, 4.3, and 4.4 across OS X, Ubuntu Linux, Arch Linux, Alpine Linux, FreeBSD 9.3, FreeBSD 10.3, and Windows 10 (using all the environments described in the "Will this work on Windows?" section).

How is it tested?

The project’s own ./go test command does it all. Combined with automatic tab-completion enabled by ./go env and pattern-matching via ./go glob, the ./go test command provides a convenient means of selecting subsets of test cases while focusing on a particular piece of behavior. (See ./go help test.)

Again, Sam Stephenson’s Bash Automated Testing System was a huge revelation. During the course of writing 260+ Bats test cases, I learned a few effective ways of setting up test harnesses, found an easy way to emit test suite info in the Bats output, and developed an assertion library I may break out into a new repository.

I also discovered Simon Kagstrom’s kcov code coverage tool which not only provides code coverage for Bash scripts (!!!) but can push the results to Coveralls! See for yourself:

Note that the coverage shows some lines as uncovered that clearly are; it seems kcov doesn’t recognize commands continued across multiple lines using \. It’s a fixable problem, and certainly not a major one. I’m just astonished it works so well and so quickly as it does!

The latest kcov currently isn’t available to Travis CI via apt-get, so I had to write the scripts/lib/kcov library to build it on-the-fly. Not an ideal solution, but I may break it out into a plugin for reuse across projects until an up-to-date version is available to Travis. (And I learned even more about how to effectively test Bash scripts that execute various binaries!)

So what’s next?

Eventually I’d like to run ShellCheck on the code, but my weird choice of prefixing internal functions with _@go. apparently doesn’t jive well with it because of the @ and . characters, even though Bash considers them valid. I may clone the ShellCheck repo and try to resolve this myself.

There’s a plugin model in-place (./go help plugins), but I haven’t actually used it yet. As I break the assertion library and other pieces into their own repos, I’ll put this model to use and see how well it works in practice. It’s possible the plugin interface may change somewhat based on that experience.

I know there’s more I want to add, some I can probably take away, and there’s no doubt the documentation could use a lot of work. (Though I’m proud that most of the documentation resides in the command scripts themselves, accessible via ./go help!) I plan to actively develop this framework as I use it on my own projects in the future—including, as of today, this blog!

This has been a labor of love for the past few weeks, and I’ve learned a lot about Bash—what you can do with it, what to watch out for (the commit history may provide entertainment for the curious—for example, Bash 4.4 was released last night, and it caused a couple of breakages that I fixed this morning), and how you can actually test it. It’s been an absolute blast diving into this challenge, and if anyone reading this is inclined to give it a try, I’d love to hear your experiences with writing your own ./go scripts using this framework. If you like it so much you’d care to contribute to its improvement, even better!

Postscript

In other news, today is the fifth anniversary of my last day at Google.

Yesterday I presented another iteration on the Google Testing Grouplet story at Netflix entitled A Game of Tests or, A Song of Freedom and Responsibility. Whereas The Convergence of Wills emphasized the need for organizations to serve their people and grant them autonomy, Netflix is known for its Freedom and Responsibility credo. Consequently, this talk emphasizes how Googlers eventually adopted common testing tools and practices willingly after the Testing Grouplet worked hard to convince them of their value.

Many thanks to Mike McGarr and Dianne Marsh for honoring me with this opportunity, and for their generous hospitality. I had a wonderful time hanging out with several Netflix folks for the entirety of the afternoon and evening, learning about Netflix culture and operations and cracking up over a couple beers at the end of the day. It was also great to see Noel Yap, another fellow ex-Googler, who’s been at Netflix for some time now—and who, in my recent bout of obsessive absent-mindedness, I neglected to contact ahead of time.

(I’ll address this recent obsession more fully in a future post, but to preview: I’ve revisited the ./go script concept, this time as a highly-modular, pure bash framework that I’ve posted to GitHub as mbland/go-script-bash. I also got sidelined for a day by my friend Brian Lefler from USDS, who innocently asked if, after my recent bash studies, I knew of a better way to implement grep -o $exp | sort | uniq -c to count unique occurences of strings matching $exp. This led me down a dark, yet enlightening path that I documented in my count-matching-lines.sh gist.)

Today I’m taking a day to hang out in Los Gatos and Mountain View, retracing old paths, awakening memories that are surprisingly more pleasant than I remembered. I’m especially thrilled that Dana Street Roasting Company is still thriving. I’m about to meet Henry Poole from CivicActions for lunch in Sunnyvale (thanks to an intro from Rob Read), and I’m really looking forward to dinner with my old Testing Grouplet partners-in-crime Ana Ulin and Antoine Picard this evening. Then it’s back to DC tomorrow.

I never thought I would ever say this, but…it’s kinda nice to be back here.