My most loathed feature of Go is the mandatory use of GOPATH:
I do not want to put my own code next to its
dependencies. Hopefully, this issue is slowly starting to
be accepted by the main authors. In the meantime, you can
workaround this problem with more opinionated tools (like gb) or
by crafting your own Makefile.
For the later, you can have a look at Filippo Valsorda’s example
or my own take which I describe in more details here. This is not
meant to be an universal Makefile but a relatively short one with
some batteries included. It comes with a simple “Hello World!” application.
Project structure§
For a standalone project, vendoring is a must-have1 as
you cannot rely on your dependencies to not
introduce backward-incompatible changes. Some packages are
using versioned URLs but most of them aren’t. There is
currently no standard tool to handle vendoring. My personal take
is to vendor all dependencies with Glide.
It is a good practice to split an application into different packages
while the main one stay fairly small. In the hellogopher example,
the CLI is handled in the cmd package while the application logic
for printing greetings is in the hello package:
Down the rabbit hole§
Let’s take a look at the various “features” of the Makefile.
GOPATH handling§
Since all dependencies are vendored, only our own project needs to be
in the GOPATH:
The base import path is hellogopher, not
github.com/vincentbernat/hellogopher: this shortens imports and
makes them easily distinguishable from imports of dependency
packages. However, your application won’t be go get-able. This is a
personal choice and can be adjusted with the $(PACKAGE) variable.
We just create a symlink from .gopath/src/hellogopher to our root
directory. The GOPATH environment variable is automatically exported
to the shell commands of the recipes. Any tool should work fine after
changing the current directory to $(BASE). For example, this
snippet builds the executable:
Vendoring dependencies§
Glide is a bit like Ruby’s Bundler. In glide.yaml, you specify
what packages you need and the constraints you want on
them. Glide computes a glide.lock file containing the exact
versions for each dependencies (including recursive dependencies) and
download them in the vendor/ folder. I choose to check into the VCS
both glide.yaml and glide.lock files. It’s also possible to only
check in the first one or to also check in the vendor/ directory. A
work-in-progress is currently ongoing to provide
a standard dependency management tool with a similar workflow.
We define two rules2:
We use a variable to invoke glide. This enables a user to easily
override it (for example, with make GLIDE=$GOPATH/bin/glide).
Using third-party tools§
Most projects need some third-party tools. We can either expect
them to be already installed or compile them in our private
GOPATH. For example, here is the lint rule:
As for glide, we let the user a chance to override which golint
executable to use. By default, it uses a private copy. But a user can
use its own copy with make GOLINT=/usr/bin/golint.
In ❶, we have the recipe to build the private copy. We simply issue
go get3 to download and build golint. In ❷, the lint rule
executes golint on each package contained in the $(PKGS)
variable. We’ll explain this variable in the next section.
Working with non-vendored packages only§
Some commands need to be provided with a list of packages. Because we
use a vendor/ directory, the shortcut ./... is not what we expect
as we don’t want to run tests on our
dependencies4. Therefore, we compose a list of packages
we care about:
If the user has provided the $(PKG) variable, we use it. For
example, if they want to lint only the cmd package, they
can invoke make lint PKG=hellogopher/cmd which is more intuitive
than specifying PKGS.
Otherwise, we just execute go list ./... but we remove anything from
the vendor directory.
Tests§
Here are some rules to run tests:
A user can invoke tests in different ways:
make test runs all tests;
make test TIMEOUT=10 runs all tests with a timeout of 10 seconds;
make test PKG=hellogopher/cmd only runs tests for the cmd package;
make test ARGS="-v -short" runs tests with the specified arguments;
make test-race runs tests with race detector enabled.
Tests coverage§
go test includes a test coverage tool. Unfortunately, it only
handles one package at a time and you have to explicitely list
the packages to be instrumented, otherwise the instrumentation is
limited to the currently tested package. If you provide too many
packages, the compilation time will skyrocket. Moreover, if you want
an output compatible with Jenkins, you’ll need some additional tools.
First, we define some variables to let the user override them. We also
require the following tools (in ❸):
gocovmerge merges profiles from different runs into a single one;
gocov-xml converts a coverage profile to the Cobertura format;
gocov is needed to convert a coverage profile to a format handled
by gocov-xml.
The rules to build those tools are similar to the rule for golint
described a few sections ago.
In ❹, for each package to test, we run go test with the
-coverprofile argument. We also explicitely provide the list of
packages to instrument to -coverpkg by using go list to get a list
of dependencies for the tested package and keeping only our owns.
Final result§
While the main goal of using a Makefile was to work around GOPATH,
it’s also a good place to hide the complexity of some operations,
notably around test coverage.
The excerpts provided in this post are a bit simplified. Have a look
at the final result for more perks!
In Go, “vendoring” is about both bundling and
dependency management. As the Go ecosystem matures, the
bundling part (fixed snapshots of dependencies) may become
optional but the vendor/ directory may stay for dependency
management (retrieval of the latest versions of dependencies
matching a set of constraints). ↩
If you don’t want to automatically update glide.lock when a
change is detected in glide.yaml, rename the target to
deps-update and make it a phony target. ↩
There is some irony for bad mouthing go get and then
immediately use it because it is convenient. ↩
I think ./... should not include the vendor/
directory by default. Dependencies should be trusted to have run
their own tests in the environment they expect them to
succeed. Unfortunately, this is unlikely to change. ↩