Go modules Dev Dependencies

[bcomnes]

Go didn't have a concept of devDependencies like node.js does for a long time. They are sort of addressing this problem in a very go-esque way it sounds.

• Go modules has something called a tools.go file: cmd/go: clarify best practice for tool dependencies golang/go#25922 (comment)

• Details on go module support for vendoring cmd/go: add modvendor sub-command golang/go#27618

• Experimental proposal thread: add experimental support for go.tools myitcv/gobin#44

• gex: a tool to help manage devDeps: https://github.com/izumin5210/gex

• https://github.com/srvc/wraperr: a project using tools.go

• https://github.com/izumin5210/grapi: another project using tools.go pattern

• https://github.com/oxequa/realize A go task runner

• Use modules to install a "dev dependency" for use before or during a build golang/go#29494 Another example of someone asking the same issue

• cmd/go: support module-local install/run of tool dependencies golang/go#27653: issue addressing some of the complications with the current solutions, canns the idea of devDeps beyond what's supported.

• Go get outside of a module scope for global deps: cmd/go: allow "go get" when outside a module in module mode golang/go#24250

• Retool: https://github.com/twitchtv/retool vendoring go bins

• go hack https://github.com/rogpeppe/gohack

• https://github.com/myitcv/gobin go bin, a proposed solution

• https://github.com/myitcv/gobin/wiki/FAQ gobin fan

• go 1.12 modules https://tip.golang.org/doc/go1.12#modules

• Go tools.go https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md

• https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module

• https://godoc.org/golang.org/x/tools/cmd/stringer a tool often used as an example

[bcomnes]

Another solution: https://github.com/nordicdyno/golang-tools

[mcandre]

Both tools.go and golang-tools have problems, due to being third party systems.

They rely on make + bash + awk to parse Go code. That's not particularly maintainable.

Nevermind the complexity of setting up a Go project for contribution on native Windows COMSPEC (Command Prompt or PowerShell) environments. I may love the WSL effort, but I don't like having requiring anyone to install WSL when working on my projects. I'm not writing polyglot scripts with tofrodos hacks. Please don't suggest batsh, or worse, Python, as a shim.

You might as well just install the tools in a Dockerfile, and always work in Docker. These workflows are heavyweight, compared to regular, built-in go mod support. One of Go's best qualities is that the built-in functionality is usually very comprehensive and very cross-platform, not requiring external systems or any specific shell interpreter.

You can't simply reuse the go mod path when determining the right path as far as go get is concerned. With these hacks, It's difficult to distinguish between binaries that require explicit ellipses (...) during installation (for multiple binaries), versus ordinary top-level artifacts.

go get -u is highly recommended for any x items from the Go extension collection, that may break across Go versions. But may or may not be desired for other items. Some developers may prefer to assume good Go semver and automatically float tool versions. Other developers know just how poorly real projects implement semver, and will prefer to pin tool versions down to a VCS ref, just in case of any remote possibility of breaking changes. So handling even just the -u flag appropriately per tool, can be challenging from the perspective of the existing go.mod spec.

Another problem with the third party systems is that each change in developer tools would require two different file changes: One to go.mod and another to tools.go. Three if you include the checksum file. That's a code smell. Caused by the Go language's resistance to implementing NPM/RubyGems pre-existing ability to denote development dependencies, particularly support for making sure that executable artifacts get pulled in correctly into $GOPATH/bin.

The most reliable system I've come up with so far is a POSIX sh script, like accio. It sets a POSIX shebang, enables -e. And then it runs a series of go get [-u] commands. With git checkout commands interspersed to enforce pinning. Finally, a go mod tidy command to fix the go.mod file. Super nasty, twice as slow as it should be (due to double-checking out the source code). And requires the user to have a POSIX shell available. And you have to remember to use LF endings. For COMSPEC users, they have to copy and paste the go get lines into their terminal session, in order to prevent the installations from occuring in the wrong file system. Executable permission is added for convenience, but sh accio is the recommended invocation syntax, to try to widen support for non-UNIX hosts. No support for tools housed in non-git version control repositories. Poor support for x collection URL's. Honestly, the pinning process is so arduous to manage external to go mod, that I'd rather just float all of my tools to latest and take the risk of breakages and malware.

You could in theory automate these things with an insane go:generate setup, or write yet another third party tool. (Which inevitably breaks as the Go language releases new versions.) I might have a try at using gotpl to simplify this. But pinning the gotpl version itself creates a chicken and egg problem. Same if using Mage. Maybe the Go template lib, without actually invoking the CLI helper? I dunno. All of these suggestions probably pollute the local file tree with redundant copies between any vendor caches versus the larger $GOPATH/src tree. Unfortunately, Go has stretched the meaning of the path in go get <path> to the limit, impossible to reuse this path without significant handling of different cases.

Not. Fun.

We want the normal go mod system to actually implement tool management as a first party system, in order to reliably resolve all of the edge cases without requiring massive extra components scrambling to solve half of them.

[bcomnes]

Yeah me to. Looks like go run @Version is in though, probably could work something out with that

golang/go#42088

[mcandre]

I am automating the low level go get, git checkout commands with a more pure Go system, accio. No need for flakey shell scripts, or makefiles, or awk, or other such nonsense. Just the Go programming language, and git.

You may now use an accio.yaml configuration file to pin your Go developer tools which go mod has not deigned to track. It's so crude, but for my projects it's a kind of local max for convenience and reproducible builds.

https://github.com/mcandre/accio

[bcomnes]

That's cool I'll try it out next time I'm on a go project.