How to use private repositories with Go modules

Since the release of 1.13, Go has native Go Modules support, with go mod. Similar to other tooling in the Go ecosystem, like go fmt for code formatting, it is intended to be the default, community-concended approach to dependency management. It supersedes existing approaches, such as Glide and Dep, and integrates smoothly within the existing Go workflow.

It works out of the box, without any additional configuration.

However, that is only true when all dependencies are publicly available. With dependencies on private packages, additional configuration is needed.

Resolving dependencies over SSH instead of HTTPS

Without any additional steps, trying to load a private dependency into the Go Modules project, will result in the following error:

go: downloading github.com/myorg/privaterepo v0.0.0-20200408100711-ed766a2975ce
go get github.com/myorg/privaterepo: github.com/myorg/[email protected]: verifying module: github.com/myorg/[email protected]: reading https://sum.golang.org/lookup/github.com/myorg/[email protected]: 410 Gone
        server response:
        not found: github.com/myorg/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/13e63a509893edc19353a80fa2c6e28db213d146f72fe43ba65c1ec86624027b: exit status 128:
                fatal: could not read Username for 'https://github.com': terminal prompts disabled

The problem here is that, by default, Go’s dependency resolving approach attempts to download the package with git over HTTPS.

We want to (instead of using HTTPS) download the private dependency over SSH, such that we can use the local SSH keys to gain access to the package repository. This assumes that you have configured SSH with credentials that can access the private repositories. If you have not, Github has an excellent tutorial how to set it up.

Because this is not something we can change in Go, what we can do is to configure git to rewrite all HTTPs to the equivalent git URLs:

git config --add --global url."[email protected]:".insteadOf https://github.com
git config --add --global url."[email protected]:".insteadOf https://bitbucket.org
git config --add --global url."[email protected]:".insteadOf https://gitlab.com

By-passing GOPROXY for private dependencies

Although this has been resolved in the newer versions of Go, in earlier versions resolving the dependencies over SSH still does not completely solve the issue. When we try to download the private dependency, with the SSH patch, the following error will occur:

go: github.com/myorg/[email protected]: reading https://proxy.golang.org/github.com/myorg/privaterepo/@v/v0.0.0-20200512204819-655dcc74320a.mod: 410 Gone

This error occurs due to the introduction of a new default for GOPROXY in Go 1.13. From that version on, Go uses a module mirror (https://proxy.golang.org) to resolve the modules. This mirror obviously will not contain or be able to find the private repositories.

To fix this, there are a couple of options:

GOPROXY A crude solution to resolve this issue is to simply skip the proxy. This will avoid Go to hit the intermediate proxy servers completely and directly contact Github. To do this, set:

export GOPROXY=direct

GOPRIVATE Another option is to specifically mark packages as private by adding them to GOPRIVATE. You can either mark specific packages:

export GOPRIVATE=github.com/myorg/privaterepo

Alternatively, you could mark all packages in your organization as private:

export GOPRIVATE=github.com/myorg

To permanently use this fix, you can either export these variables in your shell start-up scripts, or you can override the defaults of Go:

go env -w 

To verify that the changes have persisted, you can see the environment used for Go commands with:

go env | grep 'GOPROXY\|GOPRIVATE'

Conclusion

With the fixes in place you should be able to access all private repositories that you can accessible with your loaded SSH credentials. In case, you want to use HTTPS or cannot rely on SSH, there are alternative (in my opinion, more cumbersome) documented approaches that rely on Github access tokens.

Further reading

• General introduction to the workflow around go modules: https://blog.golang.org/using-go-modules
• The what and why around Go module proxies: https://arslan.io/2019/08/02/why-you-should-use-a-go-module-proxy
• About Go modules in general: https://golang.org/doc/go1.13#modules