How do I make vendoring work with Google App Engin

Question

I am trying to introduce Go vendoring (storing dependencies in a folder called vendor) to an existing App Engine project. I have stored all dependencies in the vendor folder (using Godep as a helper) and it looks right, but running the application locally I get the following error:

go-app-builder: Failed parsing input: package "golang.org/x/net/context" is imported from multiple locations: "/Users/erik/go/src/github.com/xyz/abc/vendor/golang.org/x/net/context" and "/Users/erik/go/src/golang.org/x/net/context"

I believe the two locations should resolve to the same location, as Go applications should look in the vendor folder first. Is there a way to make Appengine understand that both dependencies are the same?

Answer 1

Your project directory (where app.yaml is) is probably in the GOPATH/src. It shouldn't be. The go-app-builder will take everything in the app.yaml folder (and below) and additionally merge your GOPATH into it, meaning now you have it twice.

The solution is to move app.yaml out of the GOPATH/src folder. Additionally you'll find that goapp test works differently from goapp serve and goapp deploy when it comes to resolving dependencies.

So this is the solution I have been using (haven't used golang app engine in a while already) and it's the only setup I've found to work properly for all the goapp commands and for govendor to work properly (not sure about godep)

/GOPATH
├──/appengine
|   ├── app.yaml
|   └── aeloader.go
└──/src
    └── /MYPROJECT
         ├── main.go
         ├── /handler
         |    └── handler.go
         └── /vendor

details:

file: GOPATH/appengine/aeloader.go (NOTE the init function is necessary, probably a bug though)
package mypackage

import (
    _ "MYPROJECT"
)

func init() {
}

now run goapp serve and goapp deploy from ../GOPATH/appengine/ and goapp test ./... from ../GOPATH/src/MYPROJECT

P.S. I find the global GOPATH thing silly and simply set my GOPATH to current project folder (in the example above /GOPATH) and check the whole thing into version control.

Answer 2

I managed to resolve this error using govendor instead of Godeps. The root cause appears to have been that vendored references with their own vendored references was not resolved correctly by Godeps.

The answer provided by Su-Au Hwang is also correct - you do have to keep app.yaml separate from your source.

Answer 3

I use a Makefile to move the vendor directory to a temporary GOPATH:

TMPGOPATH := $(shell mktemp -d)

deploy:
  mv vendor $(TMPGOPATH)/src
  GOPATH=$(TMPGOPATH) gcloud app deploy
  mv $(TMPGOPATH)/src vendor 

I store this Makefile at the root of my service near the vendor directory and simply use make deploy to deploy manually or from the CI.

It works with Glide, Godeps or any tool that respects the Go vendor spec.

Please note, that you really need to move the vendor directory out of the build directory, otherwise the GoAppEngine compiler will try to build the vendor dependencies, potentially causing compile errors.

Answer 4

I just ran into this issue myself actually. The problem occurs when you're using the App Engine tools to build any package which imports something that is using vendoring, but the package you're trying to run doesn't have the import within it's vendor directory.

So, for example, if I'm trying to run package foo, which imports package bar, and both of which use the github.com/gorilla/mux library, if the bar repository has a vendor/ directory that contains gorilla/mux, but the foo package doesn't have gorilla mux in it's vendor/ directory, this error will occur.

The reason this happens is that the bar package will prioritize it's own vendor package over the one in the GOPATH, which is what foo will be using, causing a difference in the actual location of the imported paths.

The solution I found to this issue is to make sure that the foo directory is in the GOPATH and has the vendor directory properly installed. It's important to note that the vendor/ convention only works from within the GOPATH.

Answer 5

Also got the same problem. In the docs Google suggests the following:

For best results, we recommend the following:

• Create a separate directory in your app's directory for each service.
• Each service's directory should contain the service's app.yaml file and one or more .go files.
• Do not include any subdirectories in a service's directory.
• Your GOPATH should specify a directory that is outside your app's directory and contain all the dependencies that your app imports.

But this messes up my project structure, which looks like this:

GOPATH/
└── src
    └── github.com
        └── username
            └── myproject
                ├── app.yaml
                ├── cmd
                │   └── myproject
                │       └── main.go
                ├── handlers
                │   └── api.go
                ├── mw
                │   ├── auth.go
                │   └── logger.go
                └── vendor

Where the myproject directory is a git project and the vendor folder contains all dependencies. Running gcloud deploy from the myproject directory where app.yaml file lives doesn't work because first, main.go file is not in the same directory and second (from the same doc):

you must be careful not to place your source code at or below your app's directory where the app.yaml file is located

What I ended up doing is building my own custom runtime instead, which turned out to be a very clean solution.
Simply generate the Dockerfile with the following command:

gcloud beta app gen-config --custom

Modify it, then specify runtime: custom in your app.yaml and deploy normally.
The trick here is of course that you're in control what gets copied where.
Here is my Dockerfile:

# Dockerfile extending the generic Go image with application files for a
# single application.
FROM gcr.io/google-appengine/golang

ENV GOPATH /go

# The files which are copied are specified in the .dockerignore file
COPY . /go/src/github.com/username/myproject/

WORKDIR /go/src/github.com/username/myproject/

RUN go build -o dist/bin/myproject ./cmd/myproject

# All configuration parameters are passed through environment variables and specified in app.yaml
CMD ["/go/src/github.com/username/myproject/dist/bin/myproject"]

Don't forget that App Engine expects your application listening on port 8080. Check out Building Custom Runtimes doc for more details.