Connecting Mongoose on Google App Engine

Question

I've deployed a custom Node.js app on Google App engine engine successfully. However I'm having issues starting the app because mongoose times out when attempting to connect. Frustratingly, the mongoose connects just fine on my local machine with the exact same parameters.

My MongoDb URI is of the form:

 mongodb://<dbuser>:<dbpassword>@xxxx.mlab.com:<portNumber>/d‌​b-name

As the URI implies, the db is hosted by mlab. It is a sandbox instance on Google Cloud Platform. I however did not deploy the db with Google Compute Engine. I merely opted to host in on GCP when going through the setup process on mlab.

I've come across similar questions on SO but most of them do not have an accepted answer. Just varying discourse in comments with no accepted solution.

My question therefore is, what differs between my App Engine instance and my local computer when trying to connect to the above URI? Does the fact that I chose for mlab to host it on GCP matter? Would it be any different if I chose for it to be hosted on Amazon AWS? What exactly is the root cause of the issue?

For reference, here are similar questions I've found:

• I can connect to a google cloud deployed mongoDB from localhost but not from the same app when deploying on the app engine

• error connecting to mongodb with mongoose on google compute engine

• Cannot connect to mongodb after app deploys on google app engine

Also, if it helps, mongodb is used as the db for business models, the datastore for the app's Express Session, and for storing transient data from socket.io.

Error stacktrace from App Engine after deployment:

2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent This is most likely not a problem with npm itself
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent and is related to npm not being able to find a file.
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! enoent
2017-10-18 02:13:46 default[20171017t215757]
2017-10-18 02:13:46 default[20171017t215757]  npm ERR! Please include the following file with any support request:
2017-10-18 02:13:46 default[20171017t215757]  npm ERR!     /app/npm-debug.log
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! Linux 3.16.0-4-amd64
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! argv "/nodejs/bin/node" "/nodejs/bin/npm" "start"
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! node v6.11.3
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! npm  v3.10.10
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! path /app/package.json
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! code ENOENT
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! errno -2
2017-10-18 02:23:41 default[20171017t215757]  npm ERR! syscall open

Error stacktrace from running app locally in Google Cloud Shell:

/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb/lib/mongo_client.js:421
          throw err
          ^
MongoError: failed to connect to server [ds147454.mlab.com:47454] on first connect [MongoError: connection 1 to ds147454.mlab.com:47454 timed out]
    at Pool.<anonymous> (/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb-core/lib/topologies/server.js:336:35)
    at emitOne (events.js:96:13)
    at Pool.emit (events.js:188:7)
    at Connection.<anonymous> (/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb-core/lib/connection/pool.js:280:12)
    at Connection.g (events.js:292:16)
    at emitTwo (events.js:106:13)
    at Connection.emit (events.js:191:7)
    at Socket.<anonymous> (/home/myuser/src/project-id/teammate-express-server/node_modules/mongodb-core/lib/connection/connection.js:197:10)
    at Socket.g (events.js:292:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:338:8)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)

Answer 1

As it's been pointed out, you can follow the instruction on https://cloud.google.com/nodejs/getting-started/deploy-mongodb to configure Google App Engine using config.json file. (update: link no longer works, try: https://cloud.google.com/community/tutorials/nodejs-mongodb-on-appengine instead)

Since you're using mongoose the models will dictate the collection usage. i.e. save(), update(). If you're connecting using node.js directly, you can also change database/collection easily in the code.

What differs between my App Engine instance and my local computer when trying to connect to the above URI?

Google App Engine is a fully managed platform. It is in a sense serverless, you are utilising a pool of workers to up/down scale your application. When a request comes in to your application, an instance from the pool will handle the task/request.

Unlike your local machine, where request will be handled by your machine as well.

You may also find Google App Engine FAQ useful.

Does the fact that I chose for mlab to host it on GCP matter?

Not a lot. As you mentioned, you're currently on the SandBox (free) plan with GCP. At the moment this would deploy your MongoDB instance on us-central1 region, and there's no other choice of region. There are also slight variation of RAM/Storage/Price between different cloud provider.

You may be able to reduce network latency by choosing GAE on us-central1 region as well.

What exactly is the root cause of the issue?

It's difficult to know for certain what is the issue without proper stack-trace or error log describing the timeout issue.

Commonly node.js project contain a JavaScript file called config.js which has a line below:

  nconf.argv()
       .env()
       .file({ file: path.join(__dirname, 'config.json') })
       .defaults()

Which basically trying to read configuration in the order of :

• Command line argument
• Environment variables
• Config file (specified path)
• Defaults values

See also Google Cloud Platform: nodejs-getting-started

Also GoogleCloudPlatform/nodejs-docs-samples/appengine/mongodb project for an example.

Answer 2

I have had the same problem for a while but eventually this worked for me.Hope it helps.Ehen I read the the db connection details from a json file using nconf package as described by google https://cloud.google.com/community/tutorials/nodejs-mongodb-on-appengine with no mongoose I was able to solve it.Initially i was reading from env variables.

`         nconf = require('nconf');
          const mongoose = require('mongoose');
          //key.json contains mongodb connection information
          nconf.argv().env().file('keys.json');
          const user = nconf.get('mongoUser');
          const pass = nconf.get('mongoPass');
          const host = nconf.get('mongoHost');
          const port = nconf.get('mongoPort')
          const mongoDatabase=nconf.get('mongoDatabase');
          let mongo_url=`mongodb://${user}:${pass}@${host}:${port}/${mongoDatabase}`;
          mongoose.connect(mongo_url,{useNewUrlParser: true});`

You dont have to use config.json as suggested by other answers.

Answer 3

Well this took me entirely too long, but the issue was with my Dockerfile.

I was extending the node.js runtime but my Dockerfile was missing lines to copy the app directory and install npm.

To create the appropriate Dockerfile I used the following command:

gcloud beta app gen-config --custom

More information about the command and extending the node.js runtime can be found here.

My final Dockerfile after I added the dependencies I needed looks like this:

NOTE: I can only connect to my MongoDb instance after deployment. Running locally in Cloud Shell still times out.

# Dockerfile extending the generic Node image with application files for a
# single application.
FROM gcr.io/google_appengine/nodejs
COPY . /app/

RUN apt-get -y update && apt-get install -y libav-tools && apt-get install -y graphicsmagick

# You have to specify "--unsafe-perm" with npm install
# when running as root.  Failing to do this can cause
# install to appear to succeed even if a preinstall
# script fails, and may have other adverse consequences
# as well.
# This command will also cat the npm-debug.log file after the
# build, if it exists.
RUN npm install --unsafe-perm || \
  ((if [ -f npm-debug.log ]; then \
      cat npm-debug.log; \
    fi) && false)
CMD npm start