How to config Meteor on AWS/EBS using METEOR_SETTI

Question

Trying to set up a Meteor on an AWS/EBS (Amazon Web Services, Elastic Beanstalk) environment.

A Meteor dev-run can be passed a command line flag: --settings settings.json where settings.json is a file containing server/client key/value configs (as properly-formatted JSON).

Instead of passing the config file in command line, Meteor's deployment uses a METEOR_SETTINGS environment variable. If provided it is expected to contain a json document such as contents of settings.json, for example:

$ METEOR_SETTINGS=$(cat settings.json)
$ echo $METEOR_SETTINGS
{ "public": { "s3path": "https://d2v4p3rms9rvi3.cloudfront.net" } }

The problem is that when I set the value of METEOR_SETTINGS to this value in the EBS console:

AWS/EBS discards the quotes, escapes the slashes (as seen in screenshot), and sends Meteor:

{public:{s3path:https:\/\/d2v4p3rms9rvi3.cloudfront.net}}

As indicated by the node start up error:

-------------------------------------
/var/log/nodejs/nodejs.log
-------------------------------------
npm WARN deprecated backwards-incompatible changes made to `npm run-script` and
npm WARN deprecated semver behavior.

> meteor-dev-bundle@0.0.0 start /var/app/current
> node main.js


/var/app/current/programs/server/boot.js:283
}).run();
   ^
Error: METEOR_SETTINGS are not valid JSON: {public:{s3path:https:\/\/d2v4p3rms9rvi3.cloudfront.net}}
    at packages/meteor/packages/meteor.js:21:1
    at Package (packages/meteor/packages/meteor.js:42:1)
    at /var/app/current/programs/server/packages/meteor.js:1277:4
    at /var/app/current/programs/server/packages/meteor.js:1286:3
    at /var/app/current/programs/server/boot.js:242:10
    at Array.forEach (native)
    at Function._.each._.forEach (/var/app/current/node_modules/underscore/underscore.js:79:11)
    at /var/app/current/programs/server/boot.js:137:5

Bumping against this problem I tried all sorts of variations for the JSON object in the value field: escaping the quotes, enclosing the entire json part with single quotes, replacing double-quotes with single-quotes, and other attempts - neither solved it.

Question is:

How can METEOR_SETTINGS be set so that Meteor rcv & parse it correctly?

Note: one of the requirements is that the same build deploys to dev, staging and production environments. Configs need to be set separately for each environment thus if there's another way to inject the settings into the EBS environment w/o modifying the build that will also solve it.

Answer 1

NEW SOLUTION

This is thet easier way I'm able to come with. I've made a bash script that automatically generates a compressed build of your project and integrates the settings file so you don't really have to do anything.

BEFORE TO RUN THIS SCRIPT

In your meteor project create ./lib/beanstalk-settings-fix.js

/*==============================================================================
 * Globals
 *============================================================================*/
/*global process*/
/*global Meteor*/
/*global Npm*/


if (Meteor.isProduction){
  var meteorFile = process.env.METEOR_SETTINGS_FILE;
  if(meteorFile == undefined) throw new Error(
    'METEOR_SETTINGS_FILE env variable must be defined in production.')
  var fs = Npm.require('fs');
  var pjsonBuf = fs.readFileSync( meteorFile );
  Meteor.settings = JSON.parse( pjsonBuf.toString().trim());
}

HOW TO USE IT

1. Copy the next code to a text file and save it as build.sh
2. Edit the constants.
3. Give it execution permissions and run it.

You'll end with something like project-name.zip ready to upload it to your beanstalk environment. I hope you find it useful!

This solution is based on AWS forums. If you want to check the old solutions, please check the edit history.

#!/bin/bash

#===============================================================================
# DESCRIPTION:
#===============================================================================
# This script creates a build of the project ready to be uploaded to beanstalk.
# Requires pyton 2.7.x

#===============================================================================
# COMMON ISSUES:
#===============================================================================
# -If you upload the output to a sample application, it will fail.
# -Version format must be 0.0.0

#===============================================================================
# CONSTANTS
#===============================================================================
CURRENT_VERSION="1.0.0"
OUTPUT_NAME="file-name-without-extension"
PRODUCTION_SETTINGS_JSON="./your-project-directory/settings-prod.json"
PROJECT_DIRECTORY="./your-project-directory"
OUTPUT_DIRECTORY="./the-output-directory"
ROOT_URL="http://www.SOMEENVIRONMENT-env.us-west-2.elasticbeanstalk.com"
MONGO_URL="none"

#===============================================================================
# SAY HELLO
#===============================================================================
initial_directory=$(pwd)  #  This file's local path
clear
echo "COOKING OUTPUT"
echo "========================================================="

#===============================================================================
# RAW PROJECT BUILD
#===============================================================================
echo "> BUILDING RAW PROJECT"
cd $initial_directory
cd $PROJECT_DIRECTORY
rm -f -R "../build/bundle"
meteor build --directory ../build/

#===============================================================================
# SET PRODUCTION ENVIRONMENT VARIABLES
#===============================================================================
cd $initial_directory
json=`cat $PRODUCTION_SETTINGS_JSON`
cd $OUTPUT_DIRECTORY/bundle
mkdir -p .ebextensions
echo "option_settings:" >> .ebextensions/environment.config
echo "  - option_name: MONGO_URL" >> .ebextensions/environment.config
echo "    value: $MONGO_URL" >> .ebextensions/environment.config
echo "option_settings:" >> .ebextensions/environment.config
echo "  - option_name: ROOT_URL" >> .ebextensions/environment.config
echo "    value: "$ROOT_URL"" >> .ebextensions/environment.config
echo "files:" >> .ebextensions/environment.config
echo " '/tmp/settings.json':" >> .ebextensions/environment.config
echo "    content : |" >> .ebextensions/environment.config
echo "        "$json >> .ebextensions/environment.config
echo "option_settings:" >> .ebextensions/environment.config
echo "  - namespace:  aws:elasticbeanstalk:application:environment" >> .ebextensions/environment.config
echo "    option_name: METEOR_SETTINGS_FILE" >> .ebextensions/environment.config
echo "    value: '/tmp/settings.json'" >> .ebextensions/environment.config
chmod 444 .ebextensions/environment.config
echo "> ADDING 'settings.json' AS ENV VAR"

#===============================================================================
# CREATE PACKAGE.JSON
#===============================================================================
cd $initial_directory
cd $OUTPUT_DIRECTORY/bundle

# Write base package.json
echo '{
  "name": "'$OUTPUT_NAME'",
  "version": "'$CURRENT_VERSION'",
  "scripts": {
    "start": "node main.js"
  },
  "dependencies": {
  }
}' > ./package.json

# Add dependencies from meteor in packages.json
# Then add extra dependencies defined by us.
EXTRA_DEPENDENCIES='{"forever": "*"}'
meteor_packages=$(cat ./programs/server/package.json)
packages=$(cat ./package.json)
packages_updated=`python <<END
import json;
# We cannot operate directly bash variables, so we make a copy.
a = $packages
b = $meteor_packages
a['dependencies'] = b['dependencies']
for key, value in $EXTRA_DEPENDENCIES.iteritems():
    a['dependencies'].update({key: value})
print json.dumps(a, sort_keys=False, indent=4, separators=(',', ': '));
END`
echo "$packages_updated" > ./package.json
chmod 444 ./package.json
echo "> ADDING 'package.json'"

#===============================================================================
# ZIP OUTPUT
#===============================================================================
cd $initial_directory
cd $OUTPUT_DIRECTORY/bundle
zip -FSrq "../$OUTPUT_NAME-$CURRENT_VERSION.zip" .
echo "> ZIP THE OUTPUT"

#===============================================================================
# CLEAN THE HOUSE
#===============================================================================
cd $initial_directory
cd $OUTPUT_DIRECTORY
rm -R -f ./bundle
echo "> CLEAN THE HOUSE"

#===============================================================================
# SAY GOODBYE
#===============================================================================
echo "========================================================="
echo "YOU CAN UPLOAD THE PROJECT TO A BEANSTALK ENVIRONMENT NOW"

Some extra help: In case you want to check that everything went fine, you can find your final settings in your zipped output, under /.ebextensions/environment.config and your packages file under /package.json

Answer 2

After discussing this issue with AWS support I realized that AWS/EBS does not support storing JSON in environment variables. This is because the environment variables are stored as key/value strings in unencoded JSON (apparently, in CloudFormation). The bottom line here a bit disappointing:

METEOR_SETTINGS cannot be used in the AWS/EBS console

This is indeed unfortunate, however there are a couple of workarounds.

1st Workaround

Move the json configs into an s3 bucket and place the following content in a .ebextensions/app.config file:

container_commands: 
  01_setvariable: 
    command: "aws s3 cp s3://<bucket-name>/nodejs.conf /tmp/deployment/config/#etc#init#nodejs.conf 

This will entirely override /etc/init/nodejs.conf with content retrieved from your s3 bucket. Naturally there's an opportunity to set/override individual settings using fine-tuned/fancy bash scripting.

I ended up not choosing this method, because it involves another entity (an S3 bucket) and the dev iteration requires a new version deploy, which isn't terribly fast.

2nd Workaround

Note: this is a simple code-hack I came up with. It seems to put all this mess behind while not requiring much effort.

My original need was to propagate AWS/EBS env vars to the client, so I decided to bypass the METEOR_SETTINGS variable and populate Meteor.settings.public directly with env vars from node's process.env space. The whitelisting is managed by a simple list. Add a server/lib/config.js file with:

Meteor.startup(function () {
    // public settings that need to be exposed to the client can be added here
    var publicEnvs = {
        S3_PATH: 's3path'
    };
    var modified;
    _.each(publicEnvs, (value, key) => {
        let envValue = process.env[key];
        if (envValue) {
            Meteor.settings.public[value] = envValue;
            modified = true;
        }
    });
    if (modified) {
        __meteor_runtime_config__.PUBLIC_SETTINGS = Meteor.settings.public;
    }
});

Hurray, your client can access the env vars of your choice!

For example with this change, an S3_PATH environment variable defined in the EBS console can be accessed as Meteor.settings.public.s3path on the client. Quite simple, and without many moving parts :)

Answer 3

Tested another workaround.

after meteor build --directory edit main.js as following

process.argv.splice(2, 0, 'program.json');
var settingfilename = './settings.json';
if (process.env.METEOR_SETTING_FILE)
  settingfilename = process.env.METEOR_SETTING_FILE;
var settings = require(settingfilename);
if (settings) {
  try {
    process.env.METEOR_SETTINGS = JSON.stringify(settings);
  } catch (e) {
    console.error(e);
  }
}
process.chdir(require('path').join(__dirname, 'programs', 'server'));
require('./programs/server/boot.js');

and copy settings.json into bundle/ and eb init and eb deploy.

you can set the other settings file with adding METEOR_SETTING_FILE at Environment Properties in Configuration tab from EB Console.

editing file is needed after every build.

---

added the patch file to use in the build script like ed - ../build/bundle/main.js < main.js.patch

main.js.patch

8a
var settingfilename = './settings.json';
if (process.env.METEOR_SETTING_FILE)
  settingfilename = process.env.METEOR_SETTING_FILE;
var settings = require(settingfilename);
if (settings) {
  try {
    process.env.METEOR_SETTINGS = JSON.stringify(settings);
  } catch (e) {
    console.error(e);
  }
}
// console.log (JSON.stringify(process.env));
.
w