How can I interpret variables on the fly in the sh

Question

I'm reading JSON in a shell script using JQ. Here, I'm unable to interpret the variables $HOME, $HOST, $PEMFILE in my shell script on the fly.

JSON File:

{
    "script": {
    "install": "${HOME}/lib/install.sh $HOST $PEMFILE",
    "Setup": "${HOME}/lib/setup.sh $HOST $PEMFILE $VAR1 $VAR2"
    }

}

Shell Script:

#!/bin/bash
examplefile="../lib/example.json"
HOST=ec2-..-...-...-...us-west-2.compute.amazonaws.com
PEMFILE=${HOME}/test.pem

installScript=($(jq '.script.install' $examplefile))
bash "$installScript"

Is there a way I can interpret these variables on the fly without modifying the JSON?

P.S I don't want to use eval.

Answer 1

Specific solution

Here's a jq solution to the stated problem, though it will only work for "global" environment variables.

def substitute:
  gsub("\\${HOME}"; env.HOME)
  | gsub("\\$HOST"; env.HOST)
  | gsub("\\$PEMFILE"; env.PEMFILE)
  | gsub("\\$VAR1"; env.VAR1)
  | gsub("\\$VAR2"; env.VAR2)
  ;

walk( if type=="string" then substitute else . end )

If your jq does not already have walk/1, then please either upgrade your jq or snarf the def from https://github.com/stedolan/jq/blob/master/src/builtin.jq

The solution above is a bit brittle but it could easily be robustified or generalized, as shown in the next section.

General solution

walk(if type == "string"
     then gsub("\\$(?<x>[A-Za-z_][A-Za-z0-9_]+)"; "\(env[.x])") 
          | gsub("\\${(?<x>[A-Za-z_][A-Za-z0-9_]+)}"; "\(env[.x])") 
     else . end)

Answer 2

I've been hitting this on and off for years. I think I've finally got a decent pure-bash solution: uses regex matching and indirect parameter substitution

# read the file
json=$(< file.json)
echo step 0
echo "$json"

# set the relevant vars, just plain shell variables
HOST=_host_
PEMFILE=_pemfile_
VAR1=_var1_
VAR2=_var2_

# replace '$var' forms
while [[ $json =~ ("$"([[:alnum:]_]+)) ]]; do 
    json=${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
done; 
echo
echo step 1
echo "$json"

# replace '${var}' forms
while [[ $json =~ ("$""{"([[:alnum:]_]+)"}") ]]; do 
    json=${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
done
echo
echo step 2
echo "$json"

Output

step 0
{
    "script": {
    "install": "${HOME}/lib/install.sh $HOST $PEMFILE",
    "Setup": "${HOME}/lib/setup.sh $HOST $PEMFILE $VAR1 $VAR2"
    }

}

step 1
{
    "script": {
    "install": "${HOME}/lib/install.sh _host_ _pemfile_",
    "Setup": "${HOME}/lib/setup.sh _host_ _pemfile_ _var1_ _var2_"
    }

}

step 2
{
    "script": {
    "install": "/home/jackman/lib/install.sh _host_ _pemfile_",
    "Setup": "/home/jackman/lib/setup.sh _host_ _pemfile_ _var1_ _var2_"
    }

}

The magic is:

1. the regular expression, where I capture both $VAR and VAR, and

   [[ $json =~ ("$"([[:alnum:]_]+)) ]]
   # ..........1   2             21
   

2. the parameter substitution, where I search for the string "$VAR" and replace it with the indirect variable expansion ${!VAR}

   ${json//${BASH_REMATCH[1]}/${!BASH_REMATCH[2]}}
   

Answer 3

Here is a solution using env and gsub to perform the replacement.

Note that env requires the variables to be passed as environment variables as opposed to shell variables.

#!/bin/bash

examplefile="../lib/example.json"
HOST=ec2-..-...-...-...us-west-2.compute.amazonaws.com
PEMFILE=${HOME}/test.pem

export HOST
export PEMFILE
installScript=$(jq -Mr '
   .script.install | gsub("(?<x>[$][{]?\\w+[}]?)"; env[.x|gsub("[${}]+";"")] )
' $examplefile)

echo $installScript

Sample Output

/home/runner/lib/install.sh ec2-..-...-...-...us-west-2.compute.amazonaws.com /home/runner/test.pem

Try it online!

Answer 4

#!/bin/sh

TMP=$(mktemp /tmp/$$.XXX)

cat<<E_O_F > $TMP
cat <<EOF
$(cat so-dollar-variables.json)
EOF
E_O_F

. $TMP

/bin/rm "$TMP"

Answer 5

It is easy using gnu utility envsubst:

installScript=$(jq -r '.script.install' "$examplefile" | envsubst)