Create new file from templates with bash script

问题:

I have to create conf files and init.d which are very similar. These files permit to deploy new http service on my servers. These files are the same and only some parameters change from one file to another (listen_port, domain, path on server...).

As any error in these files leads to misfunction of service I would like to create these files using a bash script.

For example:

generate_new_http_service.sh 8282 subdomain.domain.com /home/myapp/rootOfHTTPService

I am looking for a kind of templating module that I could use with bash. This templating module would use some generic conf and init.d scripts to create new ones.

Do you have hints for that? If not I could use python templating engine.

回答1:

You can do this using a heredoc. e.g.

generate.sh:

#!/bin/sh

#define parameters which are passed in.
PORT=$1
DOMAIN=$2

#define the template.
cat  << EOF
This is my template.
Port is $PORT
Domain is $DOMAIN
EOF

Output:

$ generate.sh 8080 domain.com

This is my template.
Port is 8080
Domain is domain.com

or save it to a file:

$ generate.sh 8080 domain.com > result

回答2:

Template module for bash? Use sed, Luke! Here is an example of one of millions of possible ways of doing this:

$ cat template.txt 
#!/bin/sh

echo Hello, I am a server running from %DIR% and listening for connection at %HOST% on port %PORT% and my configuration file is %DIR%/server.conf

$ cat create.sh 
#!/bin/sh

sed -e "s;%PORT%;$1;g" -e "s;%HOST%;$2;g" -e "s;%DIR%;$3;g" template.txt > script.sh

$ bash ./create.sh 1986 example.com /tmp
$ bash ./script.sh 
Hello, I am a server running from /tmp and listening for connection at example.com on port 1986 and my configuration file is /tmp/server.conf
$ 

回答3:

you can do this directly in bash, you do not even need sed. Write a script like that:

#!/bin/bash

cat <<END
this is a template
with $foo
and $bar
END

then call it like so:

foo=FOO bar=BAR ./template 

回答4:

For simple file generation, basically doing

 . "${config_file}"
 template_str=$(cat "${template_file}")
 eval "echo \"${template_str}\""

would suffice.

Here ${config_file} contains the configuration variables in shell parseable format, and ${template_file} is the template file that looks like shell here document. The first line sources in the file ${config_file}, the second line puts the contents of the file ${template_file} into the shell variable template_str. Finally in the third line we build the shell command echo "${template_str}" (where the double quoted expression "${template_str}" is expanded) and evaluate it.

For an example of the contents of those two files, please refer to https://serverfault.com/a/699377/120756.

There are limitations what you can have in the template file or you need to perform shell escaping. Also if the template file is externally produced, then for security reasons you need to consider implementing a proper filtering prior to execution so that you will not for example lose your files when somebody injects the famous $(rm -rf /) in the template file.

回答5:

Here's the approach that I ended up taking to solve this problem. I found it a little more flexible than some of the above approaches, and it avoids some of the issues with quotes.

fill.sh:

#!/usr/bin/env sh

config="$1"
template="$2"
destination="$3"

cp "$template" "$destination"

while read line; do
    setting="$( echo "$line" | cut -d '=' -f 1 )"
    value="$( echo "$line" | cut -d '=' -f 2- )"

    sed -i -e "s;%${setting}%;${value};g" "$destination"
done < "$config"

template:

Template full of important %THINGS%

"Note that quoted %FIELDS% are handled correctly"

If I need %NEWLINES% then I can add them as well.

config:

THINGS=stuff
FIELDS="values work too!"
NEWLINES="those\\nnifty\\nlinebreaks"

result: Template full of important stuff

"Note that quoted "values work too!" are handled correctly"

If I need those
nifty
linebreaks then I can add them as well.

回答6:

Elegant and short solution in one line with perl

I use perl to replace variables with their values:

export world=World beautiful=wonderful
echo 'I love you, $world! You are $beautiful.' >my_template.txt
perl -pe 's|\$([A-Za-z_]+)|$ENV{$1}|g' my_template.txt

The output: I love you, World! You are wonderful.

my_template.txt can contain variables prefixed with $.

回答7:

You can use python class string.Template

$ echo 'before $X after' > template.txt

$ python  -c 'import string; print(string.Template(open("template.txt").read()).substitute({"X":"A"}))'

before A after

or

$  python  -c 'import string, sys; print(string.Template(open("template.txt").read()).substitute({"X":sys.argv[1]}))' "A"

Here $X is a placeholder in the template and {"X":"A"} is a mapping of the placeholder to a value. In the python code we read the template text from the file, create a template from it, then substitute the placeholder with the command line argument.

Alternatively you can use Ruby's ERB, if Ruby is installed on your machine.

$ echo "before <%= ENV['X'] %> after" > template.txt

$ X=A erb template.txt

before A after

Here <%= ENV['X'] %> is a placeholder. ENV['X'] reads the value from the environment variable. X=A sets the environment variable to the desired value.

回答8:

[Edit] I changed my answer from the original one, that was years ago.

I like the answer from FooF above: https://stackoverflow.com/a/30872526/3538173

Yet, I prefer not to have an intermediary variable to store the whole content of the template file in memory.

. "${config_file}"
eval "echo \"$(cat "${template_file}")\""

Example

Create a template file. Let's call it example.tpl:

Hello, ${NAME}!
Today, the weather is ${WEATHER}. Enjoy!

Create a configuration file to store your variables. Let's call it good.conf:

NAME=John
WEATHER=good

Now, in the script where you want to render the template, you can write this:

#!/usr/bin/env bash

template_file=example.tpl
config_file=good.conf

. "${config_file}"
eval "echo \"$(cat "${template_file}")\""

# Or store the output in a file
eval "echo \"$(cat "${template_file}")\"" > out

You should see this wonderful output :)

Hello, John!
Today, the weather is good. Enjoy!

Caution with eval

When you use eval, if the template file contains some instructions, they will be executed, and it can be dangerous. For example, let's change the example.tpl above with this content:

Hello, ${NAME}!
Today, the weather is ${WEATHER}. Enjoy!

I'm a hacker, hu hu! Look, fool!
$(ls /)

Now, if you render your template file, you will see this:

Hello, John!
Today, the weather is good. Enjoy!

I'm a hacker, hu hu! Look, fool!
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Now edit your file good.conf to have this content:

NAME=$(ls -l /var)
WEATHER=good

and render the template. You should see something like this:

Hello, total 8
drwxr-xr-x.  2 root root    6 Apr 11 04:59 adm
drwxr-xr-x.  5 root root   44 Sep 11 18:04 cache
drwxr-xr-x.  3 root root   34 Sep 11 18:04 db
drwxr-xr-x.  3 root root   18 Sep 11 18:04 empty
drwxr-xr-x.  2 root root    6 Apr 11 04:59 games
drwxr-xr-x.  2 root root    6 Apr 11 04:59 gopher
drwxr-xr-x.  3 root root   18 May  9 13:48 kerberos
drwxr-xr-x. 28 root root 4096 Oct  8 00:30 lib
drwxr-xr-x.  2 root root    6 Apr 11 04:59 local
lrwxrwxrwx.  1 root root   11 Sep 11 18:03 lock -> ../run/lock
drwxr-xr-x.  8 root root 4096 Oct  8 04:55 log
lrwxrwxrwx.  1 root root   10 Sep 11 18:03 mail -> spool/mail
drwxr-xr-x.  2 root root    6 Apr 11 04:59 nis
drwxr-xr-x.  2 root root    6 Apr 11 04:59 opt
drwxr-xr-x.  2 root root    6 Apr 11 04:59 preserve
lrwxrwxrwx.  1 root root    6 Sep 11 18:03 run -> ../run
drwxr-xr-x.  8 root root   87 Sep 11 18:04 spool
drwxrwxrwt.  4 root root  111 Oct  9 09:02 tmp
drwxr-xr-x.  2 root root    6 Apr 11 04:59 yp!
Today, the weather is good. Enjoy!

I'm a hacker, hu hu! Look, fool!
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
swapfile
sys
tmp
usr
var

As you can see, command injection in the configuration file and the template file is possible, and that's why you have to be extra careful:

• be sure of the content of the template file: check that there is NO command injection.
• be sure of the content of the configuration file: check that there is NO command injection as well. If the configuration file comes from someone else, you need to know and trust that person before rendering the template.

Imagine that you are a password-less sudoer, rendering the template file could result in ruining your system with a well-placed rm -rf.

As long as you control the content of these files, it is fine to use this eval templating.

If you have an external (untrusted) incoming configuration file, you should look for templating engine, that will isolate these kind of injection. For example, Jinja2 templating is quite famous in Python.

回答9:

I recently published a bash project that accomplishs just that using a jinja-like template syntax. It's called cookie. Here's a demo: