How do I use shell variables in an awk script?

Question

I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ".

First, I tried with a shell script:

$ v=123test
$ echo $v
123test
$ echo "$v"
123test

Then tried awk:

$ awk 'BEGIN{print "'$v'"}'
$ 123test
$ awk 'BEGIN{print '"$v"'}'
$ 123

Why is the difference?

Lastly I tried this:

$ awk 'BEGIN{print " '$v' "}'
$  123test
$ awk 'BEGIN{print ' "$v" '}'
awk: cmd. line:1: BEGIN{print
awk: cmd. line:1:             ^ unexpected newline or end of string 

I'm confused about this.

Answer 1

Getting shell variables into awk may be done in several ways. Some are better than others.

---

This is the best way to do it. It uses the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)

variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two

This should be compatible with most awk and variable is available in the BEGIN block as well:

Multiple variables

awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'

---

Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:

variable="line one\nline two"
echo "input data" | awk '{print var}' var="$variable"
or
awk '{print var}' var="$variable" file

This also works with multiple variables awk '{print a,b,$0}' a="$var1" b="$var2" file

---

Variable can also be added to awk using here string

awk '{print $0}' <<< "$variable"
test

This is the same as:

echo "$variable" | awk '{print $0}'

PS, this threats the variable as a file input

---

As TrueY write, you can use the ENVIRON to print Environmental Variables Setting a variable before running AWK, you can print it out like this:

X=MyVar awk 'BEGIN{print ENVIRON["X"],ENVIRON["SHELL"]}'
MyVar /bin/bash

Edit: As "that other guy" write, this does not handle backslash. Not recommended.

---

You can use a variable within the awk code, but it's messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.

If you want to make an awk that changes dynamically with use of variables, you can do it this way, bot DO NOT use it for normal variables.

variable="line one\nline two"
awk 'BEGIN {print "'"$variable"'"}'
line one
line two

Here is an example of code injection:

variable='line one\nline two" ; for (i=1;i<=1000;++i) print i"'
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
1
2
3
.
.
1000

You can add lots of commands to awk this way. Even make it crash with non valid commands.

---

It's always good to double quote variable "$variable"
If not, multiple lines will be added as a long single line.

Example:

var="Line one
This is line two"

echo $var
Line one This is line two

echo "$var"
Line one
This is line two

Other errors you can get without double quote:

variable="line one\nline two"
awk -v var=$variable 'BEGIN {print var}'
awk: cmd. line:1: one\nline
awk: cmd. line:1:    ^ backslash not last character on line
awk: cmd. line:1: one\nline
awk: cmd. line:1:    ^ syntax error

And with single quote, it does not expand the value of the variable:

awk -v var='$variable' 'BEGIN {print var}'
$variable

Answer 2

You could pass in the command-line option -v with a variable name (v) and a value (=) of the environment variable ("${v}"):

% awk -vv="${v}" 'BEGIN { print v }'
123test

Or to make it clearer (with far fewer vs):

% environment_variable=123test
% awk -vawk_variable="${environment_variable}" 'BEGIN { print awk_variable }'
123test

Answer 3

You can utilize ARGV:

v=123test
awk 'BEGIN {print ARGV[1]}' "$v"

Note that if you are going to continue into the body, you will need to adjust ARGC:

awk 'BEGIN {ARGC--} {print ARGV[2], $0}' file "$v"

Answer 4

I had to insert date at the beginning of the lines of a log file and it's done like below:

DATE=$(date +"%Y-%m-%d")
awk '{ print "'"$DATE"'", $0; }' /path_to_log_file/log_file.log

It can be redirect to another file to save

Answer 5

for i in chr{1..22} chrX chrY
do
awk -v chr="$i" '$1==chr' ../snp150.hg19.txt >> $chr.vcf.bed
echo $i
done

Answer 6

Use either of these depending how you want backslashes in the shell variables handled (avar is an awk variable, svar is a shell variable):

awk -v avar="$svar" '... avar ...' file
awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}... avar ...' "$svar" file

See http://cfajohnson.com/shell/cus-faq-2.html#Q24 for details and other options. The first method above is almost always your best option and has the most obvious semantics.