linux zip and exclude dir via bash/shell script

Question

I am trying to write a bash/shell script to zip up a specific folder and ignore certain sub-dirs in that folder.

This is the folder I am trying to zip "sync_test5":

My bash script generates an ignore list (based on) and calls the zip function like this:

#!/bin/bash

SYNC_WEB_ROOT_BASE_DIR="/home/www-data/public_html"
SYNC_WEB_ROOT_BACKUP_DIR="sync_test5"
SYNC_WEB_ROOT_IGNORE_DIR="dir_to_ignore dir2_to_ignore"

ignorelist=""
if [ "$SYNC_WEB_ROOT_IGNORE_DIR" != "" ];
then
    for ignoredir in $SYNC_WEB_ROOT_IGNORE_DIR
    do
        ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/**\*"
    done
fi

FILE="$SYNC_BACKUP_DIR/$DATETIMENOW.website.zip"
cd $SYNC_WEB_ROOT_BASE_DIR;
zip -r $FILE $SYNC_WEB_ROOT_BACKUP_DIR -x $ignorelist >/dev/null
echo "Done"

Now this script runs without error, however it is not ignoring/excluding the dirs I've specified.

So, I had the shell script output the command it tried to run, which was:

zip -r 12-08-2014_072810.website.zip sync_test5 -x  sync_test5/dir_to_ignore/**\* sync_test5/dir2_to_ignore/**\*

Now If I run the above command directly in putty like this, it works:

So, why doesn't my shell script exclude working as intended? the command that is being executed is identical (in shell and putty directly).

Answer 1

This is what I ended up with:

#!/bin/bash
# This script zips a directory, excluding specified files, types and subdirectories.
#  while zipping the directory it excludes hidden directories and certain file types

[[ "`/usr/bin/tty`" == "not a tty" ]] && . ~/.bash_profile

DIRECTORY=$(cd `dirname $0` && pwd)

if [[ -z $1 ]]; then
  echo "Usage: managed_directory_compressor /your-directory/ zip-file-name"
else
  DIRECTORY_TO_COMPRESS=${1%/}
  ZIPPED_FILE="$2.zip"

  COMPRESS_IGNORE_FILE=("\.git" "*.zip" "*.csv" "*.json" "gulpfile.js" "*.rb" "*.bak" "*.swp" "*.back" "*.merge" "*.txt" "*.sh" "bower_components" "node_modules") 
  COMPRESS_IGNORE_DIR=("bower_components" "node_modules")

  IGNORE_LIST=("*/\.*" "\.* "\/\.*"")
  if [[ -n $COMPRESS_IGNORE_FILE ]]; then
      for IGNORE_FILES in "${COMPRESS_IGNORE_FILE[@]}"; do
          IGNORE_LIST+=("$DIRECTORY_TO_COMPRESS/$IGNORE_FILES/*")  
      done
      for IGNORE_DIR in "${COMPRESS_IGNORE_DIR[@]}"; do
          IGNORE_LIST+=("$DIRECTORY_TO_COMPRESS/$IGNORE_DIR/")
      done
  fi

  zip -r "$ZIPPED_FILE" "$DIRECTORY_TO_COMPRESS" -x "${IGNORE_LIST[@]}" # >/dev/null
  # echo zip -r "$ZIPPED_FILE" "$DIRECTORY_TO_COMPRESS" -x "${IGNORE_LIST[@]}" # >/dev/null
  echo $DIRECTORY_TO_COMPRESS "compressed as" $ZIPPED_FILE.
fi

Answer 2

After a few trial and error, I have managed to fix this problem by changing this line:

ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/**\*"

to:

ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/***"

Not sure why this worked, but it does :)

Answer 3

Because backslash quotings in a variable after word splitting are not evaluated.

If you have a='123\4', echo $a would give

123\4

But if you do it directly like echo 123\4, you'd get

1234

Clearly the arguments you pass with the variable and without the variables are different.

You probably just meant to not quote your argument with backslash:

ignorelist="$ignorelist $SYNC_WEB_ROOT_BACKUP_DIR/$ignoredir/***"

Btw, what actual works is a non-evaluated glob pattern:

zip -r 12-08-2014_072810.website.zip sync_test5 -x 'sync_test5/dir_to_ignore/***' 'sync_test5/dir2_to_ignore/***'

You can verify this with

echo zip -r 12-08-2014_072810.website.zip sync_test5 -x  sync_test5/dir_to_ignore/**\* sync_test5/dir2_to_ignore/**\*

And this is my suggestion:

#!/bin/bash

SYNC_WEB_ROOT_BASE_DIR="/home/www-data/public_html"
SYNC_WEB_ROOT_BACKUP_DIR="sync_test5"
SYNC_WEB_ROOT_IGNORE_DIR=("dir_to_ignore" "dir2_to_ignore")

IGNORE_LIST=()
if [[ -n $SYNC_WEB_ROOT_IGNORE_DIR ]]; then
    for IGNORE_DIR in "${SYNC_WEB_ROOT_IGNORE_DIR[@]}"; do
        IGNORE_LIST+=("$SYNC_WEB_ROOT_BACKUP_DIR/$IGNORE_DIR/***")  ## "$SYNC_WEB_ROOT_BACKUP_DIR/$IGNORE_DIR/*"  perhaps is enough?
    done
fi

FILE="$SYNC_BACKUP_DIR/$DATETIMENOW.website.zip"  ## Where is $SYNC_BACKUP_DIR set?
cd "$SYNC_WEB_ROOT_BASE_DIR";
zip -r "$FILE" "$SYNC_WEB_ROOT_BACKUP_DIR" -x "${IGNORE_LIST[@]}" >/dev/null
echo "Done"