Synchronize shell script execution

Question

A modified version of a shell script converts an audio file from FLAC to MP3 format. The computer has a quad-core CPU. The script is run using:

./flac2mp3.sh $(find flac -type f)

This converts the FLAC files in the flac directory (no spaces in file names) to MP3 files in the mp3 directory (at the same level as flac). If the destination MP3 file already exists, the script skips the file.

The problem is that sometimes two instances of the script check for the existence of the same MP3 file at nearly the same time, resulting in mangled MP3 files.

How would you run the script multiple times (i.e., once per core), without having to specify a different file set on each command-line, and without overwriting work?

Update - Minimal Race Condition

The script uses the following locking mechanism:

  # Convert FLAC to MP3 using tags from flac file.
  #
  if [ ! -e $FLAC.lock ]; then
    touch $FLAC.lock
    flac -dc "$FLAC" | lame${lame_opts} \
      --tt "$TITLE" \
      --tn "$TRACKNUMBER" \
      --tg "$GENRE" \
      --ty "$DATE" \
      --ta "$ARTIST" \
      --tl "$ALBUM" \
      --add-id3v2 \
      - "$MP3"
    rm $FLAC.lock
  fi;

However, this still leaves a race condition.

Answer 1

You could implement locking of FLAC files that it's working on. Something like:

if (not flac locked)
  lock flac
  do work
else
  continue to next flac

Answer 2

How about writing a Makefile?

ALL_FLAC=$(wildcard *.flac)
ALL_MP3=$(patsubst %.flac, %.mp3, $(ALL_FLAC)
all: $(ALL_MP3)
%.mp3: %.flac
        $(FLAC) ...

Then do

$ make -j4 all

Answer 3

Send output to a temporary file with a unique name, then rename the file to the desired name.

flac -dc "$FLAC" | lame${lame_opts} \
      --tt "$TITLE" \
      --tn "$TRACKNUMBER" \
      --tg "$GENRE" \
      --ty "$DATE" \
      --ta "$ARTIST" \
      --tl "$ALBUM" \
      --add-id3v2 \
      - "$MP3.$$"
mv "$MP3.$$" "$MP3"

If a race condition leaks through your file locking system every once in a while, the final output will still be the result of one process.

Answer 4

The "lockfile" command provides what you're trying to do for shell scripts without the race condition. The command was written by the procmail folks specifically for this sort of purpose and is available on most BSD/Linux systems (as procmail is available for most environments).

Your test becomes something like this:

lockfile -r 3 $FLAC.lock
if test $? -eq 0 ; then
  flac -dc "$FLAC" | lame${lame_opts} \
    --tt "$TITLE" \
    --tn "$TRACKNUMBER" \
    --tg "$GENRE" \
    --ty "$DATE" \
    --ta "$ARTIST" \
    --tl "$ALBUM" \
    --add-id3v2 \
    - "$MP3"
fi
rm -f $FLAC.lock

Alternatively, you could make lockfile keep retrying indefinitely so you don't need to test the return code, and instead can test for the output file for determining whether to run flac.

Answer 5

If you don't have lockfile and cannot install it (in any of its versions - there are several implementations) a robust and portable atomic mutex is mkdir.

If the directory you attempt to create already exists, mkdir will fail, so you can check for that; when creation succeeds, you have a guarantee that no other cooperating process is in the critical section at the same time as your code.

if mkdir "$FLAC.lockdir"; then
    # you now have the exclusive lock
    : critical section
    : code goes here
    rmdir "$FLAC.lockdir"
else
    : nothing? to skip this file
    # or maybe sleep 1 and loop back and try again
fi

For completeness, maybe also look for flock if you are on a set of platforms where that is reliably made available and need a performant alternative to lockfile.

Answer 6

In bash it's possible to set noclobber option to avoid file overwriting.

help set | egrep 'noclobber|-C'