bash: start multiple chained commands in backgroun

问题:

I'm trying to run some commands in paralel, in background, using bash. Here's what I'm trying to do:

forloop {
  //this part is actually written in perl
  //call command sequence
  print `touch .file1.lock; cp bigfile1 /destination; rm .file1.lock;`;
}

The part between backticks (``) spawns a new shell and executes the commands in succession. The thing is, control to the original program returns only after the last command has been executed. I would like to execute the whole statement in background (I'm not expecting any output/return values) and I would like the loop to continue running.

The calling program (the one that has the loop) would not end until all the spawned shells finish.

I could use threads in perl to spawn different threads which call different shells, but it seems an overkill...

Can I start a shell, give it a set of commands and tell it to go to the background?

回答1:

I haven't tested this but how about

print `(touch .file1.lock; cp bigfile1 /destination; rm .file1.lock;) &`;

The parentheses mean execute in a subshell but that shouldn't hurt.

回答2:

Another way is to use the following syntax:

{ command1; command2; command3; } &
wait

Note that the & goes at the end of the command group, not after each command. The semicolon after the final command is necessary, as are the space after the first bracket and before the final bracket. The wait at the end ensures that the parent process is not killed before the spawned child process (the command group) ends.

You can also do fancy stuff like redirecting stderr and stdout:

{ command1; command2; command3; } 2>&2 1>&1 &

Your example would look like:

forloop() {
    { touch .file1.lock; cp bigfile1 /destination; rm .file1.lock; } &
}
# ... do some other concurrent stuff
wait # wait for childs to end

回答3:

Thanks Hugh, that did it:

adrianp@frost:~$ (echo "started"; sleep 15; echo "stopped")
started
stopped
adrianp@frost:~$ (echo "started"; sleep 15; echo "stopped") &
started
[1] 7101
adrianp@frost:~$ stopped

[1]+  Done                    ( echo "started"; sleep 15; echo "stopped" )
adrianp@frost:~$ 

The other ideas don't work because they start each command in the background, and not the command sequence (which is important in my case!).

Thank you again!

回答4:

for command in $commands
do
    "$command" &
done
wait

The ampersand at the end of the command runs it in the background, and the wait waits until the background task is completed.

回答5:

GavinCattell got the closest (for bash, IMO), but as Mad_Ady pointed out, it would not handle the "lock" files. This should:

If there are other jobs pending, the wait will wait for those, too. If you need to wait for only the copies, you can accumulate those PIDs and wait for only those. If not, you could delete the 3 lines with "pids" but it's more general.

In addition, I added checking to avoid the copy altogether:

pids=
for file in bigfile*
do
    # Skip if file is not newer...
    targ=/destination/$(basename "${file}")
    [ "$targ" -nt "$file" ] && continue

    # Use a lock file:  ".fileN.lock" for each "bigfileN"
    lock=".${file##*/big}.lock"
    ( touch $lock; cp "$file" "$targ"; rm $lock ) &
    pids="$pids $!"
done
wait $pids

Incidentally, it looks like you're copying new files to an FTP repository (or similar). If so, you could consider a copy/rename strategy instead of the lock files (but that's another topic).

回答6:

The facility in bash that you're looking for is called Compound Commands. See the man page for more info:

Compound Commands A compound command is one of the following:

   (list) list  is  executed  in a subshell environment (see COMMAND EXECUTION ENVIRONMENT below).  Variable assignments and
          builtin commands that affect the shell's environment do not remain in effect after  the  command  completes.   The
          return status is the exit status of list.

   { list; }
          list  is  simply  executed in the current shell environment.  list must be terminated with a newline or semicolon.
          This is known as a group command.  The return status is the exit status of list.  Note that unlike the metacharac‐
          ters  (  and  ),  {  and  } are reserved words and must occur where a reserved word is permitted to be recognized.
          Since they do not cause a word break, they must be separated from list by whitespace or another shell  metacharac‐
          ter.

There are others, but these are probably the 2 most common types. The first, the parens, will run a list of command in series in a subshell, while the second, the curly braces, will a list of commands in series in the current shell.

parens

% ( date; sleep 5; date; )
Sat Jan 26 06:52:46 EST 2013
Sat Jan 26 06:52:51 EST 2013

curly braces

% { date; sleep 5; date; }
Sat Jan 26 06:52:13 EST 2013
Sat Jan 26 06:52:18 EST 2013

回答7:

Run the command by using an at job:

# date
# jue sep 13 12:43:21 CEST 2012
# at 12:45
warning: commands will be executed using /bin/sh
at> command1
at> command2
at> ...
at> CTRL-d
at> <EOT>
job 20 at Thu Sep 13 12:45:00 2012

The result will be sent to your account by mail.

回答8:

I stumbled upon this thread here and decided to put together a code snippet to spawn chained statements as background jobs. I tested this on BASH for Linux, KSH for IBM AIX and Busybox's ASH for Android, so I think it's safe to say it works on any Bourne-like shell.

processes=0;
for X in `seq 0 10`; do
   let processes+=1;
   { { echo Job $processes; sleep 3; echo End of job $processes; } & };
   if [[ $processes -eq 5 ]]; then
      wait;
      processes=0;
   fi;
done;

This code runs a number of background jobs up to a certain limit of concurrent jobs. You can use this, for example, to recompress a lot of gzipped files with xz without having a huge bunch of xz processes eat your entire memory and make your computer throw up: in this case, you use * as the for's list and the batch job would be gzip -cd "$X" | xz -9c > "${X%.gz}.xz".

回答9:

run the commands in a subshell:

(command1 ; command2 ; command3) &

回答10:

Try to put commands in curly braces with &s, like this:

{command1 & ; command2 & ; command3 & ; }

This does not create a sub-shell, but executes the group of commands in the background.

HTH

回答11:

I don't know why nobody replied with the proper solution:

my @children;
for (...) {
    ...
    my $child = fork;
    exec "touch .file1.lock; cp bigfile1 /destination; rm .file1.lock;" if $child == 0;
    push @children, $child;
}
# and if you want to wait for them to finish,
waitpid($_) for @children;

This causes Perl to spawn children to run each command, and allows you to wait for all the children to complete before proceeding.

By the way,

print `some command`

and

system "some command"

output the same contents to stdout, but the first has a higher overhead, as Perl has to capture all of "some command"'s output

回答12:

Forking in a for loop:

for i in x; do ((a; b; c;)&); done

Example:

for i in 500 300 100; do ((printf "Start $i: "; date; dd if=/dev/zero of=testfile_$i bs=1m count=$i 2>/dev/null; printf "End $i: "; date;)&) && sleep 1; done

回答13:

Just in case that someone is still interested, you can do it without calling a subshell like this:

print `touch .file1.lock && cp bigfile1 /destination && rm .file1.lock &`;

回答14:

You can use GNU parallel command to run jobs in parallel. It is more safe are faster.

My guess is that you are trying to copy multiple large files from source to destination. And for that you can do that in parallel with below statement.

$ ls *|parallel -kj0 --eta 'cp {} /tmp/destination'

As we have used -j0 option, all the files will be copied in parallel. In case if you need to reduce the number of parallel process then you can use -j<n> where <n> is the number of parallel process to be executed.

Parallel will also collect the output of the process and report it in a sequential manner (with -k option) which other job control mechanism cannot do.

--eta option will give you a details statistics of the process that is going on. So we can know how may of the process have been completed and how long will it take to get finished.