How do I get both STDOUT and STDERR to go to the t

Question

I have a script which will be run interactively by non-technical users. The script writes status updates to STDOUT so that the user can be sure that the script is running OK.

I want both STDOUT and STDERR redirected to the terminal (so that the user can see that the script is working as well as see if there was a problem). I also want both streams redirected to a log file.

I've seen a bunch of solutions on the net. Some don't work and others are horribly complicated. I've developed a workable solution (which I'll enter as an answer), but it's kludgy.

The perfect solution would be a single line of code that could be incorporated into the beginning of any script that sends both streams to both the terminal and a log file.

EDIT: Redirecting STDERR to STDOUT and piping the result to tee works, but it depends on the users remembering to redirect and pipe the output. I want the logging to be fool-proof and automatic (which is why I'd like to be able to embed the solution into the script itself.)

Answer 1

A year later, here's an old bash script for logging anything. For example,
teelog make ... logs to a generated log name (and see the trick for logging nested makes too.)

#!/bin/bash
me=teelog
Version="2008-10-9 oct denis-bz"

Help() {
cat <<!

    $me anycommand args ...

logs the output of "anycommand ..." as well as displaying it on the screen,
by running
    anycommand args ... 2>&1 | tee `day`-command-args.log

That is, stdout and stderr go to both the screen, and to a log file.
(The Unix "tee" command is named after "T" pipe fittings, 1 in -> 2 out;
see http://en.wikipedia.org/wiki/Tee_(command) ).

The default log file name is made up from "command" and all the "args":
    $me cmd -opt dir/file  logs to `day`-cmd--opt-file.log .
To log to xx.log instead, either export log=xx.log or
    $me log=xx.log cmd ...
If "logdir" is set, logs are put in that directory, which must exist.
An old xx.log is moved to /tmp/\$USER-xx.log .

The log file has a header like
    # from: command args ...
    # run: date pwd etc.
to show what was run; see "From" in this file.

Called as "Log" (ln -s $me Log), Log anycommand ... logs to a file:
    command args ... > `day`-command-args.log
and tees stderr to both the log file and the terminal -- bash only.

Some commands that prompt for input from the console, such as a password,
don't prompt if they "| tee"; you can only type ahead, carefully.

To log all "make" s, including nested ones like
    cd dir1; \$(MAKE)
    cd dir2; \$(MAKE)
    ...
export MAKE="$me make"

!
  # See also: output logging in screen(1).
    exit 1
}


#-------------------------------------------------------------------------------
# bzutil.sh  denisbz may2008 --

day() {  # 30mar, 3mar
    /bin/date +%e%h  |  tr '[A-Z]' '[a-z]'  |  tr -d ' '
}

edate() {  # 19 May 2008 15:56
    echo `/bin/date "+%e %h %Y %H:%M"`
}

From() {  # header  # from: $*  # run: date pwd ...
    case `uname` in Darwin )
        mac=" mac `sw_vers -productVersion`"
    esac
    cut -c -200 <<!
${comment-#} from: $@
${comment-#} run: `edate`  in $PWD `uname -n` $mac `arch` 

!
    # mac $PWD is pwd -L not -P real
}

    # log name: day-args*.log, change this if you like --
logfilename() {
    log=`day`
    [[ $1 == "sudo" ]]  &&  shift
    for arg
    do
        log="$log-${arg##*/}"  # basename
        (( ${#log} >= 100 ))  &&  break  # max len 100
    done
            # no blanks etc in logfilename please, tr them to "-"
    echo $logdir/` echo "$log".log  |  tr -C '.:+=[:alnum:]_\n' - `
}

#-------------------------------------------------------------------------------
case "$1" in
-v* | --v* )
    echo "$0 version: $Version"
    exit 1 ;;
"" | -* )
    Help
esac

    # scan log= etc --
while [[ $1 == [a-zA-Z_]*=* ]]; do
    export "$1"
    shift
done

: ${logdir=.}
[[ -w $logdir ]] || {
    echo >&2 "error: $me: can't write in logdir $logdir"
    exit 1
    }
: ${log=` logfilename "$@" `}
[[ -f $log ]]  &&
    /bin/mv "$log" "/tmp/$USER-${log##*/}"


case ${0##*/} in  # basename
log | Log )  # both to log, stderr to caller's stderr too --
{
    From "$@"
    "$@"
} > $log  2> >(tee /dev/stderr)  # bash only
    # see http://wooledge.org:8000/BashFAQ 47, stderr to a pipe
;;

* )
#-------------------------------------------------------------------------------
{
    From "$@"  # header: from ... date pwd etc.

    "$@"  2>&1  # run the cmd with stderr and stdout both to the log

} | tee $log
    # mac tee buffers stdout ?

esac

Answer 2

Use "tee" to redirect to a file and the screen. Depending on the shell you use, you first have to redirect stderr to stdout using

./a.out 2>&1 | tee output

or

./a.out |& tee output

In csh, there is a built-in command called "script" that will capture everything that goes to the screen to a file. You start it by typing "script", then doing whatever it is you want to capture, then hit control-D to close the script file. I don't know of an equivalent for sh/bash/ksh.

Also, since you have indicated that these are your own sh scripts that you can modify, you can do the redirection internally by surrounding the whole script with braces or brackets, like

  #!/bin/sh
  {
    ... whatever you had in your script before
  } 2>&1 | tee output.file

Answer 3

I created a script called "RunScript.sh". The contents of this script is:

${APP_HOME}/${1}.sh ${2} ${3} ${4} ${5} ${6} 2>&1 | tee -a ${APP_HOME}/${1}.log

I call it like this:

./RunScript.sh ScriptToRun Param1 Param2 Param3 ...

This works, but it requires the application's scripts to be run via an external script. It's a bit kludgy.

Answer 4

EDIT: I see I got derailed and ended up answering a different question from the one asked. The answer to the real question is at the bottom of Paul Tomblin's answer. (If you want to enhance that solution to redirect stdout and stderr separately for some reason, you could use the technique I describe here.)

---

I've been wanting an answer that preserves the distinction between stdout and stderr. Unfortunately all of the answers given so far that preserve that distinction are race-prone: they risk programs seeing incomplete input, as I pointed out in comments.

I think I finally found an answer that preserves the distinction, is not race prone, and isn't terribly fiddly either.

First building block: to swap stdout and stderr:

my_command 3>&1 1>&2 2>&3-

Second building block: if we wanted to filter (e.g. tee) only stderr, we could accomplish that by swapping stdout&stderr, filtering, and then swapping back:

{ my_command 3>&1 1>&2 2>&3- | stderr_filter;} 3>&1 1>&2 2>&3-

Now the rest is easy: we can add a stdout filter, either at the beginning:

{ { my_command | stdout_filter;} 3>&1 1>&2 2>&3- | stderr_filter;} 3>&1 1>&2 2>&3-

or at the end:

{ my_command 3>&1 1>&2 2>&3- | stderr_filter;} 3>&1 1>&2 2>&3- | stdout_filter

To convince myself that both of the above commands work, I used the following:

alias my_command='{ echo "to stdout"; echo "to stderr" >&2;}'
alias stdout_filter='{ sleep 1; sed -u "s/^/teed stdout: /" | tee stdout.txt;}'
alias stderr_filter='{ sleep 2; sed -u "s/^/teed stderr: /" | tee stderr.txt;}'

Output is:

...(1 second pause)...
teed stdout: to stdout
...(another 1 second pause)...
teed stderr: to stderr

and my prompt comes back immediately after the "teed stderr: to stderr", as expected.

Footnote about zsh:

The above solution works in bash (and maybe some other shells, I'm not sure), but it doesn't work in zsh. There are two reasons it fails in zsh:

1. the syntax 2>&3- isn't understood by zsh; that has to be rewritten as 2>&3 3>&-
2. in zsh (unlike other shells), if you redirect a file descriptor that's already open, in some cases (I don't completely understand how it decides) it does a built-in tee-like behavior instead. To avoid this, you have to close each fd prior to redirecting it.

So, for example, my second solution has to be rewritten for zsh as {my_command 3>&1 1>&- 1>&2 2>&- 2>&3 3>&- | stderr_filter;} 3>&1 1>&- 1>&2 2>&- 2>&3 3>&- | stdout_filter (which works in bash too, but is awfully verbose).

On the other hand, you can take advantage of zsh's mysterious built-in implicit teeing to get a much shorter solution for zsh, which doesn't run tee at all:

my_command >&1 >stdout.txt 2>&2 2>stderr.txt

(I wouldn't have guessed from the docs I found that the >&1 and 2>&2 are the thing that trigger zsh's implicit teeing; I found that out by trial-and-error.)

Answer 5

Approaching half a decade later...

I believe this is the "perfect solution" sought by the OP.

Here's a one liner you can add to the top of your Bash script:

exec > >(tee -a $HOME/logfile) 2>&1

Here's a small script demonstrating its use:

#!/usr/bin/env bash

exec > >(tee -a $HOME/logfile) 2>&1

# Test redirection of STDOUT
echo test_stdout

# Test redirection of STDERR
ls test_stderr___this_file_does_not_exist

(Note: This only works with Bash. It will not work with /bin/sh.)

Adapted from here; the original did not, from what I can tell, catch STDERR in the logfile. Fixed with a note from here.

Answer 6

Use the script command in your script (man 1 script)

Create a wrapper shellscript (2 lines) that sets up script() and then calls exit.

Part 1: wrap.sh

#!/bin/sh
script -c './realscript.sh'
exit

Part 2: realscript.sh

#!/bin/sh
echo 'Output'

Result:

~: sh wrap.sh 
Script started, file is typescript
Output
Script done, file is typescript
~: cat typescript 
Script started on fr. 12. des. 2008 kl. 18.07 +0100
Output

Script done on fr. 12. des. 2008 kl. 18.07 +0100
~: