Pipe only STDERR through a filter

Question

Is there any way, in bash, to pipe STDERR through a filter before unifying it with STDOUT? That is, I want

STDOUT ────────────────┐
                       ├─────> terminal/file/whatever
STDERR ── [ filter ] ──┘

rather than

STDOUT ────┐
           ├────[ filter ]───> terminal/file/whatever
STDERR ────┘

Answer 1

A naive use of process substitution seems to allow filtering of stderr separately from stdout:

:; ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 )
out
e:err

Note that stderr comes out on stderr and stdout on stdout, which we can see by wrapping the whole thing in another subshell and redirecting to files o and e

( ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 ) ) 1>o 2>e

Answer 2

Take a look at named pipes:

$ mkfifo err
$ cmd1 2>err |cat - err |cmd2

Answer 3

I find the use of bash process substitution easier to remember and use as it reflects the original intention almost verbatim. For example:

$ cat ./p
echo stdout
echo stderr >&2
$ ./p 2> >(sed -e 's/s/S/') | sed 's/t/T/'
sTdout
STderr

uses the first sed command as a filter on stderr only and the second sed command to modify the joined output.

Note that the white space after 2> is mandatory for the command to be parsed correctly.

Answer 4

The last part of this page of the Advanced Bash Scripting Guide is "redirecting only stderr to a pipe".

# Redirecting only stderr to a pipe.

exec 3>&1                              # Save current "value" of stdout.
ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
#              ^^^^   ^^^^
exec 3>&-                              # Now close it for the remainder of the script.

# Thanks, S.C.

This may be what you want. If not, some other part of the ABSG should be able to help you, it is excellent.

Answer 5

TL;DR:

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

This will work in both bash and zsh. Bash is pretty much ubiquitous these days, however, if you really do need a (really gnarly) solution for POSIX sh, then see here.

---

Explanation

By far, the easiest way to do this is to redirect STDERR via process substitution:

Process substitution allows a process’s input or output to be referred to using a filename. It takes the form of

>(list)

The process list is run asynchronously, and its input or output appears as a filename.

So what you get with process substituion is a filename.

Just like you could do:

$ cmd 2> filename

you can do

$ cmd 2> >(filter >&2)

The >&2 redirect's filter's STDOUT back to the original STDERR.

Answer 6

TL;DR: (bash and zsh)

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

---

Many answers on the StackExchange network have the form:

cat /non-existant 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'

This has a built-in assumption: that file descriptor 3 isn't being used for something else.

Instead, use a named file descriptor, and {ba,z}sh will allocate the next available file descriptor >= 10:

cat /non-existant {tmp}>&1 1>&2 2>&$tmp {tmp}>&- | sed 's/e/E/g'

Note that named file descriptors aren't supported by POSIX sh.

The other issue with the above is that the command cannot be piped to further commands without again swapping STDOUT and STDERR back to their original values.

To allow onward piping in POSIX sh, (and still assuming FD 3 is not it use) it gets complicated:

(cmd 2>&1 >&3 3>&- | stderr-filter >&2 3>&-) 3>&1

So, Given the assumption and gnarly syntax of this, you're likely to be better off using the simpler bash/zsh syntax shown in the TL;DR above, and explained here.