Buffering problem when piping output between CLI p

Question

I'm trying to tail apache error logs through a few filters.

This works perfectly:

tail -fn0 /var/log/apache2/error.log | egrep -v "PHP Notice|File does not exist"

but there are some literal "\n" in the output which I want to replace with an actual new line so I pipe into perl:

tail -fn0 /var/log/apache2/error.log | egrep -v "PHP Notice|File does not exist" | perl -ne 's/\\n/\n/g; print"$_"'

This seems to have some caching issue (first page hit produces nothing, second page hit and two loads of debugging info comes out), It also seems a bit tempramental.

So I tried sed:

tail -fn0 /var/log/apache2/error.log | egrep -v "PHP Notice|File does not exist" | sed 's/\\n/\n/g'

which seems to suffer the same problem.

Answer 1

Correct, when you use most programs to file or pipe they buffer output. You can control this in some cases: the GNU grep family accepts the --line-buffered option, specifically for use in pipelines like this. Also, in Perl you can use $| = 1; for the same effect. (sed doesn't have any such option that I'm aware of.)

It's the stuff at the beginning or middle of the pipeline that will be buffering, not the end (which is talking to your terminal so it will be line buffered) so you want to use egrep --line-buffered.

Answer 2

Looks like you can use -u for sed as in:

tail -f myLog | sed -u "s/\(joelog\)/^[[46;1m\1^[[0m/g" | sed -u 's/\\n/\n/g'

which tails the log, highlights 'joelog', and then adds linebreaks where there are '\n'

source: http://www-01.ibm.com/support/docview.wss?uid=isg1IZ42070