Awk, pipe and tail -f giving unexpected behavior [

问题:

This question already has an answer here:

• Piping tail output though grep twice 2 answers

Here is my sample log file.http://pastebin.com/DwWeFhJk.

When I am doing

tail -f log | awk '{if (NF>3) {print $1}; }'

the result I am getting is correct

64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10

But when I am doing:

tail -f log |
awk '{if (NF>3) {print $1}; }' |
awk '{print $1}'

I am not getting any output. Even no output in case of

tail -f log | awk '{if (NF>3) {print $1}; }' | grep "64"

I am not getting the reason why the output of the first awk is not getting passed as the input of the second awk/grep after the pipe.

回答1:

When the output of the first awk is going to the terminal, the output is line-buffered, so each line is printed as it is produced. When the output is going to the second awk or the grep, it is fully buffered. The output won't be sent until the buffer is full. When enough extra records are appended to the log, then the second awk will a buffer full of data to process. Until then, nothing will happen.

回答2:

You start the command with tail -f, that keeps the output open and therefore does not send a needed newline to the other commands.

This works perfectly fine:

cat log | awk '{if (NF>3) {print $1}; }' | grep 64

So, the problem is buffering. The middle awk is doing normal buffering instead of interactive buffering. This works (non-portably) with mawk:

tail -f log | mawk -W interactive '{if (NF>3) {print $1}; }' | awk '{print}'

You could read GNU description of the issue.

In any case, just check that the awk used in the middle could be told to buffer interactively.

Added:

The command system("") seems to unblock the buffering. It is POSIX, but does not work with mawk.

tail -f log | awk '{if (NF>3) {print $1}; system("")}' | awk '{print}'

回答3:

search for "parallel --pipe" in the link to avoid the buffering

https://www.gnu.org/software/parallel/parallel_tutorial.html