Linux: Block until a string is matched in a file &

Question

Is there some one-line way in bash/GNU tools to block until there's a string matched in a file? Ideally, with timeout. I want to avoid multi-line loop.

Update: Seems like I should have emphasize that I want the process to end when the string is matched.

Answer 1

I had a similar requirement and came up with the following.

The one-liner you are after is the line which starts with "timeout ...." and the rest of the code is the prep work needed to provide the one-liner with the information it will need and to clean up afterwards.

##
## Start up the process whose log file we want to monitor for a specific pattern.
##
touch file_to_log_nohup_output.log
nohup "some_command" "some_args" >> file_to_log_nohup_output.log 2>&1 &
my_cmd_pid=$!


## Specify what our required timeout / pattern and log file to monitor is
my_timeout=10m
my_logfile="/path/to/some_command's/log/file.txt"
my_pattern="Started all modules."


## How does this work?
## - In a bash sub shell, started in the background, we sleep for a second and
##   then execute tail to monitor the application's log file.
## - Via the arguments passed to it, tail has been configured to exit if the
##   process whose log file it is monitoring dies.
## - The above sub shell, is executed within another bash sub shell in which
##   we identify the process id of the above sub shell and echo it to stdout.
## - Lastly, in that sub shell we wait for the sub shell with tail running in
##   it as a child process, to terminate and if it does terminate, we redirect
##   any output from its stderr stream to /dev/null.
## - The stdout output of the above sub shell is piped into another sub shell
##   in which we setup a trap to watch for an EXIT event, use head -1 to read
##   the process id of the tail sub shell and finally start a grep process
##   to grep the stdout for the requested pattern. Grep will quit on the first
##   match found. The EXIT trap will kill the process of the tail sub shell
##   if the sub shell running grep quits.
##
## All of this is needed to tidy up the monitoring child processes for
## tail'ing + grep'ing the application log file.
##
## Logic of implementing the above sourced from: http://superuser.com/a/1052328


timeout ${my_timeout} bash -c '((sleep 1; exec tail -q -n 0 --pid=$0 -F "$1" 2> /dev/null) & echo $! ; wait $! 2>/dev/null ) | (trap "kill \${my_tail_pid} 2>/dev/null" EXIT; my_tail_pid="`head -1`"; grep -q "$2")' "${my_cmd_pid}" "${my_logfile}" "${my_pattern}" 2>/dev/null &


##
## We trap SIGINT (i.e. when someone presses ctrl+c) to clean up child processes.
##
trap 'echo "Interrupt signal caught. Cleaning up child processes: [${my_timeout_pid} ${my_cmd_pid}]." >> "file_to_log_nohup_output.log"; kill ${my_timeout_pid} ${my_cmd_pid} 2> /dev/null' SIGINT
wait ${my_timeout_pid}
my_retval=$?
trap - SIGINT


## If the time out expires, then 'timeout' will exit with status 124 otherwise
## it exits with the status of the executed command (which is grep in this
## case).
if [ ${my_retval} -eq 124 ]; then
    echo "Waited for [${my_timeout}] and the [${my_pattern}] pattern was not encountered in application's log file."
    exit 1
else
    if [ ${my_retval} -ne 0 ]; then
        echo "An issue occurred whilst starting process. Check log files:"
        echo "  * nohup output log file: [file_to_log_nohup_output.log]"
        echo "  * application log file: [${my_logfile}]"
        echo "  * application's console log file (if applicable)"
        exit 1
    else
        info_msg "Success! Pattern was found."
        exit 0
    fi
fi

I've implemented the above into a stand alone script which can be used to run a command wait for its log file to have the required pattern, with a timeout.

Available here: run_and_wait.sh

Answer 2

wait for file to appear

while [ ! -f /path/to/the.file ] 
do sleep 2; done

wait for string to apper in file

while ! grep "the line you're searching for" /path/to/the.file  
do sleep 10; done

https://superuser.com/a/743693/129669

Answer 3

$ tail -f path | sed /pattern/q

or, if you want to suppress the output of non-matching lines:

$ tail -f path | sed -n '/pattern/{p; q;}'

A simple-minded way to add a timeout is to do:

$ cmd& sleep 10; kill $! 2> /dev/null

(Suppress the errors from the kill so that if the process terminates before the time expires, you don't get the "No such process" warning). Note that this is not at all robust, since it is possible that cmd will terminate and the pid count will wrap around and some other command will have that pid by the time the timer expires.

Answer 4

Thanks both for answers, but the important part was that the process blocks until found, then ends. I found this:

grep -q 'PATTERN' <(tail -f file.log)

-q is not much portable, but I will only use Red Hat Enterprise Linux so it's ok. And with timeout:

timeout 180 grep -q 'PATTERN' <(tail -f file.log)

Answer 5

Take a look at the --max-count option:

tail -f file.log | grep -m 1 'PATTERN'

It will exit after the first line that matches PATTERN.

---

EDIT: take note of @Karoly's comment below. If the file.log velocity is slow, it's possible that the grep process will block until additional content is added to the file after the matching line.

echo 'context PATTERN line' >> file.log  ## grep shows the match but doesn't exit

will print the matching line, but it will not exit until additional content is appended to the file (even if it doesn't have a newline yet):

echo -n ' ' >> file.log  ## Now the grep process exits

In some cases (such as a high-velocity log file), this isn't a big deal, because new content is probably going to be added to the file soon anyway.

Also note that this behavior does not happen when reading from a console as stdin, so it appears to be a difference in the way grep reads from a pipe:

$ grep -m1 'PATTERN' -      # manually type PATTERN and enter, exits immediately
$ cat | grep -m1 'PATTERN'  # manually type PATTERN and enter, and it hangs

Answer 6

tail -f file | grep word | head -n1

Will post snip with async timeout

For now: How to include a timer in Bash Scripting?

The linked answer defines a 'run_or_timeout' function that does what you are looking for in a very bash-savvy way