Parse contents of a file between certain timestamp

Question

I have a log file that contains a lot of information and I would like to only parse the contents of that file which fall within the last 24 hours

Each line in the file begins with a timestamp such as Jan 31 13:13:02 and then has a log message.

I currently have a batch file that finds the start and end time like this

start=$(date +"%b  %d %H:%M:%S" --date="-1 day")
end=$(date +"%b  %d %H:%M:%S")

I was then hoping to use these times along with a grep -c "data_to_find" to find the number of occurrences of a certain log message so that I can then act on this later.

In short, How can I take into account the times and then grep the content for the number of occurrences of a string within said file?

I am on a linux system and have no issue with any solution that uses SED, AWK, GREP etc.

Answer 1

Not so simple without writing a shell script (especially if it's not sorted).

I would try something like this to get all the lines between 1 day ago and now (interpolate as needed), and then grep -c pipe whatever you want from output. Note below assumes date format is something like Jan 31 13:13:02 (2 spaces between Month and Day)

#!/bin/bash
yest=$(date -d "1 days ago" '+%s')
today=$(date '+%s')

while read -r line; do
  date=
  [[ $line =~ ^[[:alpha:]]+[[:blank:]][[:blank:]][0-9]+[[:blank:]][0-9]+':'[0-9]+':'[0-9]+ ]] && date="${BASH_REMATCH[0]}"
  [[ -n $date ]] && date=$(date -d "$date" '+%s')
  [[ -n $date && $date -ge $yest && $date -le $today ]] && echo "$line"
done < logfile

Answer 2

This can be tricky because the time with the exact second you are looking for may not exist if there was no log entry at that time.

Another possibility may be to add a marker to the log each time you read it and then just look at entries after your last marker. So, when you have read the log, let's say it is called "log.txt" you could do:

echo marker >> log.txt

Then, when you want to read the log starting from your last marker, you can find the last marker like this:

LASTMARKER=$(grep -n marker log.txt | tail -1 | awk -F: '{print $1}')

and read everything after it like this:

sed -n "$LASTMARKER,$ p" log.txt

Answer 3

sed -n "/^${start}/,/^${end}/ { /${YourPattern}/ p}" | wc -l

Awk could certainly do that in one script. In sed, counting is a bit complex so wc is used (a bit faster than sed -n '$ ='). Yourpattern is a regex that suite to your log message to find