How to get the part of file after the line that ma

Question

I have a file with about 1000 lines. I want the part of my file after the line which matches my grep statement.

i.e.

$ cat file | grep 'TERMINATE'     // Its found on line 534

So, I want the file from line 535 to line 1000 for further processing.

How do I do it ?

Answer 1

If I understand your question correctly you do want the lines after TERMINATE, not including the TERMINATE-line. awk can do this in a simple way:

awk '{if(found) print} /TERMINATE/{found=1}' your_file

Explanation:

1. Although not best practice you could rely on the fact that all vars defaults to 0 or the empty string if not defined. So the first expression (if(found) print) will not print anything to start off with.
2. After the printing is done we check if the this is the starter-line (that should not be included).

This will print all lines after the TERMINATE-line.

---

Generalization:

• You have a file with start- and end-lines and you want the lines between those lines excluding the start- and end-lines.
• start- and end-lines could be defined by a regular expression matching the line.

Example:

$ cat ex_file.txt 
not this line
second line
START
A good line to include
And this line
Yep
END
Nope more
...
never ever
$ awk '/END/{found=0} {if(found) print} /START/{found=1}' ex_file.txt 
A good line to include
And this line
Yep
$

Explanation:

1. If the end-line is found no printing should be done. Note that this check is done before the actual printing to exclude the end-line from the result.
2. Print the current line if found is set.
3. If the start-line is found then set found=1 so that the following lines are printed. Note that this check is done after the actual printing to exclude the start-line from the result.

Notes:

• The code rely on the fact that all awk-vars defaults to 0 or the empty string if not defined. This is valid but may not be best practice so you could add a BEGIN{found=0} to the start of the awk-expression.
• If multiple start-end-blocks is found they are all printed.

Answer 2

A tool to use here is awk:

cat file | awk 'BEGIN{ found=0} /TERMINATE/{found=1}  {if (found) print }'

How does this work:

1. We set the variable 'found' to zero, evaluating false
2. if a match for 'TERMINATE' is found with the regular expression, we set it to one.
3. If our 'found' variable evaluates to True, print :)

The other solutions might consume a lot of memory if you use them on very large files.

Answer 3

This could be a one way of doing it. If you know what line of the file you have your grep word and how many lines you have in your file:

grep -A466 'TERMINATE' file

Answer 4

The following will print the line matching TERMINATE till the end of the file:

sed -n -e '/TERMINATE/,$p'

Explained: -n disables default behavior of sed of printing each line after executing its script on it, -e indicated a script to sed, /TERMINATE/,$ is an address (line) range selection meaning the first line matching the TERMINATE regular expression (like grep) to the end of the file ($), and p is the print command which prints the current line.

This will print from the line that follows the line matching TERMINATE till the end of the file:
(from AFTER the matching line to EOF, NOT including the matching line)

sed -e '1,/TERMINATE/d'

Explained: 1,/TERMINATE/ is an address (line) range selection meaning the first line for the input to the 1st line matching the TERMINATE regular expression, and d is the delete command which delete the current line and skip to the next line. As sed default behavior is to print the lines, it will print the lines after TERMINATE to the end of input.

Edit:

If you want the lines before TERMINATE:

sed -e '/TERMINATE/,$d'

And if you want both lines before and after TERMINATE in 2 different files in a single pass:

sed -e '1,/TERMINATE/w before
/TERMINATE/,$w after' file

The before and after files will contain the line with terminate, so to process each you need to use:

head -n -1 before
tail -n +2 after

Edit2:

IF you do not want to hard-code the filenames in the sed script, you can:

before=before.txt
after=after.txt
sed -e "1,/TERMINATE/w $before
/TERMINATE/,\$w $after" file

But then you have to escape the $ meaning the last line so the shell will not try to expand the $w variable (note that we now use double quotes around the script instead of single quotes).

I forgot to tell that the new line is important after the filenames in the script so that sed knows that the filenames end.

Edit: 2016-0530

Sébastien Clément asked: "How would you replace the hardcoded TERMINATE by a variable?"

You would make a variable for the matching text and then do it the same way as the previous example:

matchtext=TERMINATE
before=before.txt
after=after.txt
sed -e "1,/$matchtext/w $before
/$matchtext/,\$w $after" file

to use a variable for the matching text with the previous examples:

## Print the line containing the matching text, till the end of the file:
## (from the matching line to EOF, including the matching line)
matchtext=TERMINATE
sed -n -e "/$matchtext/,\$p"

## Print from the line that follows the line containing the 
## matching text, till the end of the file:
## (from AFTER the matching line to EOF, NOT including the matching line)
matchtext=TERMINATE
sed -e "1,/$matchtext/d"

## Print all the lines before the line containing the matching text:
## (from line-1 to BEFORE the matching line, NOT including the matching line)
matchtext=TERMINATE
sed -e "/$matchtext/,\$d"

The important points about replacing text with variables in these cases are:

1. Variables ($variablename) enclosed in single quotes ['] won't "expand" but variables inside double quotes ["] will. So, you have to change all the single quotes to double quotes if they contain text you want to replace with a variable.
2. The sed ranges also contain a $ and are immediately followed by a letter like: $p, $d, $w. They will also look like variables to be expanded, so you have to escape those $ characters with a backslash [\] like: \$p, \$d, \$w.

Answer 5

These will print all lines from the last found line "TERMINATE" till end of file:

LINE_NUMBER=`grep -o -n TERMINATE $OSCAM_LOG|tail -n 1|sed "s/:/ \\'/g"|awk -F" " '{print $1}'`
tail -n +$LINE_NUMBER $YOUR_FILE_NAME

Answer 6

As a simple approximation you could use

grep -A100000 TERMINATE file

which greps for TERMINATE and outputs up to 100000 lines following that line.

From man page

-A NUM, --after-context=NUM

Print NUM lines of trailing context after matching lines. Places a line containing a group separator (--) between contiguous groups of matches. With the -o or --only-matching option, this has no effect and a warning is given.