Problem with Bash output redirection

Question

I was trying to remove all the lines of a file except the last line but the following command did not work, although file.txt is not empty.

$cat file.txt |tail -1 > file.txt

$cat file.txt

Why is it so?

Answer 1

As Lewis Baumstark says, it doesn't like it that you're writing to the same filename.

This is because the shell opens up "file.txt" and truncates it to do the redirection before "cat file.txt" is run. So, you have to

tail -1 file.txt > file2.txt; mv file2.txt file.txt

Answer 2

You can use sed to delete all lines but the last from a file:

sed -i '$!d' file

• -i tells sed to replace the file in place; otherwise, the result would write to STDOUT.
• $ is the address that matches the last line of the file.
• d is the delete command. In this case, it is negated by !, so all lines not matching the address will be deleted.

Answer 3

Redirecting from a file through a pipeline back to the same file is unsafe; if file.txt is overwritten by the shell when setting up the last stage of the pipeline before tail starts reading off the first stage, you end up with empty output.

Do the following instead:

tail -1 file.txt >file.txt.new && mv file.txt.new file.txt

...well, actually, don't do that in production code; particularly if you're in a security-sensitive environment and running as root, the following is more appropriate:

tempfile="$(mktemp file.txt.XXXXXX)"
chown --reference=file.txt -- "$tempfile"
chmod --reference=file.txt -- "$tempfile"
tail -1 file.txt >"$tempfile" && mv -- "$tempfile" file.txt

Another approach (avoiding temporary files, unless <<< implicitly creates them on your platform) is the following:

lastline="$(tail -1 file.txt)"; cat >file.txt <<<"$lastline"

(The above implementation is bash-specific, but works in cases where echo does not -- such as when the last line contains "--version", for instance).

Finally, one can use sponge from moreutils:

tail -1 file.txt | sponge file.txt

Answer 4

It seems to not like the fact you're writing it back to the same filename. If you do the following it works:

$cat file.txt | tail -1 > anotherfile.txt

Answer 5

Just for this case it's possible to use

cat < file.txt | (rm file.txt; tail -1 > file.txt)

That will open "file.txt" just before connection "cat" with subshell in "(...)". "rm file.txt" will remove reference from disk before subshell will open it for write for "tail", but contents will be still available through opened descriptor which is passed to "cat" until it will close stdin. So you'd better be sure that this command will finish or contents of "file.txt" will be lost