Problem with Bash output redirection

Question

I was trying to remove all the lines of a file except the last line but the following command did not work, although file.txt is not empty.

$cat file.txt |tail -1 > file.txt

$cat file.txt

Why is it so?

Answer 1

Before 'cat' gets executed, Bash has already opened 'file.txt' for writing, clearing out its contents.

In general, don't write to files you're reading from in the same statement. This can be worked around by writing to a different file, as above:

$cat file.txt | tail -1 >anotherfile.txt
$mv anotherfile.txt file.txt

or by using a utility like sponge from moreutils:

$cat file.txt | tail -1 | sponge file.txt

This works because sponge waits until its input stream has ended before opening its output file.

Answer 2

echo "$(tail -1 file.txt)" > file.txt

Answer 3

This works nicely in a Linux shell:

replace_with_filter() {
  local filename="$1"; shift
  local dd_output byte_count filter_status dd_status
  dd_output=$("$@" <"$filename" | dd conv=notrunc of="$filename" 2>&1; echo "${PIPESTATUS[@]}")
  { read; read; read -r byte_count _; read filter_status dd_status; } <<<"$dd_output"
  (( filter_status > 0 )) && return "$filter_status"
  (( dd_status > 0 )) && return "$dd_status"
  dd bs=1 seek="$byte_count" if=/dev/null of="$filename"
}

replace_with_filter file.txt tail -1

dd's "notrunc" option is used to write the filtered contents back, in place, while dd is needed again (with a byte count) to actually truncate the file. If the new file size is greater or equal to the old file size, the second dd invocation is not necessary.

The advantages of this over a file copy method are: 1) no additional disk space necessary, 2) faster performance on large files, and 3) pure shell (other than dd).

Answer 4

tail -1 > file.txt will overwrite your file, causing cat to read an empty file because the re-write will happen before any of the commands in your pipeline are executed.

Answer 5

When you submit your command string to bash, it does the following:

1. Creates an I/O pipe.
2. Starts "/usr/bin/tail -1", reading from the pipe, and writing to file.txt.
3. Starts "/usr/bin/cat file.txt", writing to the pipe.

By the time 'cat' starts reading, 'file.txt' has already been truncated by 'tail'.

That's all part of the design of Unix and the shell environment, and goes back all the way to the original Bourne shell. 'Tis a feature, not a bug.

Answer 6

tmp=$(tail -1 file.txt); echo $tmp > file.txt;