Trouble with piping through sed

Question

I am having trouble piping through sed. Once I have piped output to sed, I cannot pipe the output of sed elsewhere.

wget -r -nv http://127.0.0.1:3000/test.html

Outputs:

2010-03-12 04:41:48 URL:http://127.0.0.1:3000/test.html [99/99] -> "127.0.0.1:3000/test.html" [1]
2010-03-12 04:41:48 URL:http://127.0.0.1:3000/robots.txt [83/83] -> "127.0.0.1:3000/robots.txt" [1]
2010-03-12 04:41:48 URL:http://127.0.0.1:3000/shop [22818/22818] -> "127.0.0.1:3000/shop.29" [1]

I pipe the output through sed to get a clean list of URLs:

wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g'

Outputs:

http://127.0.0.1:3000/test.html
http://127.0.0.1:3000/robots.txt
http://127.0.0.1:3000/shop

I would like to then dump the output to file, so I do this:

wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' > /tmp/DUMP_FILE

I interrupt the process after a few seconds and check the file, yet it is empty.

Interesting, the following yields no output (same as above, but piping sed output through cat):

wget -r -nv http://127.0.0.1:3000/test.html 2>&1 | grep --line-buffered -v ERROR | sed 's/^.*URL:\([^ ]*\).*/\1/g' | cat

Why can I not pipe the output of sed to another program like cat?

Answer 1

you can also use awk. since your URL appears in field 3, you can use $3, and you can remove the grep as well.

awk '!/ERROR/{sub("URL:","",$3);print $3}' file

Answer 2

When sed is writing to another process or to a file, it will buffer data.

Try adding the --unbuffered options to sed.