CURL Progress Bar: How to pipe and extract numbers

Question

This is what I have so far:

[my1@graf home]$ curl -# -o f1.flv 'http://osr.com/f1.flv' |  grep -o '*[0-9]*'
####################################################################### 100.0%

I wish to use grep and only extract the percentage from that progress bar that CURL outputs.

I think my regex is not correct and I am also not sure if this grep will take effect of the the percentage being continuously updated?

What I am trying to do is basically get CURL only to give me the percentage number as the output and nothing else.

Thank you for any help.

Answer 1

You need to use .* not * in your regex.

grep -o '.*[0-9].*'

That will catch all text though, so maybe try:

grep -p '[0-9]+'

Answer 2

With curl 7.36.0 (should also work for other versions) you can extract the percentage in the following way:

curl ... 2>&1 -# | stdbuf -oL tr '\r' '\n' | grep -o '[0-9]*\.[0-9]'

Here ... stands for options/filenames. This outputs a sequence of percentage numbers.

Curl uses carriage returns \r in its output, so you need tr to transform them first into \n because grep is line oriented. You also need to modify output buffer settings with stdbuf to get the percentage numbers immediately after curl outputs them.

Answer 3

Try this:

curl source -o dest -# 2> tmp&

grep -o ".....%" tmp | tail -n1

Answer 4

You can't get the progress info like that through grep; it doesn't make sense.

curl writes the progress bar to stderr, so you have to redirect to stdout before you can grep it:

$ curl -# -o f1.flv 'http://osr.com/f1.flv' 2>&1 | grep 1 | less results in:

^M                                                                           0.0
%^M######################################################################## 100.
0%^M######################################################################## 100
.0%^M######################################################################## 10
0.0%

Are you expecting a continual stream of numbers that you are redirecting somewhere else? Or do you expect to grab the numbers at a single point?

If it's the former, this sort of half-assedly works on a small file:

$ curl -# -o f1.flv 'http://osr.com/f1.flv' 2>&1 | sed  's/#//g' -
 100.0%                                                                    0.0%

But it's useless on a large file. The output doesn't print until the download is finished, probably because curl seems to be sending ^H's to the terminal. There might be a better way to sed it, but I wouldn't hold my breath.

$ curl -# -o l.tbz 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2009/06/2009-06-02-05-mozilla-1.9.1/firefox-3.5pre.en-US.linux-x86_64.tar.bz2' 2>&1 | sed 's/#//g' -
 100.0%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%