Read git clone's output in real time

Question

I need to read the status ( Receiving objects XXX%) of a git clone process, but can't figure out how.

I am using subprocess.Popen but I cant' figure out how to grab the line I need:

proc = subprocess.Popen([GIT, "-C",IDIR+"/"+REPO,"clone",URL],shell=False,bufsize=0,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)

This is the typical output:

Cloning into 'xxx'...
remote: Reusing existing pack: 5250, done.
remote: Counting objects: 1652, done.
remote: Compressing objects: 100% (1428/1428), done.
remote: Total 6902 (delta 389), reused 0 (delta 0)
Receiving objects: XXX% (6902/6902), 27.66 MiB | 1.51 MiB/s, done.
Resolving deltas: 100% (2010/2010), done.
Checking connectivity... done.
Checking out files: 100% (3266/3266), done.

Edit

I have tried all of the suggestions both in this thread and in the one suggested as a duplicate, but none appear to work.

This suggestion results in "Cloning into 'test'... done" but none of the other output:

popen = subprocess.Popen(["git", "clone", "https://github.com/xxx/test.git"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT,shell=False )
for line in popen.stdout:
  print "out newline:"
  print line
print "done"

This is the output which does not contain any of the stats:

out newline:
Cloning into 'test'...

done

Answer 1

I think a subprocess.communicate() should do the trick.

I usually do it like this:

process=subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutput,stderror=process.communicate()
for line in stdoutput:
    if line.startswith('Receiving objects'):
        print line

Answer 2

I assume that you want the real-time communication to display some sort of progress while the process is still running.

The problem is that the normal stdout stream is buffered. What you need is the unbuffered stream. You can obtain it using the os module, for example:

  fd = proc.stdout.fileno()
  while proc.returncode is None:
      l = os.read(fd, 1000)   # Read a bit of data
      print l

Answer 3

First of all, I suggest combining stdout and stderr to have everything in one place i.e. call Popen with stderr=subprocess.STDOUT.

Second, you need to execute git with --progress because the git-clone man page says:

--progress

Progress status is reported on the standard error stream by default when
it is attached to a terminal, unless -q is specified.  This flag forces
progress status even if the standard error stream is not directed to a
terminal.

and your git is not attached to a terminal.

So the code should look like this:

popen = subprocess.Popen(["git", "clone", "--progress", url],
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
for line in popen.stdout:
  print "out newline:"
  print line
print "done"

I suggest reading the answers to the "Getting realtime output using subprocess" question to improve your program.

Answer 4

[edited as per the comments] Calling git clone with the --progress flag redirects the output ok.

This works for me.

import subprocess

popen = subprocess.Popen(["git", "clone", "--progress", "[e-mail]:xxx/yyy.git"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in popen.stdout:
    print line,

print "done"

Gives the output

$ python test.py 
Cloning into 'yyy'...
remote: Counting objects: 1076, done.
remote: Compressing objects: 100% (761/761), done.
remote: Total 1076 (delta 488), reused 576 (delta 227)
Receiving objects: 100% (1076/1076), 6.24 MiB | 260.00 KiB/s, done.
Resolving deltas: 100% (488/488), done.
Checking connectivity... done.
done

[edit] Like Cristian Ciupitu points out you don't need the iter() just for line in popen.stdout: works just as well (or not depending on version).