After scanning the urllib2
source, it seems that connections are automatically closed even if you do specify keep-alive.
Why is this?
As it is now I just use httplib
for my persistent connections... but wonder why this is disabled (or maybe just ambiguous) in urllib2.
It's a well-known limit of urllib2 (and urllib as well). IMHO the best attempt so far to fix it and make it right is Garry Bodsworth's coda_network for Python 2.6 or 2.7 -- replacement, patched versions of urllib2 (and some other modules) to support keep-alive (and a bunch of other smaller but quite welcome fixes).
You might also check out httplib2, which supports persistent connections. Not quite the same as urllib2 (in the sense that it only does http and not "any kind of url"), but easier than httplib (and imho also easier than urllib2 if you really want to do http).