Plone: pypi specific download issue behind a proxy

Have a weird problem here.

I am in a network that uses an authenticated proxy to connect to the outside world. I do not control this proxy server.

This fact has in the past tripped me up with use of wget, curl, etc. (this is on Mac OSX 10.8). So, I set the environment variables:

HTTP_PROXY=proxyserveraddress:3128
FTP_PROXY=proxyserveraddress:3128
RSYNC_PROXY=proxyserveraddress:3128

This when added to the bash rc (is it profile?) file, has made the network connections transparent to the shell. There is also a system proxy setting that points the same way.

Now, I installed Plone and wanted to plone.app.ldap egg to the installation.

I ran bin/buildout after updating buildout.cfg (to include this egg), and I get:

Installing instance.
Download error on http://pypi.python.org/simple/plone.app.ldap/: [Errno 61] Connection     refused -- Some packages may not be found!
Download error on http://pypi.python.org/simple/plone.app.ldap/: [Errno 61] Connection refused -- Some packages may not be found!
Couldn't find index page for 'plone.app.ldap' (maybe misspelled?)
Download error on http://pypi.python.org/simple/: [Errno 61] Connection refused -- Some  packages may not be found!
Getting distribution for 'plone.app.ldap'.
While:
  Installing instance.
  Getting distribution for 'plone.app.ldap'.
Error: Couldn't find a distribution for 'plone.app.ldap'.
*************** PICKED VERSIONS ****************
[versions]

*************** /PICKED VERSIONS ***************

Next, I edited bin/buildout and added the following lines:

import os

print os.environ["HTTP_PROXY"]

I ran bin/buildout again.

Sure enough, it spat out

proxyserveraddress:3128

That means that python is reading in environment variables.

I went a step further (and this where it gets interesting) and unset the proxy setting (just to see if it would matter):

MacBook-Pro:zinstance macpro$export HTTP_PROXY=
MacBook-Pro:zinstance macpro$ bin/buildout

Installing instance.
Download error on http://dist.plone.org: [Errno 61] Connection refused -- Some packages may not be found!
Download error on http://download.zope.org/ppix/: [Errno 61] Connection refused -- Some packages may not be found!
Download error on http://download.zope.org/distribution/: [Errno 61] Connection refused -- Some packages may not be found!
Download error on http://effbot.org/downloads: [Errno 61] Connection refused -- Some packages may not be found!
Download error on http://dist.plone.org/release/4.3.2: [Errno 61] Connection refused -- Some packages may not be found!
Download error on http://pypi.python.org/simple/plone.app.ldap/: [Errno 61] Connection refused -- Some packages may not be found!
Download error on http://pypi.python.org/simple/plone.app.ldap/: [Errno 61] Connection refused -- Some packages may not be found!
Couldn't find index page for 'plone.app.ldap' (maybe misspelled?)
Download error on http://pypi.python.org/simple/: [Errno 61] Connection refused -- Some packages may not be found!
Getting distribution for 'plone.app.ldap'.
While:
  Installing instance.
  Getting distribution for 'plone.app.ldap'.
Error: Couldn't find a distribution for 'plone.app.ldap'.
*************** PICKED VERSIONS ****************
[versions]

*************** /PICKED VERSIONS ***************

So, you unset the proxy and other servers (plone, zope, effbot) start complaining. That means that my proxy settings were working fine for those downloads but in some God-I-wish-knew-why way, they are not good enough for pypi.python.org.

What is so special about pypi.python.org? Does it use some special port for connecting (unlike the other servers contacted by buildout) that may be shut off by the network proxy server?? Before anyone asks, yes, I can browse to the pypi.python.org URL fine using Chrome browser.

I have established that I have a sensible proxy setting. Its pypi that is messing up buildout.

It is possible that the secured socket authentication used by pypi is causing your problems as it probably uses a different port that you are not including in your proxy forwarding list. You may be able to resolve this in the same manner as your other poxy settings.

Another way round would be to manually download and install the required packages from pypi before running the build - if the dependencies are already satisfied they the access will not be used.

You could also consider running a local pypi server and directing to that - you would still need to download to that server the packages but if you are doing multiple installations it would be worth it.