Persistent DNS caching not utilized by Python'

Question

I used to have very slow DNS lookups on my Ubuntu machine when connecting through one Modem/ISP. I followed instructions (such as those here) to use persistent DNS caching so I don't do repeated DNS look ups, and everything became much faster. However, I noticed that when pulling/pushing repos on mercurial, it was painfully slow to even transfer very small changes. Looking deeper, I found that all the time was taken by _socket.getaddrinfo, i.e., DNS lookups. Why is it that even though all web browsers use the info in /etc/pdnsd.conf to use old dns lookups, pythons sockets do not. How can I get mercurial to use cached lookups?

Update

It might also be because getaddrinfo tries to get both ipv4 and ipv6 addresses, and only returns the answer after it fails to find the ipv6 address. I am not sure how to check this though, i.e., whether the delay is due to ipv6, non-persistence or both.

Update

It might be related to ipv6 requests, as suggested in the answer here, I will have to check that once I get a chance.

Answer 1

Thanks to this post on the python list, it seems if you use BIND, then python's getaddrinfo uses cached DNS lookups. After installing BIND and starting it, then repeated DNS lookups are instantaneous.

Answer 2

Pythons socket.getaddrinfo() uses the OS’s getaddrinfo() (over libc). – This has nothing to do with Python at all. If pdns is already configured to be in the resolution-chain, then different resolution-speeds are rooted by different requests. – Look more closely at which exact getaddrinfo() requests are fast, which are slow.