How can I check if a URL exists with Django’s vali

Question

I want to check in django if a URL exists and if it does I want to show something on screen, i.e.:

if URL_THAT_POINTS_TO_SOME_PDF exists 
     SHOW_SOMETHING

Answer 1

Anything based on the verify_exists parameter to django.core.validators.URLValidator will stop working with Django 1.5 — the documentation helpfully says nothing about this, but the source code reveals that using that mechanism in 1.4 (the latest stable version) leads to a DeprecationWarning (you'll see it has been removed completely in the development version):

if self.verify_exists:
    import warnings
    warnings.warn(
        "The URLField verify_exists argument has intractable security "
        "and performance issues. Accordingly, it has been deprecated.",
        DeprecationWarning
        )

There are also some odd quirks with this method related to the fact that it uses a HEAD request to check URLs — bandwidth-efficient, sure, but some sites (like Amazon) respond with an error (to HEAD, where the equivalent GET would have been fine), and this leads to false negative results from the validator.

I would also (a lot has changed in two years) recommend against doing anything with urllib2 in a template — this is completely the wrong part of the request/response cycle to be triggering potentially long-running operations: consider what happens if the URL does exist, but a DNS problem causes urllib2 to take 10 seconds to work that out. BAM! Instant 10 extra seconds on your page load.

I would say the current best practice for making possibly-long-running tasks like this asynchronous (and thus not blocking page load) is using django-celery; there's a basic tutorial which covers using pycurl to check a website, or you could look into how Simon Willison implemented celery tasks (slides 32-41) for a similar purpose on Lanyrd.