Downloading file from the right Apache Mirror with

Question

To use a specific example, I want to download the binary release of Hadoop 2.7.2. The web site points to http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz which then points to the closest mirror by location. For me that is http://xenia.sote.hu/ftp/mirrors/www.apache.org/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz.

I want to actually download this in a shell script (a Dockerfile to be specific). I would prefer to use a location-agnostic URL for the download so that if someone runs the script on the other end of the world, they would not use the same mirror.

Is there a URL I could use with wget or curl that dynamically redirects to the closest mirror? What would that URL be for this specific file?

Answer 1

The source code of closer.lua actually states that the action and filename query parameters may be used to produce a redirect to the requested file on the automatically chosen mirror instead of the usual HTML mirror selection page.

So you can download the files directly via this URL: https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz:

GET /dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: www.apache.org



HTTP/1.1 302 Found
Cache-Control: max-age=3600
Connection: Keep-Alive
Content-Length: 0
Date: Mon, 13 Mar 2017 18:08:00 GMT
Expires: Mon, 13 Mar 2017 19:08:00 GMT
Keep-Alive: timeout=30, max=100
Location: http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
Server: Apache/2.4.7 (Ubuntu)