Redirect with no auth

问题:

According to the docs, it should be as simple as:

data = self.http_pool.urlopen('GET', file_url,
                              preload_content=False,
                              retries=max_download_retries) 



request.add_unredirected_header(key, header)
Add a header that will not be added to a redirected request.

But I cannot seem to find any examples on how this can be achieved.

I am using the pyupdater to download updates from bitbucket and launch the newest version of exe. I am using this library to create a script that connects to bitbucket fine, but then it later redirects to Amazon with nauthorization: Basic <redacted>\r\n\r\n (this is bitbucket auth) meaning I get 'HTTP/1.1 400 Bad Request\r\n'. Amazon does not support basic auth. This should be easily solvable, but I cannot find much on this issue.

The solutions presented here, require Recreating each redirected request manually. This would become an ever-growing list and get tedious very quickly, if I had to do this for new file I uploaded. It also does not continue the rest of the script, but rather downloads to the same directory.

As this is how Pyupdater handles the downloads this is where the issue would likely be solved.

Line 366 of downloader.py:

data = self.http_pool.urlopen('GET', file_url,
                                              preload_content=False,
                                              retries=max_download_retries)

Any ideas on how to fix this so it no longer creates this error.

Full Error (ctrl f -> 400):

Python main.py
DEBUG:root:Version - 2.5.1
DEBUG:pyupdater.client:PyUpdater Version 2.5.1
Current version is  1.3
{'authorization': 'Basic <redacted>'}
DEBUG:pyupdater.client:Setting up directories...
DEBUG:pyupdater.client:Downloading key file
DEBUG:pyupdater.client.downloader:Url for request: https://api.bitbucket.org/2.0/repositories/ brofewfefwefewef/eee/downloads/keys.gz
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
send: b'GET /2.0/repositories/ brofewfefwefewef/eee/downloads/keys.gz HTTP/1.1\r\nHost: api.bitbucket.org\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET /2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz HTTP/1.1" 302 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://api.bitbucket.org/2.0/repositories/ brofewfefwefewef/eee/downloads/keys.gz -> https://bbuseruploads.s3.amazonaws.com/a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=<redacted>&Expires=1515976464&AWSAccessKeyId=<redacted>&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bbuseruploads.s3.amazonaws.com
header: Server header: Vary header: Content-Type header: X-OAuth-Scopes header: Strict-Transport-Security header: Date header: Location header: X-Served-By header: ETag header: X-Static-Version header: X-Content-Type-Options header: X-Accepted-OAuth-Scopes header: X-Credential-Type header: X-Render-Time header: Connection header: X-Request-Count header: X-Frame-Options header: X-Version header: Content-Length send: b'GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=<redacted>&Expires=1515976464&AWSAccessKeyId=<redacted>&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1\r\nHost: bbuseruploads.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
DEBUG:urllib3.connectionpool:https://bbuseruploads.s3.amazonaws.com:443 "GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/3fc0be6d-ca69-42d3-9711-fbb5cfd2bc38/keys.gz?Signature=<redacted>&Expires=1515976464&AWSAccessKeyId=<redacted>&versionId=n.ymY11KRkq36Xozy25aChvfUT.YzTf5&response-content-disposition=attachment%3B%20filename%3D%22keys.gz%22 HTTP/1.1" 400 None
DEBUG:pyupdater.client.downloader:Resource URL: https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz
DEBUG:pyupdater.client.downloader:Got content length of: None
DEBUG:pyupdater.client.downloader:Content-Length not in headers
DEBUG:pyupdater.client.downloader:Callbacks will not show time left or percent downloaded.
DEBUG:pyupdater.client.downloader:Using file as storage since the file is too large
DEBUG:pyupdater.client.downloader:Block size: 1036
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'downloading', 'percent_complete': '-.-%', 'time': '--:--'}
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'finished', 'percent_complete': '-.-%', 'time': '00:00'}
DEBUG:pyupdater.client.downloader:Download Complete
DEBUG:pyupdater.client.downloader:No hash to verify
WARNING:pyupdater.client.downloader:Downloaded file is very large, reading it in to memory may crash the app
DEBUG:pyupdater.client:Failed to decompress gzip file
DEBUG:pyupdater.client:Version file download failed
header: x-amz-request-id header: x-amz-id-2 header: Content-Type header: Transfer-Encoding header: Date header: Connection header: Server {'authorization': 'Basic <redacted>'}
DEBUG:pyupdater.client:Not a gzipped file (b'<?')
Traceback (most recent call last):
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\pyupdater\client\__init__.py", line 440, in _get_key_data
    decompressed_data = _gzip_decompress(data)
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\dsdev_utils\helpers.py", line 58, in gzip_decompress
    data = decompressed_file.read()
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 276, in read
    return self._buffer.read(size)
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 463, in read
    if not self._read_gzip_header():
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 411, in _read_gzip_header
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'<?')
DEBUG:pyupdater.client:Loading version file...
DEBUG:pyupdater.client:Downloading online version file
DEBUG:pyupdater.client.downloader:Url for request: https://api.bitbucket.org/2.0/repositories/ brofewfefwefewef/eee/downloads/versions.gz
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.bitbucket.org
send: b'GET /2.0/repositories/ brofewfefwefewef/eee/downloads/versions.gz HTTP/1.1\r\nHost: api.bitbucket.org\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
DEBUG:urllib3.connectionpool:https://api.bitbucket.org:443 "GET /2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz HTTP/1.1" 302 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz -> https://bbuseruploads.s3.amazonaws.com/a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/0b04c4a8-dd59-49d2-9cd7-95d22379a5e6/versions.gz?Signature=<redacted>&Expires=1515976465&AWSAccessKeyId=<redacted>&versionId=jLhOcIbVAU4xRghD3kB2NfB4iLqUr7PM&response-content-disposition=attachment%3B%20filename%3D%22versions.gz%22
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bbuseruploads.s3.amazonaws.com
header: Server header: Vary header: Content-Type header: X-OAuth-Scopes header: Strict-Transport-Security header: Date header: Location header: X-Served-By header: ETag header: X-Static-Version header: X-Content-Type-Options header: X-Accepted-OAuth-Scopes header: X-Credential-Type header: X-Render-Time header: Connection header: X-Request-Count header: X-Frame-Options header: X-Version header: Content-Length send: b'GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/0b04c4a8-dd59-49d2-9cd7-95d22379a5e6/versions.gz?Signature=<redacted>&Expires=1515976465&AWSAccessKeyId=<redacted>&versionId=jLhOcIbVAU4xRghD3kB2NfB4iLqUr7PM&response-content-disposition=attachment%3B%20filename%3D%22versions.gz%22 HTTP/1.1\r\nHost: bbuseruploads.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nauthorization: Basic <redacted>\r\n\r\n'
DEBUG:urllib3.connectionpool:https://bbuseruploads.s3.amazonaws.com:443 "GET /a0e395b6-0c54-4efb-9074-57ec4190020b/downloads/0b04c4a8-dd59-49d2-9cd7-95d22379a5e6/versions.gz?Signature=<redacted>&Expires=1515976465&AWSAccessKeyId=<redacted>&versionId=jLhOcIbVAU4xRghD3kB2NfB4iLqUr7PM&response-content-disposition=attachment%3B%20filename%3D%22versions.gz%22 HTTP/1.1" 400 None
reply: 'HTTP/1.1 400 Bad Request\r\n'
DEBUG:pyupdater.client.downloader:Resource URL: https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/versions.gz
DEBUG:pyupdater.client.downloader:Got content length of: None
DEBUG:pyupdater.client.downloader:Content-Length not in headers
DEBUG:pyupdater.client.downloader:Callbacks will not show time left or percent downloaded.
DEBUG:pyupdater.client.downloader:Using file as storage since the file is too large
DEBUG:pyupdater.client.downloader:Block size: 1036
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'downloading', 'percent_complete': '-.-%', 'time': '--:--'}
DEBUG:pyupdater.client.downloader:{'total': None, 'downloaded': 519, 'status': 'finished', 'percent_complete': '-.-%', 'time': '00:00'}
DEBUG:pyupdater.client.downloader:Download Complete
DEBUG:pyupdater.client.downloader:No hash to verify
WARNING:pyupdater.client.downloader:Downloaded file is very large, reading it in to memory may crash the app
DEBUG:pyupdater.client:Failed to decompress gzip file
DEBUG:pyupdater.client:Version file download failed
DEBUG:pyupdater.client:Not a gzipped file (b'<?')
Traceback (most recent call last):
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\pyupdater\client\__init__.py", line 417, in _get_manifest_from_http
    decompressed_data = _gzip_decompress(data)
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\lib\site-packages\dsdev_utils\helpers.py", line 58, in gzip_decompress
    data = decompressed_file.read()
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 276, in read
    return self._buffer.read(size)
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 463, in read
    if not self._read_gzip_header():
  File "C:\Users\Django\AppData\Local\Continuum\miniconda3\Lib\gzip.py", line 411, in _read_gzip_header
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b'<?')
DEBUG:dsdev_utils.paths:Changing to Directory --> C:\Users\Django\AppData\Local\any\main
DEBUG:pyupdater.client:Found version file on file system
DEBUG:pyupdater.client:Loaded version file from file system
DEBUG:dsdev_utils.paths:Moving back to Directory --> C:\Users\Django\privacy 4
DEBUG:pyupdater.client:Data type: <class 'bytes'>
DEBUG:pyupdater.client:App key is None
DEBUG:pyupdater.client:Version Data:
{'latest': {'main': {'stable': {'win': '1.4.0.2.0'}}}, 'updates': {'main': {'1.3.0.2.0': {'win': {'file_hash': '807c743b8c29f0053f4f9d9e6a8895b0e037f77480e7065c1470c2aba1cb08a0', 'file_size': 12194381, 'filename': 'main-win-1.3.zip', 'patch_hash': '29fec1006c2736eb78cc859f89e165af942daae6d9ac994a1a686d9b7b418ef6', 'patch_name': 'main-win-5', 'patch_size': 147}}, '1.4.0.2.0': {'win': {'file_hash': 'd59a22a95229f0a9c64909c646bfba31daf6bf8689dc16c9c93180c1602e9d3c', 'file_size': 12195571, 'filename': 'main-win-1.4.zip', 'patch_hash': 'baf3eba3a4b3184919ed9e57c3e8be9494a50862b40b1590ecb64e39e71a4ce3', 'patch_name': 'main-win-6', 'patch_size': 479625}}}}, 'signature': '<redacted>'}
DEBUG:dsdev_utils.helpers:Version str: 1.3
DEBUG:pyupdater.client:Failed version file verification

For those that want to replicate the error for themselves I’ve written the steps I have taken exactly.

回答1:

Edit-1:

You need use below code for your main.py without any changes to downloader.py

from __future__ import print_function
import urllib3.poolmanager

orig_urlopen = urllib3.poolmanager.PoolManager.urlopen


def new_urlopen(self, method, url, redirect=True, **kw):
    if "s3.amazonaws.com" in url and 'authorization' in self.headers:
        self.headers.pop('authorization')
    return orig_urlopen(self, method, url, redirect, **kw)


urllib3.poolmanager.PoolManager.urlopen = new_urlopen


import logging

from selenium import webdriver

logging.basicConfig(level=logging.DEBUG)
from client_config import ClientConfig
from pyupdater.client import Client, AppUpdate

import http.client as http_client

http_client.HTTPConnection.debuglevel = 1


def check_for_update():
    client = Client(ClientConfig(), refresh=True, headers={'basic_auth': '<username>:<password>'})
    app_update = client.update_check(ClientConfig.APP_NAME, ClientConfig.APP_VERSION, channel='stable')
    if app_update is not None:
        if app_update.download():
            if isinstance(app_update, AppUpdate):
                app_update.extract_restart()
                return True
            else:
                app_update.extract()
                return True
    return False


def main():
    print('Current version is ', ClientConfig.APP_VERSION)
    if check_for_update():
        print('there\'s a new update :D')
    # driver = webdriver.Firefox()
    # driver.get('http://stackoverflow.com')


if __name__ == "__main__":
    main()

original answer You need to use monkey patching for this. Below patch should do the job

import urllib3.poolmanager

orig_urlopen = urllib3.poolmanager.PoolManager.urlopen


def new_urlopen(self, method, url, redirect=True, **kw):
    if "s3.amazonaws.com" in url and 'Authorization' in self.headers:
        self.headers.pop('Authorization')
    return orig_urlopen(self, method, url, redirect, **kw)


urllib3.poolmanager.PoolManager.urlopen = new_urlopen

A sample test worked for me with the above patch

import urllib3

pool = urllib3.PoolManager()

pool.headers.update({'Authorization': 'Basic XYZ=='})
r = pool.urlopen('GET', 'https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz')
print(r.data)

You need to execute the code before import pyupdater

回答2:

I just checked and I believe it is a problem with pyupdater (I don't know what it is, never used).

It seems to assume that all the response's body will be compressed in GZIP. There is no flag I can find that would prevent this assumption. The actual content is actually not compressed at all.

Here is some relevant code from pyupdater:

pyupdater/client/__init__.py:

def _get_manifest_from_http(self):
    log.debug('Downloading online version file')
    try:
        fd = _FD(self.version_file, self.update_urls, verify=self.verify,
                 urllb3_headers=self.urllib3_headers)
        data = fd.download_verify_return()
        try:
            import ipdb
            ipdb.set_trace()
            decompressed_data = _gzip_decompress(data)
        except IOError:
            log.debug('Failed to decompress gzip file')
            # Will be caught down below.
            # Just logging the error
            raise
        log.debug('Version file download successful')
        # Writing version file to application data directory
        self._write_manifest_2_filesystem(decompressed_data)
        return decompressed_data
    except Exception as err:
        log.debug('Version file download failed')
        log.debug(err, exc_info=True)
        return None

Here is a sample of data I receive:

ipdb> data
b'{"type": "error", "error": {"message": "keys.gz"}}'

I believe you should open a ticket on https://github.com/JMSwag/PyUpdater and see if they can help you further.

回答3:

Requests is a pretty fantastic library, don't waste time with anything else, unless there is a really good reason:

import requests
import zlib

def download(url, username, password):
    r = requests.get(url, auth=requests.auth.HTTPBasicAuth(username, password))
    r.raise_for_status()
    return zlib.decompress(r.content, 15 + 32)

download('https://api.bitbucket.org/2.0/repositories/brofewfefwefewef/eee/downloads/keys.gz', 'brofewfefwefewef', your_password)

Also, it's probably worth noting that the credentials here aren't shouldn't be used anymore. Basic Auth can be decoded pretty simply.