'requests.post' Files Upload Large File: (

Question

Here I'm trying to use requests.post for file upload.

Wrote procedure.

import requests
def upload_file_to_gcs():
    url = 'http://127.0.0.1:8500/save-data-to-gcs/'
    f = {'file': ('Product_Master.csv', open('C:/Projects/bf/Product_Master.csv', 'rb')), 'file_name': 'Product_Master.csv'}
    r = requests.post(url, files=f)
    print r

upload_file_to_gcs()

Here is procedure written against url: save-data-to-gcs

Note: In this i'm reading file object using request.FILES

def save_data_to_gcs(request):
    file_name = '/gs/bucket-name/' + request.FILES['file'].name # change bucket/object names to suit your needs
    writable_file_name = files.gs.create(file_name, mime_type='application/octet-stream',
                                     acl='public-read')
    with files.open(writable_file_name, 'a') as f:
        f.write(request.FILES['file'].read())
    files.finalize(writable_file_name)
    return HttpResponse('', mimetype='application/text')

Above procedures working for less or equal to ~1.5 Mb size files. But if we go beyond ~2.0 MB then App Engine throwing an error:

Exception in request:
Traceback (most recent call last):
  File "/base/data/home/apps/s~bfu/101.371906891057843424/common/zip-packages/django-1.1.zip/django/core/handlers/base.py", line 92, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/s~bfu/101.371906891057843424/myapp/utils.py", line 50, in save_data_to_gcs
    logging.error(request.FILES['file'].name)
  File "/base/data/home/apps/s~bfu/101.371906891057843424/common/zip-packages/django-1.1.zip/django/utils/datastructures.py", line 203, in __getitem__
    raise MultiValueDictKeyError, "Key %r not found in %r" % (key, self)
MultiValueDictKeyError: "Key 'file' not found in <MultiValueDict: {}>"

Am i missing something here?, Please guide on the same.

Summary: Here am trying to upload a file through python on GCS (Google Cloud Storage).

Answer 1

This works fairly well for me (Python3 with requests):

def upload_file(local_file, remote_file):  
    params = {"file": os.path.basename(remote_file),
              "folder": os.path.dirname(remote_file),
              "submit": "Submit"}
    with open(local_file, 'rb') as file_:
        try:
           response = requests.post(url=URL, data=params, auth=(USER, PASSWORD),
                                    files={"zip_file": file_}, verify=False)
        except TimeoutError:
            print("Connection timed out!")
        else:
            print(response)