MultiPartParserError :- Invalid boundary

Question

Im trying to send some data and file using Python requests module to my django rest application but get the below error.

    raise MultiPartParserError('Invalid boundary in multipart: %s' % boundary)
MultiPartParserError: Invalid boundary in multipart: None

Code:-

import requests
payload={'admins':[
                    {'first_name':'john'
                    ,'last_name':'white'
                    ,'job_title':'CEO'
                    ,'email':'[e-mail]'
                    },
                    {'first_name':'lisa'
                    ,'last_name':'markel'
                    ,'job_title':'CEO'
                    ,'email':'[e-mail]'
                    }
                    ],
        'company-detail':{'description':'We are a renowned engineering company'
                    ,'size':'1-10'
                    ,'industry':'Engineering'
                    ,'url':'http://try.com'
                    ,'logo':''
                    ,'addr1':'1280 wick ter'
                    ,'addr2':'1600'
                    ,'city':'rkville'
                    ,'state':'md'
                    ,'zip_cd':'12000'
                    ,'phone_number_1':'408-393-254'
                    ,'phone_number_2':'408-393-221'
                    ,'company_name':'GOOGLE'}
        }
files = {'upload_file':open('./test.py','rb')}
import json
headers = {'content-type' : 'application/json'}      
headers = {'content-type' : 'multipart/form-data'}      

#r = requests.post('http://127.0.0.1:8080/api/create-company-profile/',data=json.dumps(payload),headers=headers,files=files)
r = requests.post('http://127.0.0.1:8080/api/create-company-profile/',data=payload,headers=headers,files=files)
print r.status_code
print r.text

Django code:-

class CompanyCreateApiView(CreateAPIView):
    parser_classes = (MultiPartParser, FormParser,)
    def post(self, request, *args, **kwargs):
        print 'request ==', request.data

Answer 1

I tried your code with a CreateAPIView from the django-rest-framework. After fixing all the preliminary errors that your code produces, I could not reproduce a boundary error.

Directory structure:

my_site/ 
     myapp/
         views.py
         serializers.py
         urls.py
         models.py
     mysite/
         settings.py
         urls.py

my_app/views.py:

from rest_framework.generics import CreateAPIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, FormParser

from myapp.serializers import CompanySerializer 
from myapp.models import Company

class CompanyCreateApiView(CreateAPIView):
    parser_classes = (MultiPartParser, FormParser,)  #Used to parse the Request.

    queryset = Company.objects.all()  #The contents of the Response.
    serializer_class = CompanySerializer  #Determines how the contents of the Response will be converted to json. 
                                          #Required. Defined in myapp/serializers.py

    def post(self, request, *args, **kwargs):
        print('data ==', request.data)
        print('myjson ==', request.data["myjson"].read())
        print('mydata ==', request.data["mydata"].read())

        queryset = self.get_queryset()
        serializer = CompanySerializer(queryset, many=True)

        return Response(serializer.data)

myapp/serializers.py:

from rest_framework import serializers
from myapp.models import Company

#Directions for converting a model instance into json:
class CompanySerializer(serializers.ModelSerializer):
    class Meta:
        model = Company  #The model this serializer applies to.

        #By default all fields are converted to json.  
        #To limit which fields should be converted:
        fields = ("name", "email")
        #Or, if its easier you can do this:
        #exclude = ('id',)

myapp/urls.py:

from django.conf.urls import url

from . import views

urlpatterns = (
    url(r'^company/', views.CompanyCreateApiView.as_view() ),
)

my_site/urls.py:

from django.conf.urls import include, url
from django.contrib import admin


urlpatterns = [
    url(r'^admin/', include(admin.site.urls) ),

    url(r'^myapp/', include("myapp.urls") ),
]

myapp/models.py:

from django.db import models

# Create your models here.

class Company(models.Model):
    name = models.CharField(max_length=50)
    email = models.CharField(max_length=50)

    def __str__(self):
        return "{} {}".format(self.name, self.email)

my_site/settings.py:

...
...
DEFAULT_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
)

THIRD_PARTY_APPS = (
    'rest_framework',

)

LOCAL_APPS = (
    'myapp',

)

INSTALLED_APPS = DEFAULT_APPS + THIRD_PARTY_APPS + LOCAL_APPS

REST_FRAMEWORK = {
    # Use Django's standard `django.contrib.auth` permissions,
    # or allow read-only access for unauthenticated users.
    #'DEFAULT_PERMISSION_CLASSES': [
    #    'rest_framework.permissions.DjangoModelPermissionsOrAnonReadOnly'
    #]
}

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
)
    ...
    ...

Note that I didn't have to disable csrf tokens.

requests_client.py:

import requests

import json
from io import StringIO 

my_dict = {
    'admins': [
                {'first_name':'john'
                ,'last_name':'white'
                ,'job_title':'CEO'
                ,'email':'[e-mail]'
                },

                {'first_name':'lisa'
                ,'last_name':'markel'
                ,'job_title':'CEO'
                ,'email':'[e-mail]'
                }

    ],
    'company-detail': {
                'description': 'We are a renowned engineering company'
                ,'size':'1-10'
                ,'industry':'Engineering'
                ,'url':'http://try.com'
                ,'logo':''
                ,'addr1':'1280 wick ter'
                ,'addr2':'1600'
                ,'city':'rkville'
                ,'state':'md'
                ,'zip_cd':'12000'
                ,'phone_number_1':'408-393-254'
                ,'phone_number_2':'408-393-221'
                ,'company_name':'GOOGLE'
    }
}

url = 'http://localhost:8000/myapp/company/'

#StringIO creates a file-like object in memory, rather than on disk:

#python3.4:
#with StringIO(json.dumps(my_dict)) as json_file, open("data.txt", 'rb') as data_file:

#python2.7:
with StringIO(json.dumps(my_dict).decode('utf-8')) as json_file, open("data.txt", 'rb') as data_file:

    myfiles = [
        ("mydata", ("data.txt", data_file, "text/plain")),
        ("myjson", ("json.json", json_file, "application/json")),
    ]

    r = requests.post(url, files=myfiles) 

print(r.status_code)
print(r.text)

Output in request_client.py terminal window:

200 
[{"name":"GE","email":"[e-mail]"},{"name":"APPL","email":"[e-mail]"}]

Output in django server window:

...
...
Quit the server with CONTROL-C.

data == <QueryDict: {'mydata': [<InMemoryUploadedFile: data.txt (text/plain)>], 
'myjson': [<InMemoryUploadedFile: json.json (application/json)>]}>

myjson == b'{"admins": [{"first_name": "john", "last_name": "white", "email": "[e-mail]", "job_title": "CEO"}, 
{"first_name": "lisa", "last_name": "markel", "email": "[e-mail]", "job_title": "CEO"}], "company-detail": 
{"description": "We are a renowned engineering company", "phone_number_2": "408-393-221", "phone_number_1": "408-393-254",
 "addr2": "1600", "addr1": "1280 wick ter", "logo": "", "size": "1-10", "city": "rkville", "url": "http://try.com", 
"industry": "Engineering", "state": "md", "company_name": "GOOGLE", "zip_cd": "12000"}}'

mydata == b'line 1\nline 2\nline 3\nline 4\n'

[18/Dec/2015 13:41:57] "POST /myapp/company/ HTTP/1.1" 200 75

Answer 2

In order to use requests, I had to disable csrf tokens in my django project, otherwise I got errors when sending post requests:

CSRF verification failed. Request aborted.

You are seeing this message because this site requires a CSRF cookie when submitting forms. This cookie is required for security reasons, to ensure that your browser is not being hijacked by third parties.

settings.py:

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    #'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

Next, your payload is not json data. json data is a string, but your payload is a python dictionary. And, you can't just wrap quotes around your dictionary because json data cannot contain single quotes--which you used exclusively. So, you need to convert your python dictionary to json.

Here's the requests client:

import requests
import json
from io import StringIO 

my_dict = {
    'admins': [
                {'first_name':'john'
                ,'last_name':'white'
                ,'job_title':'CEO'
                ,'email':'[e-mail]'
                },

                {'first_name':'lisa'
                ,'last_name':'markel'
                ,'job_title':'CEO'
                ,'email':'[e-mail]'
                }

    ],
    'company-detail': {
                'description': 'We are a renowned engineering company'
                ,'size':'1-10'
                ,'industry':'Engineering'
                ,'url':'http://try.com'
                ,'logo':''
                ,'addr1':'1280 wick ter'
                ,'addr2':'1600'
                ,'city':'rkville'
                ,'state':'md'
                ,'zip_cd':'12000'
                ,'phone_number_1':'408-393-254'
                ,'phone_number_2':'408-393-221'
                ,'company_name':'GOOGLE'
    }
}

url = 'http://localhost:8000/myapp/upload/'


#StringIO creates a file-like object in memory, rather than on disk:
with StringIO(json.dumps(my_dict)) as json_file, open("data.txt", 'rb') as data_file:

    myfiles = [
        ("mydata", ("data.txt", data_file, "text/plain")),
        ("myjson", ("json.json", json_file, "application/json")),
    ]

    r = requests.post(url, files=myfiles) 

print(r.text)

See the Advanced Usage section of the requests docs.

Here it is with a django class based view (albeit not a Django Rest Framework class view):

from django.shortcuts import render
from django.http import HttpResponse

# Create your views here.

from django.views.generic import View

from django.views.generic import View

class FileUploadView(View):
    def post(self, request):
        file_dict = request.FILES
        print(file_dict)

        for name in file_dict:
            print(name)
            print(file_dict[name].read())
            print('*' * 50)

        return HttpResponse('thanks')

Output in client window:

django186p34)~/python_programs$ python django_client.py 
thanks
(django186p34)~/python_programs$ 

Output in django server window:

...
...
Quit the server with CONTROL-C.

<MultiValueDict: {'myjson': [<InMemoryUploadedFile: json.json (application/json)>], 
'mydata': [<InMemoryUploadedFile: data.txt (text/plain)>]}>
**************************************************
myjson
b'{"admins": [{"job_title": "CEO", "last_name": "white", "first_name": "john", "email": "[e-mail]"}, 
{"job_title": "CEO", "last_name": "markel", "first_name": "lisa", "email": "[e-mail]"}], "company-detail": 
{"description": "We are a renowned engineering company", "city": "rkville", "state": "md", "company_name": "GOOGLE", 
"addr1": "1280 wick ter", "url": "http://try.com", "phone_number_2": "408-393-221", "industry": "Engineering", "logo": "", "addr2": "1600",
 "phone_number_1": "408-393-254", "size": "1-10", "zip_cd": "12000"}}'
**************************************************
mydata
b'line 1\nline 2\nline 3\nline 4\n'
**************************************************
[16/Dec/2015 07:34:06] "POST /myapp/upload/ HTTP/1.1" 200 6

Apparently, the requests <--> django mind meld allows you to mix POST data with multipart/form-data, and you can do this:

with open('data.txt', 'rb') as f:
    myfile = {'myfile': f}
    r = requests.post(url, data={"hello": "world"}, files=myfile) 

print(r.text)

Coupled with this view:

from django.http import HttpResponse
from django.views.generic import View

class FileUploadView(View):
    def post(self, request):
        x = request.POST
        print(x)
        print(x["hello"])

        file_dict = request.FILES
        print(file_dict)
        print('*' * 50)

        for name in file_dict:
            print(name)
            print(file_dict[name].read())
            print('*' * 50)

        return HttpResponse('thanks')

I get the following output in the server window:

...
...
Quit the server with CONTROL-C.

<QueryDict: {'hello': ['world']}>
world
<MultiValueDict: {'myfile': [<InMemoryUploadedFile: data.txt ()>]}>
**************************************************
myfile
b'line 1\nline 2\nline 3\nline 4\n'
**************************************************
[16/Dec/2015 8:04:17] "POST /myapp/upload/ HTTP/1.1" 200 6

To allow for UTF-8 characters in your dictionary, you will have to take some extra steps when converting to json. See here.

Answer 3

When uploading file with parameters:

1. Don't overwrite the headers

2. Put other parameters together with upload_file in files dict.

   input ={"md5":"xxxx","key":"xxxxx","sn":"xxxx"}
   files = {"pram1":"abc",
            "pram2":json.dumps(input),
            "upload_file": open('/home/gliu/abc', 'rb')}    
   res = requests.post(url, files=files)
   

Answer 4

Okay, I forgot about your headers. According to the spec:

Content-Type   = "Content-Type" ":" media-type

MIME provides for a number of "multipart" types -- encapsulations of one or more entities within a single message-body. All multipart types share a common syntax, ... and MUST include a boundary parameter as part of the media type value.

Here is what a request containing multipart/form-data looks like:

POST /myapp/company/ HTTP/1.1
Host: localhost:8000
Content-Length: 265
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.9.0
Connection: keep-alive
Content-Type: multipart/form-data; boundary=63c5979328c44e2c869349443a94200e   

--63c5979328c44e2c869349443a94200e
Content-Disposition: form-data; name="hello"

world
--63c5979328c44e2c869349443a94200e
Content-Disposition: form-data; name="mydata"; filename="data.txt"

line 1
line 2
line 3
line 4

--63c5979328c44e2c869349443a94200e--

See how the sections of data are separated by the boundary:

--63c5979328c44e2c869349443a94200e--

The idea is to use something for a boundary that is unlikely to appear in the data. Note that the boundary was included in the Content-Type header of the request.

That request was produced by this code:

import requests

myfile = {'mydata': open('data.txt','rb')}

r = requests.post(url, 
        #headers = myheaders
        data = {'hello': 'world'}, 
        files = myfile
) 

It looks like you were paying careful attention to the following note in the django-rest-framework docs:

Note: When developing client applications always remember to make sure you're setting the Content-Type header when sending data in an HTTP request.

If you don't set the content type, most clients will default to using 'application/x-www-form-urlencoded', which may not be what you wanted.

But when you are using requests, if you specify the Content-Type header yourself, then requests assumes that you know what you're doing, and it doesn't overwrite your Content-Type header with the Content-Type header it would have provided.

You didn't provide the boundary in your Content-Type header--as required. How could you? You didn't assemble the body of the request and create a boundary to separate the various pieces of data, so you couldn't possibly know what the boundary is.

When the django-rest-framework note says that you should include a Content-Type header in your request, what that really means is:

You or any programs you use to create the request need to include a Content-Type header.

So @AChampion was exactly right in the comments: let requests provide the Content-Type header, after all the requests docs advertise:

Requests takes all of the work out of Python HTTP/1.1

requests works like this: if you provide a files keyword arg, then requests uses a Content-Type header of multipart/form-data and also specifies a boundary in the header; then requests assembles the body of the request using the boundary. If you provide a data keyword argument then requests uses a Content-Type of application/x-www-form-urlencoded, which just assembles all the keys and values in the dictionary into this format:

x=10&y=20

No boundary required.

And, if you provide both a files keyword arg and a data keyword arg, then requests uses a Content-Type of multipart/form-data.

Answer 5

By the way, here is a less painful way to read django's voluminous html error responses when using requests while developing:

$ python requests_client.py > error.html  (send output to the file error.html)

Then in your browser do:

File > Open File

and navigate to error.html. Read the error description in your browser, then track down what's wrong in your django project.

Then come back to your terminal window and hit the up arrow key on your keyboard to get back to:

$ python requests_client.py > error.html 

Hit Return, then refresh the browser window that is displaying error.html. Repeat as necessary. (Don't make the mistake of repeatedly refreshing your browser and thinking that is sending a new request--you have to run request_client.py again to send a new request).