HttpWebRequest FileUpload causes OutOfMemoryExcept

问题:

I always get this OutOfMemoryException:

 System.OutOfMemoryException was not handled.
 Message=Exception of type 'System.OutOfMemoryException' was thrown.
 Source=System
 StackTrace:
    at System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
    at System.Net.ScatterGatherBuffers.Write(Byte[] buffer, Int32 offset, Int32 count)
    at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
    at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
    …

... and memory usage is steadily increasing (according to the Windows task manager) when I execute following source code:

Imports System.Net
Imports System.Text
Imports System.IO

Public Class HttpFileUploader

    Public Function uploadFile(ByVal containa As CookieContainer, ByVal uri As String, ByVal filePath As String, ByVal fileParameterName As String, ByVal contentType As String, ByVal otherParameters As Specialized.NameValueCollection) As String

        Dim boundary As String = "---------------------------" & DateTime.Now.Ticks.ToString("x")
        Dim newLine As String = System.Environment.NewLine
        Dim boundaryBytes As Byte() = System.Text.Encoding.ASCII.GetBytes(newLine & "--" & boundary & newLine)
        Dim request As Net.HttpWebRequest = Net.WebRequest.Create(uri)

        request.ContentType = "multipart/form-data; boundary=" & boundary
        request.Method = "POST"
        request.CookieContainer = containa
        request.AllowAutoRedirect = True
        request.Timeout = -1

        Using requestStream As IO.Stream = request.GetRequestStream()

            Dim formDataTemplate As String = "Content-Disposition: form-data; name=""{0}""{1}{1}{2}"

            For Each key As String In otherParameters.Keys

                requestStream.Write(boundaryBytes, 0, boundaryBytes.Length)
                Dim formItem As String = String.Format(formDataTemplate, key, newLine, otherParameters(key))
                Dim formItemBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(formItem)
                requestStream.Write(formItemBytes, 0, formItemBytes.Length)

            Next key

            requestStream.Write(boundaryBytes, 0, boundaryBytes.Length)

            Dim headerTemplate As String = "Content-Disposition: form-data; name=""{0}""; filename=""{1}""{2}Content-Type: {3}{2}{2}"
            Dim header As String = String.Format(headerTemplate, fileParameterName, filePath, newLine, contentType)
            Dim headerBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(header)
            requestStream.Write(headerBytes, 0, headerBytes.Length)

            Using fileStream As New IO.FileStream(filePath, IO.FileMode.Open, IO.FileAccess.Read)

                Dim buffer(4096) As Byte
                Dim bytesRead As Int32 = fileStream.Read(buffer, 0, buffer.Length)

                Do While (bytesRead > 0)
                    ' the exception is triggered by the following line:     
                    requestStream.Write(buffer, 0, bytesRead)
                    bytesRead = fileStream.Read(buffer, 0, buffer.Length)
                Loop

            End Using

            Dim trailer As Byte() = System.Text.Encoding.ASCII.GetBytes(newLine & "--" + boundary + "--" & newLine)
            requestStream.Write(trailer, 0, trailer.Length)
            requestStream.Close()

        End Using

        Dim response As Net.WebResponse = Nothing
        Dim responseText = ""

        Try

            response = request.GetResponse()

            Using responseStream As IO.Stream = response.GetResponseStream()

                Using responseReader As New IO.StreamReader(responseStream)

                    responseText = responseReader.ReadToEnd()

                End Using

            End Using

        Catch exception As Net.WebException

            MsgBox(exception.Message)

        Finally

            response.Close()
            response = Nothing
            request = Nothing

        End Try

        Return responseText

    End Function

End Class

I have marked with a comment the line that triggers the exception.

I'm using 10 threads for uploading different files (having sizes up to 250 MB).

Why am I running out of memory with this code?

回答1:

This thread seems to shed some light on large byte streams:

http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/1af59645-cdef-46a9-9eb1-616661babf90/

When using 10 threads to send 250mb files at once, you might actually be running out of memory! What spec machine are you using?

UPDATE

I'm all for having a go, so this is what I've done. I created a rapidshare account, but got bored in reading about the complexities of the upload api. Therefore I'm not actually sure if this code works or not. However, it does run. I noticed that running your code above, doesn't seem do do anything during the file read. However, as soon as GetResponseStream is called, it appears to upload the file there and then. Hmm, we want ot to upload it during our stream reading process.

So I tried the suggestion on Jeffrey Richter's Blog and had a quick look here as well for some other upload examples. Now I stated above that I didn't dive into the complexities of RapidShares upload api, rather than the streaming operation itself.

See the code below. It's identical apart from the introduction of a MemoryStream for the header information and I removed the footer content also (to get the byte count correct).

    Dim ms As New MemoryStream()
    Dim formDataTemplate As String = "Content-Disposition: form-data; name=""{0}""{1}{1}{2}"

    For Each key As String In otherParameters.Keys
        ms.Write(boundaryBytes, 0, boundaryBytes.Length)
        Dim formItem As String = String.Format(formDataTemplate, key, newLine, otherParameters(key))
        Dim formItemBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(formItem)
        ms.Write(formItemBytes, 0, formItemBytes.Length)
    Next key

    ms.Write(boundaryBytes, 0, boundaryBytes.Length)

    Dim headerTemplate As String = "Content-Disposition: form-data; name=""{0}""; filename=""{1}""{2}Content-Type: {3}{2}{2}"
    Dim header As String = String.Format(headerTemplate, fileParameterName, filePath, newLine, contentType)
    Dim headerBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(header)
    ms.Write(headerBytes, 0, headerBytes.Length)

    Dim length As Long = ms.Length
    length += New FileInfo(filePath).Length
    request.ContentLength = length

    Using requestStream As IO.Stream = request.GetRequestStream()
        Dim bheader() As Byte = ms.ToArray()
        requestStream.Write(bheader, 0, bheader.Length)
        Using fileStream As New IO.FileStream(filePath, IO.FileMode.Open, IO.FileAccess.Read)

            Dim buffer(4096) As Byte
            Dim bytesRead As Int32 = fileStream.Read(buffer, 0, buffer.Length)

            Do While (bytesRead > 0)
                requestStream.Write(buffer, 0, bytesRead)
                bytesRead = fileStream.Read(buffer, 0, buffer.Length)
            Loop
        End Using
        requestStream.Close()
    End Using

The whole point of this was to pre-allocate the ContentLength property so it starts streaming immediately. Rather than do the work at GetResponseStream, it does the work on the requestStream.Write command. The bytes in the MemoryStream are counted, then written, then the size of the file is added to the length variable and set as the ContentLength. Does this work?!

回答2:

Following code is working:

Imports System.Net
Imports System.Text
Imports System.IO

Public Class HttpFileUploader

    Public Function uploadFile(ByVal containa As CookieContainer, ByVal uri As String, ByVal filePath As String, ByVal fileParameterName As String, ByVal contentType As String, ByVal otherParameters As Specialized.NameValueCollection) As String

        Dim boundary As String = "---------------------------" & DateTime.Now.Ticks.ToString("x")
        Dim newLine As String = System.Environment.NewLine
        Dim boundaryBytes As Byte() = System.Text.Encoding.ASCII.GetBytes(newLine & "--" & boundary & newLine)
        Dim request As Net.HttpWebRequest = Net.WebRequest.Create(uri)

        request.ContentType = "multipart/form-data; boundary=" & boundary
        request.Method = "POST"
        request.CookieContainer = containa
        request.AllowAutoRedirect = True
        request.Timeout = -1
        request.KeepAlive = True
        request.AllowWriteStreamBuffering = False

        Dim ms As New MemoryStream()
        Dim formDataTemplate As String = "Content-Disposition: form-data; name=""{0}""{1}{1}{2}"

        For Each key As String In otherParameters.Keys
            ms.Write(boundaryBytes, 0, boundaryBytes.Length)
            Dim formItem As String = String.Format(formDataTemplate, key, newLine, otherParameters(key))
            Dim formItemBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(formItem)
            ms.Write(formItemBytes, 0, formItemBytes.Length)
        Next key

        ms.Write(boundaryBytes, 0, boundaryBytes.Length)

        Dim headerTemplate As String = "Content-Disposition: form-data; name=""{0}""; filename=""{1}""{2}Content-Type: {3}{2}{2}"
        Dim header As String = String.Format(headerTemplate, fileParameterName, filePath, newLine, contentType)
        Dim headerBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(header)
        ms.Write(headerBytes, 0, headerBytes.Length)

        Dim length As Long = ms.Length
        length += New FileInfo(filePath).Length
        request.ContentLength = length

        Using requestStream As IO.Stream = request.GetRequestStream()
            Dim bheader() As Byte = ms.ToArray()
            requestStream.Write(bheader, 0, bheader.Length)
            Using fileStream As New IO.FileStream(filePath, IO.FileMode.Open, IO.FileAccess.Read)

                Dim buffer(4096) As Byte
                Dim bytesRead As Int32 = fileStream.Read(buffer, 0, buffer.Length)

                Do While (bytesRead > 0)
                    requestStream.Write(buffer, 0, bytesRead)
                    bytesRead = fileStream.Read(buffer, 0, buffer.Length)
                Loop
            End Using
            requestStream.Close()
        End Using

        Dim response As Net.WebResponse = Nothing
        Dim responseText = ""

        Try

            response = request.GetResponse()

            Using responseStream As IO.Stream = response.GetResponseStream()

                Using responseReader As New IO.StreamReader(responseStream)

                    responseText = responseReader.ReadToEnd()

                End Using

            End Using

        Catch exception As Net.WebException

            MsgBox(exception.Message)

        Finally

            response.Close()
            response = Nothing
            request = Nothing
        End Try

        Return responseText

    End Function

End Class

Many many thanks to Tom and the guy from http://blogs.msdn.com/b/johan/archive/2006/11/15/are-you-getting-outofmemoryexceptions-when-uploading-large-files.aspx!!!

回答3:

Dim WebClient1 As New WebClient
WebClient1.UploadFile(...blablabla...)