Why multipart is not generating close events

Question

Hi I won't to handle upload stream by myself without touching a disk drive. So, the natural selection for me was multiparty module.

I took the general example and according the instruction from page https://npmjs.org/package/multiparty I changed form.parse to non callback request. In that case the disk won't be touched.

My code looks like this:

multiparty = require("multiparty")
http = require("http")
util = require("util")

# show a file upload form
http.createServer((req, res) ->
  if req.url is "/upload" and req.method is "POST"
    form = new multiparty.Form()

    form.on 'error', (err) ->
      console.log "Error received #{err}"

    form.on 'aborted',  ->
      console.log "Aborted"

    form.on 'part', (part) ->
      console.log "Part"

    form.on 'close', (part) ->
      console.log "close received"
      res.writeHead 200,
        "content-type": "text/plain"
      res.end "received upload:\n\n"

    form.on 'progress', (bytesReceived, bytesExpected) ->
      console.log "Received #{bytesReceived}, #{bytesExpected}"

    form.parse req
  else
    res.writeHead 200,
      "content-type": "text/html"

    res.end "<form action=\"/upload\" enctype=\"multipart/form-data\" method=\"post\">" + "<input type=\"text\" name=\"title\"><br>" + "<input type=\"file\" name=\"upload\" multiple=\"multiple\"><br>" + "<input type=\"submit\" value=\"Upload\">" + "</form>"
).listen 8080 

the console output looks like this:

Part
Part
Received 64983, 337353
Received 130519, 337353
Aborted
Error received Error: Request aborted

The close event is not generated so I don't know when the end of reading the socket. If I change the line:

form.parse req

to:

form.parse req, (err, fields, files) ->
  res.writeHead 200,
    "content-type": "text/plain"

  res.write "received upload:\n\n"
  res.end util.inspect(
    fields: fields
    files: files
  )

Then everything is fine and the close event is called. But the file is stored on the disk. The console looks like this:

Part
Part
Received 65536, 337353
Received 131072, 337353
Received 196608, 337353
Received 262144, 337353
Received 327680, 337353
Received 337353, 337353
close received

Any idea what's wrong?

Answer 1

I think that you need at least to read any received part somewhere in order to complete the request processing, and trigger the close event :

form.on('part', function(part) {
    part.pipe(fs.createWriteStream('/dev/null'));
});

Without this, I guess it is like your request is still waiting to be processed. (Note that there is more elegant, less scripty way than that "> /dev/null" stuff, but it is just as an example ;)

Answer 2

In this case, since you are not piping the data to a file, close is emitted only when all data has been piped out of the req object, causing req to internally emit it's finish event, which triggers the multiparty close event.

In practice, if you need the close event, always pull the data out of the part, whether it is to a blackhole or a meaningful location (file or temporary holding stream). This also means it can be very tricky to use the close event for continuing control flow.

If you don't want to do anything with the data you can blackhole it like so:

form.on("part", function(part) {
    out = new stream.Writable(); // require("stream") to get this native class
    out._write = function (chunk, encoding, done) {
        done(); // Don't do anything with the data
    };
    part.pipe(out);
});

The confusing part is that code which does not access the part will call the close event for small files, but not be called on large files (how I ran into this issue). The reason for that is that the close event isn't bound to whether you have read the data from the part but whether all data has been transferred out of the req.

Node streams only transmit a few chunks at a time between their buffers. Thus, small files are able to transfer their entire buffer from req to part without filling part's internal buffer. Larger files will fill the buffer of the part but still have data left over in the req, requiring a read on the part to get the data remaining in the req buffer.

This fact also makes it tricky to use the close event for anything regarding control flow. In example, a part can still be processing, but close will be sent, again because close isn't dependent on part completion, but rather on req being empty. Confusing!