Send data by parts in node.js express

Question

I was searching the web and documentation for node.js express module and it seems there is no way to send data by parts. I have a file rendered not very fast and I want to send parts of it before everything is rendered.

So here are my questions:

1. Is there a method on response to send data by parts?
2. What does response.end()?
3. If there is no way to send data by parts - what is the rationale behind? I would say it looks more blocking than non-blocking if that's true. Browser can load information faster if data is given earlier.

Sample simplified code:

app.get(..) {
  renderFile(file, function(data) {
    response.send(data);
  });
  response.end();
)

This piece of code sends only the first chunk of data. I checked - data is given correctly and callback is called more than one time.

Of course I can append data to the one variable and then write response.send(data); but I don't like this approach - it is not the way it should work.

Answer 1

Express extends Connect which extends HTTP I believe. Looking at the HTTP API, it seems that what you are looking for is response.write, which takes a chunk of data to send as a response body. I then believe you can use response.end on the end signal.

Answer 2

The response object is a writable stream. Just write to it, and Express will even set up chunked encoding for you automatically.

response.write(chunk);

You can also pipe to it if whatever is creating this "file" presents itself as a readable stream. http://nodejs.org/api/stream.html

Answer 3

The response object can be written to. In your case, the following would be a solution:

renderFile(file, function(data) {
  response.send(data);
});
response.end();

If you have the opportunity to get the file as a stream this can be streamlined further:

file.getReadableStream().pipe(response);

Answer 4

Besides using Response.write() method to chunk send and Response.end() to mark the end!

If you have a stream! You can pipe it directly to the Response object! And all get handled automatically!

See the example bellow for a file readable stream:

var http = require('http');
var fs = require('fs');

http.createServer(function(req, res) {
  // The filename is simple the local directory and tacks on the requested url
  var filename = __dirname+req.url;

  // This line opens the file as a readable stream
  var readStream = fs.createReadStream(filename);

  // This will wait until we know the readable stream is actually valid before piping
  readStream.on('open', function () {
    // This just pipes the read stream to the response object (which goes to the client)
    readStream.pipe(res); // <====== here
  });

  // This catches any errors that happen while creating the readable stream (usually invalid names)
  readStream.on('error', function(err) {
    res.end(err);
  });
}).listen(8080);

You can do that with any Stream. Just pipe it to Respond object!

The example above come from here:

https://nodejs.org/en/knowledge/advanced/streams/how-to-use-fs-create-read-stream/

Bottom line

Use pipe! And it's nice too look and understand better!

Here the link from the doc!

https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options

The readable.pipe() method attaches a Writable stream to the readable, causing it to switch automatically into flowing mode and push all of its data to the attached Writable.

And note about chaining

it is possible to attach multiple Writable streams to a single Readable stream.

The readable.pipe() method returns a reference to the destination stream making it possible to set up chains of piped streams

example:

const fs = require('fs');
const r = fs.createReadStream('file.txt'); // read stream
const z = zlib.createGzip(); // write stream
const w = fs.createWriteStream('file.txt.gz'); // write stream
r.pipe(z).pipe(w);

Read the doc for more!

Answer 5

the easy way to do this is to convert the data into a readable stream and pipe it to the response object. here is my nodejs code. hope this might help

var express= require('express');
var bodyParser = require('body-parser');
const JSONStreamStringify = require('json-stream-stringify');

var app=express();
var jsonParser = bodyParser.json();

app.post('/node_test_pipe', jsonParser, function (req, res) {
    console.log("Request from:"+req.url);
    var obj={
             my:1,
             two:2
            }
    JSONStreamStringify(obj).pipe(res);
    var obj={
             my:3,
             two:4
            }
    JSONStreamStringify(obj).pipe(res);
    });

app.listen(8080);