Fetch vs Request

Question

I'm consuming a JSON stream and am trying to use fetch to consume it. The stream emits some data every few seconds. Using fetch to consume the stream gives me access to the data only when the stream closes server side. For example:

var target; // the url.
var options = {
  method: "POST",
  body: bodyString,
} 
var drain = function(response) {
  // hit only when the stream is killed server side.
  // response.body is always undefined. Can't use the reader it provides.
  return response.text(); // or response.json();
};
var listenStream = fetch(target, options).then(drain).then(console.log).catch(console.log);

/*
    returns a data to the console log with a 200 code only when the server stream has been killed.
*/

However, there have been several chunks of data already sent to the client.

Using a node inspired method in the browser like this works every single time an event is sent:

var request = require('request');
var JSONStream = require('JSONStream');
var es = require('event-stream');

request(options)
.pipe(JSONStream.parse('*'))
.pipe(es.map(function(message) { // Pipe catches each fully formed message.
      console.log(message)
 }));

What am I missing? My instinct tells me that fetch should be able to mimic the pipe or stream functionality.

Answer 1

response.body gives you access to the response as a stream. To read a stream:

fetch(url).then(response => {
  const reader = response.body.getReader();

  reader.read().then(function process(result) {
    if (result.done) return;
    console.log(`Received a ${result.value.length} byte chunk of data`);
    return reader.read().then(process);
  }).then(() => {
    console.log('All done!');
  });
});

Here's a working example of the above.

Fetch streams are more memory-efficient than XHR, as the full response doesn't buffer in memory, and result.value is a Uint8Array making it way more useful for binary data. If you want text, you can use TextDecoder:

fetch(url).then(response => {
  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  reader.read().then(function process(result) {
    if (result.done) return;
    const text = decoder.decode(result.value, {stream: true});
    console.log(text);
    return reader.read().then(process);
  }).then(() => {
    console.log('All done!');
  });
});

Here's a working example of the above.

Soon TextDecoder will become a transform stream, allowing you to do response.body.pipeThrough(new TextDecoder()), which is much simpler and allows the browser to optimise.

As for your JSON case, streaming JSON parsers can be a little big and complicated. If you're in control of the data source, consider a format that's chunks of JSON separated by newlines. This is really easy to parse, and leans on the browser's JSON parser for most of the work. Here's a working demo, the benefits can be seen at slower connection speeds.

I've also written an intro to web streams, which includes their use from within a service worker. You may also be interested in a fun hack that uses JavaScript template literals to create streaming templates.

Answer 2

Turns out I could get XHR to work - which doesn't really answer the request vs. fetch question. It took a few tries and the right ordering of operations to get it right. Here's the abstracted code. @jaromanda was right.

var _tryXhr = function(target, data) {
  console.log(target, data);
  var xhr = new XMLHttpRequest();

  xhr.onreadystatechange = function () {
    console.log("state change.. state: "+ this.readyState);
    console.log(this.responseText);
    if (this.readyState === 4) {
      // gets hit on completion.
    }
    if (this.readyState === 3) {
       // gets hit on new event
    }
  };

  xhr.open("POST", target);
  xhr.setRequestHeader("cache-control", "no-cache");
  xhr.setRequestHeader("Content-Type", "application/json");
  xhr.send(data);   
};