Get the whole response body when the response is c

Question

I'm making a HTTP request and listen for "data":

response.on("data", function (data) { ... })

The problem is that the response is chunked so the "data" is just a piece of the body sent back.

How do I get the whole body sent back?

Answer 1

Over at https://groups.google.com/forum/?fromgroups=#!topic/nodejs/75gfvfg6xuc, Tane Piper provides a good solution very similar to scriptfromscratch's, but for the case of a JSON response:

  request.on('response',function(response){
     var data = [];
     response.on('data', function(chunk) {
       data.push(chunk);
     });
     response.on('end', function() {
       var result = JSON.parse(data.join(''))
       return result
     });
   });`

This addresses the issue that OP brought up in the comments section of scriptfromscratch's answer.

Answer 2

I never worked with the HTTP-Client library, but since it works just like the server API, try something like this:

var data = '';
response.on('data', function(chunk) {
  // append chunk to your data
  data += chunk;
});

response.on('end', function() {
  // work with your data var
});

See node.js docs for reference.

Answer 3

request.on('response', function (response) {
  var body = '';
  response.on('data', function (chunk) {
    body += chunk;
  });
  response.on('end', function () {
    console.log('BODY: ' + body);
  });
});
request.end();

Answer 4

The reason it's messed up is because you need to call JSON.parse(data.toString()). Data is a buffer so you can't just parse it directly.

Answer 5

If you don't mind using the request library

var request = require('request');
request('http://www.google.com', function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the google web page.
  }
})

Answer 6

In order to support the full spectrum of possible HTTP applications, Node.js's HTTP API is very low-level. So data is received chunk by chunk not as whole.
There are two approaches you can take to this problem:

1) Collect data across multiple "data" events and append the results
together prior to printing the output. Use the "end" event to determine
when the stream is finished and you can write the output.

var http = require('http') ;
http.get('some/url' , function (resp) {
    var respContent = '' ;
    resp.on('data' , function (data) {
        respContent += data.toString() ;//data is a buffer instance
    }) ;
    resp.on('end' ,  function() {
        console.log(respContent) ;
    }) ;
}).on('error' , console.error) ;

2) Use a third-party package to abstract the difficulties involved in
collecting an entire stream of data. Two different packages provide a
useful API for solving this problem (there are likely more!): bl (Buffer
List) and concat-stream; take your pick!

var http = require('http') ;
var bl = require('bl') ;

http.get('some/url', function (response) {
    response.pipe(bl(function (err, data) {
        if (err) {
            return console.error(err)
        }
        data = data.toString() ;
        console.log(data) ;
    })) 
}).on('error' , console.error) ;