Meteor: Saving images from urls to AWS S3 storage

Question

I am trying, server-side, to take an image from the web by it's url (i.e. http://www.skrenta.com/images/stackoverflow.jpg) and save this image to my AWS S3 bucket using Meteor, the aws-sdk meteorite package as well as the http meteor package.

This is my attempt, which indeed put a file in my bucket (someImageFile.jpg), but the image file is corrupted then and cannot be displayed by a browser or a viewer application.

Probably I am doing something wrong with the encoding of the file. I tried many combinations and none of them worked. Also, I tried adding ContentLength and/or ContentEncoding with different encodings like binary, hex, base64 (also in combination with Buffer.toString("base64"), none of them worked. Any advice will be greatly appreciated!

This is in my server-side-code:

var url="http://www.skrenta.com/images/stackoverflow.jpg";
HTTP.get(url, function(err, data) {
    if (err) {
        console.log("Error: " + err);
    } else {
        //console.log("Result: "+JSON.stringify(data));
        //uncommenting above line fills up the console with raw image data
        s3.putObject({
                ACL:"public-read",
                Bucket:"MY_BUCKET",
                Key: "someImageFile.jpg",
                Body: new Buffer(data.content,"binary"),
                ContentType: data.headers["content-type"], // = image/jpeg
                //ContentLength: parseInt(data.headers["content-length"]),
                //ContentEncoding: "binary"
            },
            function(err,data){ // CALLBACK OF HTTP GET
                if(err){
                    console.log("S3 Error: "+err);
                }else{
                    console.log("S3 Data: "+JSON.stringify(data));
                }
            }
    );
    }
});

Actually I am trying to use the filepicker.io REST API via HTTP calls, i.e. for storing a converted image to my s3, but for this problem this is the minimum example to demonstrate the actual problem.

Answer 1

The best solution is to look at what has already been done in this regard:

https://github.com/Lepozepo/S3

Also filepicker.so seems pretty simple:

Integrating Filepicker.IO with Meteor

Answer 2

After several trial an error runs I gave up on Meteor.HTTP and put together the code below, maybe it will help somebody when running into encoding issues with Meteor.HTTP.

Meteor.HTTP seems to be meant to just fetch some JSON or text data from remote APIs and such, somehow it seems to be not quiet the choice for binary data. However, the Npm http module definitely does support binary data, so this works like a charm:

var http=Npm.require("http");

url = "http://www.whatever.com/check.jpg";

var req = http.get(url, function(resp) {
    var buf = new Buffer("", "binary");
    resp.on('data', function(chunk) {
        buf = Buffer.concat([buf, chunk]);
    });
    resp.on('end', function() {
        var thisObject = {
            ACL: "public-read",
            Bucket: "mybucket",
            Key: "myNiceImage.jpg",
            Body: buf,
            ContentType: resp.headers["content-type"],
            ContentLength: buf.length
        };
        s3.putObject(thisObject, function(err, data) {
            if (err) {
                console.log("S3 Error: " + err);
            } else {
                console.log("S3 Data: " + JSON.stringify(data));
            }
        });
    });
});