Is it possible to store arbitrary data in GAE Gola

Question

I am creating a large database application in Google App Engine Go. Most of my pieces of data are small, so I have no problem storing them in Datastore. However, I know I will run into a few entries that will be a few megabytes big, so I will have to use Blobstore to save them.

Looking at the reference for Blobstore, it appears that the service was mainly intended to be used for files being uploaded to the service. What are the functions I need to call to store arbitrary data in the Blobstore like I would in Datastore? I can already convert the data to []byte and I don't need to index anything in the blob, just to store and fetch it by ID.

Answer 1

As @yumaikas said, the Files API is deprecated. If this data comes from some sort of a user upload, you should modify the upload form to work with Blobstore Upload URLs (in particular, setting the encoding to multipart/form-data or multipart/mixed and naming all file upload fields file, except the ones that you don't want to be stored in blobstore).

However, if that is not possible (e.g. you don't have control over the user input, or you have to pre-process the data on the server before you store it in Blobstore), then you'll either have to use the deprecated Files API, or upload the data using the URLFetch API.

Here's a complete example application that will store a sample file for you in Blobstore.

package sample

import (
    "bytes"
    "net/http"
    "mime/multipart"

    "appengine"
    "appengine/blobstore"
    "appengine/urlfetch"
)

const SampleData = `foo,bar,spam,eggs`

func init() {
    http.HandleFunc("/test", StoreSomeData)
    http.HandleFunc("/upload", Upload)
}

func StoreSomeData(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)

    // First you need to create the upload URL:
    u, err := blobstore.UploadURL(c, "/upload", nil)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("%s", err)
        return
    }

    // Now you can prepare a form that you will submit to that URL.
    var b bytes.Buffer
    fw := multipart.NewWriter(&b)
    // Do not change the form field, it must be "file"!
    // You are free to change the filename though, it will be stored in the BlobInfo.
    file, err := fw.CreateFormFile("file", "example.csv")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("%s", err)
        return
    }
    if _, err = file.Write([]byte(SampleData)); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("%s", err)
        return
    }
    // Don't forget to close the multipart writer.
    // If you don't close it, your request will be missing the terminating boundary.
    fw.Close()

    // Now that you have a form, you can submit it to your handler.
    req, err := http.NewRequest("POST", u.String(), &b)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("%s", err)
        return
    }
    // Don't forget to set the content type, this will contain the boundary.
    req.Header.Set("Content-Type", fw.FormDataContentType())

    // Now submit the request.
    client := urlfetch.Client(c)
    res, err := client.Do(req)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("%s", err)
        return
    }

    // Check the response status, it should be whatever you return in the `/upload` handler.
    if res.StatusCode != http.StatusCreated {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("bad status: %s", res.Status)
        return
    }
    // Everything went fine.
    w.WriteHeader(res.StatusCode)
}

func Upload(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)

    // Here we just checked that the upload went through as expected.
    if _, _, err := blobstore.ParseUpload(r); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        c.Errorf("%s", err)
        return
    }
    // Everything seems fine. Signal the other handler using the status code.
    w.WriteHeader(http.StatusCreated)
}

Now if you curl http://localhost:8080/test, it will store a file in the Blobstore.

Important: I'm not exactly sure how you would be charged for bandwidth for the request that you make to your own app. At the worst case, you will be charged for internal traffic, which is cheaper from normal bandwidth iirc.

Answer 2

There are two ways that you could write files to the blobstore

One is to use a deprecated API documented at the end of the page for the blobstore. Their example code is below.

The approach that they are going to be switching to is storing files in Google cloud storage and serving them via the blobstore.

The other approach would be to simulate a user upload in some fashion. Go has an http client that can send files to be uploaded to web addresses. That would be a hacky way to do it though.

var k appengine.BlobKey
w, err := blobstore.Create(c, "application/octet-stream")
if err != nil {
        return k, err
}
_, err = w.Write([]byte("... some data ..."))
if err != nil {
        return k, err
}
err = w.Close()
if err != nil {
        return k, err
}
return w.Key()