Speed up fetching posts for my social network app

Question

I have an array of keys which lead to post objects for my social network like so /posts/id/(post info)

When I load the posts I load /posts/0 and then /posts/1 etc using the observeSingleEventOfType(.Value) method.

I use a lazyTableView to load 30 at a time and it is quite slow. Is there any way I can use one of the query methods or another way of making it faster even if I have to restructure the data in my JSON tree.

I am coming from Parse re-implementing my app and so far the experience as been quite good. Just this one thing I am a bit stuck on. Thanks in advance for the help!

EDIT:

func loadNext(i: Int) { 

    // check if exhists
    let ideaPostsRef = Firebase(url: "https://APPURL")

    ideaPostsRef.childByAppendingPath(i.description).observeSingleEventOfType(.Value, withBlock: {
        (snapshot) in

        if i % 29 == 0 && i != 0 && !self.hitNull { return }
            // false if nil
            // true if not nil
        if !(snapshot.value is NSNull) {
            let postJSON  = snapshot.value as! [String: AnyObject]
            print("GOT VALID \(postJSON)")
            let post = IdeaPost(message: postJSON["message"] as! String, byUser: postJSON["user"] as! String, withId: i.description)
            post.upvotes = postJSON["upvotes"] as! Int
            self.ideaPostDataSource.append(post)
            self.loadNext(i + 1)
        } else {
            // doesn't exhist
            print("GOT NULL RETURNING AT \(i)")
            self.doneLoading = true
            self.hitNull = true
            return
        }
    }
}

This recursive function essentially runs getting the value for key number i from firebase. If it is NSNULL it knows that is the last possible post to load and never does again. If NSNULL doesn't get hit but i % 29 == 0 then it returns as a base case so only 30 posts are loaded at a time (0 indexed). When I set doneLoading to true tableView.reloadData() is called using a property observer.

Here is a sample of what the array I am fetching looks like

"ideaPosts" : [ {
    "id" : 0,
    "message" : "Test",
    "upvotes" : 1,
    "user" : "Anonymous"
  }, {
    "id" : 1,
    "message" : "Test2",
    "upvotes" : 1,
    "user" : "Anonymous"
  } ]

Answer 1

Update: we now also cover this question in an AskFirebase episode.

Loading many items from Firebase doesn't have to be slow, since you can pipeline the requests. But your code is making this impossible, which indeed will lead to suboptimal performance.

In your code, you request an item from the server, wait for that item to return and then load the next one. In a simplified sequence diagram that looks like:

Your app                     Firebase 
                             Database

        -- request item 1 -->
                               S  L
                               e  o
                               r  a
                               v  d
                               e  i
        <-  return item  1 --  r  n
                                  g
        -- request item 2 -->
                               S  L
                               e  o
                               r  a
                               v  d
                               e  i
                               r  n
        <-  return item  2 --     g
        -- request item 3 -->
                 .
                 .
                 .
        -- request item 30-->
                               S  L
                               e  o
                               r  a
                               v  d
                               e  i
                               r  n
                                  g
        <-  return item 30 --

In this scenario you're waiting for 30 times your roundtrip time + 30 times the time it takes to load the data from disk. If (for the sake of simplicity) we say that roundtrips take 1 second and loading an item from disk also takes one second that least to 30 * (1 + 1) = 60 seconds.

In Firebase applications you'll get much better performance if you send all the requests (or at least a reasonable number of them) in one go:

Your app                     Firebase 
                             Database

        -- request item 1 -->
        -- request item 2 -->  S  L
        -- request item 3 -->  e  o
                 .             r  a
                 .             v  d
                 .             e  i
        -- request item 30-->  r  n
                                  g
        <-  return item  1 --     
        <-  return item  2 --      
        <-  return item  3 --
                 .
                 .
                 .
        <-  return item 30 --

If we again assume a 1 second roundtrip and 1 second of loading, you're waiting for 30*1 + 1 = 31 seconds.

So: all requests go through the same connection. Given that, the only difference between get(1), get(2), get(3) and getAll([1,2,3]) is some overhead for the frames.

I set up a jsbin to demonstrate the behavior. The data model is very simple, but it shows off the difference.

function loadVideosSequential(videoIds) {
  if (videoIds.length > 0) {
    db.child('videos').child(videoIds[0]).once('value', snapshot => {
      if (videoIds.length > 1) {
        loadVideosSequential(videoIds.splice(1), callback)
      }
    });
  }
}

function loadVideosParallel(videoIds) {
  Promise.all(
    videoIds.map(id => db.child('videos').child(id).once('value'))
  );
}

For comparison: sequentially loading 64 items takes 3.8 seconds on my system, while loading them pipelined (as the Firebase client does natively) it takes 600ms. The exact numbers will depend on your connection (latency and bandwidth), but the pipelined version should always be significantly faster.