AngularFire2 infinite scrolling

Question

I'm trying to implement an infinite scrolling with Ionic2 and Firebase.

I use AngularFire2. What I'd like to do is to add new items to the fetched list and not to reload the whole list.

let query$:Observable<any> = this.af.database.list(`quests/`, {
    query: {
        orderByChild: 'date_published',
        limitToFirst: this.recentChunkLimit$ //Subject where I push new limit length
    }
}).publishReplay(1).refCount();

However, when I query list like that, the whole list is reloaded each time via websockets making each next update slower and slower. Here is a screenshot of Network websockets tab: And also I noticed that requests are made 2 times for each next chunk (though I put publishReplay). And it is happening in all apps where I used AngularFire2. I might misunderstand something though. I definitely need some clarification.

//==========Edit============

Now, I somehow managed to implement what I want without reloading the whole list each time. Not the best implementation but it works. Basically, I created an observable array and load new values into it by subscribing to the next chunk observable (where I also get the last element to start with). However the later problem still remains - in the sockets display I get data requested 2 times.

Answer 1

Using observables for query options just does not work like that. There is no facility in the underlying SDK to dynamically modify a query's limitToFirst and there's no way of doing it in AngularFire2.

Each time an observable query option emits a new value, a new Firebase ref is created. You can see it in the source here.

However, it would be possible to create an observable that represents an infinite list by doing something like this:

import { Observable } from "rxjs/Observable";
import { Subject } from "rxjs/Subject";
import rxjs/add/observable/defer";
import rxjs/add/observable/zip";
import rxjs/add/operator/concatMap";
import rxjs/add/operator/filter";
import rxjs/add/operator/first";
import rxjs/add/operator/map";
import rxjs/add/operator/scan";
import rxjs/add/operator/share";
import rxjs/add/operator/startWith";

const pageSize = 100;
let notifier = new Subject<any>();
let last: Observable<any>;

let infiniteList = Observable

  // Use zip to combine the notifier's emissions with the last
  // child value:

  .zip(notifier, Observable.defer(() => last))

  // Use concatMap to emit a page of children into the
  // composed observable (note that first is used to complete
  // the inner list):

  .concatMap(([unused, last]) => this.af.database.list("quests", {
      query: {

        // If there is a last value, start at that value but ask
        // for one more:

        limitToFirst: last ? (pageSize + 1) : pageSize,
        orderByChild: "date_published",
        startAt: last
      }
    })
    .first()
  )

  // Use scan to accumulate the page into the infinite list:

  .scan((acc, list) => {

    // If this isn't the initial page, the page was started
    // at the last value, so remove it from the beginning of
    // the list:

    if (acc.length > 0) {
      list.shift();
    }
    return acc.concat(list);
  }, [])

  // Use share so that the last observable (see below) doesn't
  // result in a second subscription:

  .share();

// Each time a page is emitted, map to its last child value so
// that it can be fed back into the composed infinite list:

last = infiniteList
  .filter((list) => list.length > 0)
  .map((list) => list[list.length - 1].date_published)
  .startWith(null);

infiniteList.subscribe((list) => console.log(list));

// Each time the notifier emits, another page will be retrieved
// and added to the infinite list:

notifier.next();
notifier.next();
notifier.next();

That will work, but if the child upon which you are ordering has duplicate values, AngularFire2 won't be able to page through the results reliably until this issue is re-opened and resolved.

The resultant list is static. That is, children already paged in to the list won't be updated if the database changes. Implementing a dynamic list is more challenging, as duplicated and missing children can easily be effected by the limit-based paging mechanism.

---

Since writing this answer, I've made available tested implementations of forward and reverse, non-realtime and realtime infinite list observables in a library of Firebase observables that I have open sourced. See this GitHub repo.

Answer 2

To add on to cartant's answer, if you wish to start at the end and retrieve list items in reverse, here is how you do it (I've added comments where code was changed).

import { Observable } from "rxjs/Observable";
import { Subject } from "rxjs/Subject";
import rxjs/add/observable/defer";
import rxjs/add/observable/zip";
import rxjs/add/operator/concatMap";
import rxjs/add/operator/filter";
import rxjs/add/operator/first";
import rxjs/add/operator/map";
import rxjs/add/operator/scan";
import rxjs/add/operator/share";
import rxjs/add/operator/startWith";

const pageSize = 100;
let notifier = new Subject<any>();
let last: Observable<any>;

let infiniteList = Observable

  .zip(notifier, Observable.defer(() => last))

  .concatMap(([unused, last]) => this.af.database.list("quests", {
      query: {

        // Use limitToLast to move upward the list instead of downward

        limitToLast: last ? (pageSize + 1) : pageSize,
        orderByChild: "date_published",

        // Use endAt to start at the end of the list

        endAt: last
      }
    })
    .first()
  )

  .scan((acc, list) => {

    // Swap the roles of acc and list, as we want to 
    // concatenate from the beginning

    if (list.length > 0) {
      acc.shift();
    }
    return list.concat(acc);
  }, [])

  .share();



last = infiniteList
  .filter((list) => list.length > 0)

  // Use the first child in this list as the next endAt value

  .map((list) => list[0].date_published)

  // Use undefined instead of null, as endAt: null in angularfire2
  // will search for the last child that is null

  .startWith(undefined);

infiniteList.subscribe((list) => console.log(list));

notifier.next();
notifier.next();
notifier.next();