RxJava and cursor based RESTful pagination

问题:

I'm working with the Spotify API and am hoping to chain a few paginated results using RxJava. Spotify uses cursor based pagination, so solutions like the one from @lopar will not work.

The response is from this call and looks something like this (imagine there are 50 items):

{
  "artists" : {
    "items" : [ {
      "id" : "6liAMWkVf5LH7YR9yfFy1Y",
      "name" : "Portishead",
      "type" : "artist"
    }],
    "next" : "https://api.spotify.com/v1/me/following?type=artist&after=6liAMWkVf5LH7YR9yfFy1Y&limit=50",
    "total" : 119,
    "cursors" : {
      "after" : "6liAMWkVf5LH7YR9yfFy1Y"
    },
    "limit" : 50,
    "href" : "https://api.spotify.com/v1/me/following?type=artist&limit=50"
  }
}

Right now, I'm getting the first 50 results like this, using retrofit:

public class CursorPager<T> {
    public String href;
    public List<T> items;
    public int limit;
    public String next;
    public Cursor cursors;
    public int total;

    public CursorPager() {
    }
}

public class ArtistsCursorPager {
    public CursorPager<Artist> artists;

    public ArtistsCursorPager() {
    }
}

then

public interface SpotifyService  {

    @GET("/me/following?type=artist")
    Observable<ArtistsCursorPager> getFollowedArtists(@Query("limit") int limit);

    @GET("/me/following?type=artist")
    Observable<ArtistsCursorPager> getFollowedArtists(@Query("limit") int limit, @Query("after") String spotifyId);

}

and

mSpotifyService.getFollowedArtists(50)
        .flatMap(result -> Observable.from(result.artists.items))
        .flatMap(this::responseToArtist)
        .sorted()
        .toList()
        .subscribe(new Subscriber<List<Artist>>() {
            @Override
            public void onNext(List<Artist> artists) {
                callback.onSuccess(artists);
            }
            // ...
        });

I'd like to return all (in this case 119) artists in callback.success(List<Artist>). I'm new to RxJava, so I'm unsure if there is a smart way to do this.

回答1:

The only problem with the recursive solution is the stack over flow problem. A way to do it without recursion is

Observable<ArtistsCursorPager> allPages = Observable.defer(() ->
{
    BehaviorSubject<Object> pagecontrol = BehaviorSubject.create("start");
    Observable<ArtistsCursorPager> ret = pageControl.asObservable().concatMap(aKey ->
    {
        if (aKey != null && aKey.equals("start")) {
            return Observable.getFollowedArtists(50).doOnNext(page -> pagecontrol.onNext(page.cursors.after));
        } else if (aKey != null && !aKey.equals("")) {
            return Observable.getFollowedArtists(50,aKey).doOnNext(page -> pagecontrol.onNext(page.cursors.after));
        } else {
            return Observable.<ArtistsCursorPager>empty().doOnCompleted(()->pagecontrol.onCompleted());
        }        
    });
    return ret;
});

See the solutions to this question.

回答2:

There´s not unique way to do this. In my case what I did was make some recursive calls using mergeWith

private Observable<String> getUUIDsQuery(JsonObject response) {
    final Observable<String> uuidsQuery = createUuidsQuery(response);
    return hasPagination(response) ? paginate(response, uuidsQuery) : uuidsQuery;
}

   private Observable<String> paginate(JsonObject response, Observable<String> uuidsQuery) {
    return request(getPaginationUri(response))
            .flatMap(res -> uuidsQuery.mergeWith(getUUIDsQuery(res)));
}

Hope to help you to give you an idea.

回答3:

Thanks for opening my eyes that I didn't read your question properly. Here is the best solution i can suggest you without using the normal recursive function but the RxJava way.

PublishSubject<Integer> limit = PublishSubject.create();

        limit.concatMap(integer -> Observable.just(integer + 1)) // Assuming this gives network result based upon the artist id you provided
                .doOnNext(integer -> {
                    // Based on the network result i make my conditions here. Pass the 50th artist id here. otherwise call onCompleted.
                    Timber.d("Publish: doOnNext: %d", integer);
                    if (integer < 10) {
                        // Pass the artist id to make the next call
                        limit.onNext(integer);
                    } else {
                        limit.onCompleted();
                    }
                })
                .toList()
                .subscribe(integers -> Timber.d("Publish: All the results"));

        limit.onNext(1); // For demo, I'm starting here with 1. You have to pass the artist id here

The output looks like following:

Publish: doOnNext: 2       
Publish: doOnNext: 3       
Publish: doOnNext: 4       
Publish: doOnNext: 5       
Publish: doOnNext: 6       
Publish: doOnNext: 7       
Publish: doOnNext: 8       
Publish: doOnNext: 9       
Publish: doOnNext: 10      
Publish: All the results

The toList() operator gives you list of all the responses you've got in the end when all the calls have been made. Have a look at reduce() operator as well.