How to page the dataset

Question

Hi I want to enable web pagination that used to work and it broke during an update to the environment or my implementation (not sure which). I want to paginate a list ordered by GAE Since objects tend to have a "natural ordering" ie time, numbers, words etc could this code be already available somewhere usable? I tried doing something like the example from Joey G that Google listed here: http://code.google.com/appengine/articles/paging.html My effort that takes the parameter called bookmark from the URL query:

  next = None
  bookmark = self.request.get("bookmark") 
  category = self.request.get('cg')#category parameter
  if bookmark:
    bookmark = datetime.strftime(bookmark[:-7], "%Y-%m-%d %H:%M:%S")  
  else:       
    bookmark = datetime.strftime(datetime.now(), "%Y-%m-%d %H:%M:%S") 

  if cg: 
    articles = Articles.all().filter("category =", cg).filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)    

It used to work and now the paging is broken, perhaps because I don't completely understand how to handle the modified and datetime objects. Could you help me on the way? The URL to my use case is www.koolbusiness.com/li Thanks

EDIT/UPDATE: Here's the current code that displays the list page. It only uses IN at a few places so supposedly it could get rewritten to work without IN the difficulty is that it branches so much due the the many combinations possible ie searching with or with nocategory, category with or with no search and also geospatially but I don't think that I'm trying something impossible it's just that a need to learn more python (there is lambda programming involved that I only pasted in and don't fully grasp) and get a clearer structure when all the combinations are handled.

class I18NListPage(FBBaseHandler,I18NHandler):

  def get(self, cursor=None, limit=60, PAGESIZE = 10, twittername = None):
    client = OAuthClient('twitter', self) 
    if client.get_cookie():
      info = client.get('/account/verify_credentials')
      twittername = info['screen_name']
    if (users.is_current_user_admin()):
       timeline = datetime.now () - timedelta (days = limit)   
    else:
        timeline = datetime.now () - timedelta (days = limit)
    logo = ''             
    if util.get_host().endswith('.br'):
      cookie_django_language = 'pt-br'
      logo = 'montao'   
      translation.activate(cookie_django_language)
      self.request.COOKIES['django_language'] = cookie_django_language
      dispatch= 'template/montaoli.html'
    else:
        cookie_django_language = self.request.get('cookie_django_language', '') if self.request.get('cookie_django_language', '') else self.request.get('hl', '')
        dispatch= 'template/li.html'
    if cookie_django_language:
      if cookie_django_language == 'unset':
        del self.request.COOKIES['django_language']
      else:
        self.request.COOKIES['django_language'] = cookie_django_language
      self.reset_language()       
    next = None
    bookmark = self.request.get("bookmark") 
    if not bookmark:
       bookmark = = str(time.time())

    category = self.request.get('cg')
    q = self.request.get('q').encode("utf-8")
    w = self.request.get('q')
    cg = self.request.get('cg')
    t = self.request.get('t') 
    f = self.request.get('f')

    if cg and not t and not q and not f:#category without search
        ads = Ad.all().filter("category =", cg).filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)    
    elif q and not t and not cg and not f:#search without category
        ads = Ad.all().search(self.request.get('q')).filter("published =", True)    
        ads = filter(lambda x: x.modified > timeline, ads)
        ads = filter(lambda x: x.modified <= bookmark, ads)
        ads = ads[:PAGESIZE+1]
        ads = sorted(ads, key=lambda x: x.modified, reverse=True)
        #iterate list keeping elements that are on timeline newer than bookmark
    elif q and not t and cg and not f:
        ads = Ad.all().search(q).filter("type =", 's').filter("category =", cg).filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)    
    elif t and not f:
        ads = Ad.all().filter("type =", 'w').filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)    
    elif f == 'c':
        ads = Ad.all().filter("company_ad =", True).filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)
    elif f == 'p':
        ads = Ad.all().filter("company_ad =", False).filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)
    else:
        if util.get_host().find('onta') > 1:
            ads = Ad.all().filter("modified >", timeline).filter("published =", True).filter("url IN", ['www.montao.com.br','montao']).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)              
        else:
            ads = Ad.all().filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)
    if util.get_host().find('onta') > 1 and f == 'c':
        ads = Ad.all().filter("company_ad =", True).filter("modified >", timeline).filter("published =", True).filter("url IN", ['www.montao.com.br','montao']).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)
    elif util.get_host().find('onta') > 1 and f == 'p':
        ads = Ad.all().filter("company_ad =", False).filter("modified >", timeline).filter("published =", True).filter("url IN", ['www.montao.com.br','montao']).filter("modified <=", bookmark ).order("-modified").fetch(PAGESIZE+1)  


    if self.request.get('lat'):
        m=int(self.request.get('r')) if self.request.get('r') else 804670
        logging.info(m)
        lat = self.request.get('lat')
        lon = self.request.get('lon') if self.request.get('lon') else self.request.get('lng')
        ads = Ad.proximity_fetch(Ad.all().filter("modified >", timeline).filter("published =", True).filter("modified <=", bookmark ).order("-modified") ,db.GeoPt(lat, lon),max_results=PAGESIZE+1, max_distance=m)
        ads = sorted(ads, key=lambda x: x.modified, reverse=True)
    if ads and len(ads) == PAGESIZE+1:
      next = ads[-1].modified
      ads = ads[:PAGESIZE]     

    template_values = {'twittername':twittername,'request':self.request,'lat':self.request.get('lat'),'lon':self.request.get('lon'),'q':q,'w':w,'cg':cg,'t':t,'logo':logo,'ads':ads, 'next':next, 'user':users.get_current_user(), 'bookmark':bookmark,'q':q, 'user_url': users.create_logout_url(self.request.uri) if users.get_current_user() else 'login',
            'cg':category,'admin':users.is_current_user_admin(),}
    template_values.update(dict(current_user=self.current_user, facebook_app_id=FACEBOOK_APP_ID))
    path = os.path.join(os.path.dirname(__file__), dispatch)
    self.response.out.write(template.render(path, template_values))

UPDATE 2: I tried using the django paginator class from paginator import Paginator, InvalidPage, EmptyPage in this this case the following code actually pages the data set:

        articles = Articles.all()
        paginator = Paginator(articles,PAGESIZE)
        articles = paginator.page(page)

so I find this solution tempting since it's so easy and readable and hope that you can comment onwards.

Answer 1

You may need to base64 encoded your bookmark before you pass it to client. I have tried [Your site][1], but it is impossible to answer your question clearly without the server side error message.

Moreover, as the paging article has mentioned, using datetime as bookmark directly will cause issues while there are more than one articles posted in the same time period (in your case, the time period will be one minute). You may want to consider using this format instead.

2008-10-26 04:38:00|aee15ab24b7b3718596e3acce04fba85

or using seconds to represent the time.

1310427763.47|aee15ab24b7b3718596e3acce04fba85

should I set it to datetime.now() if it's the first page that gets viewed?

You can get the first page results by simply ignore the modified filter. For example:

if bookmark:
    suggestions = Suggestion.all().order("-when")
        .filter('when <=', bookmark).fetch(PAGESIZE+1)
else:
    suggestions = Suggestion.all().order("-when").fetch(PAGESIZE+1)

or using the current time as bookmark.

import time
bookmark = str(time.time())

Answer 2

Query cursors are built for exactly this. You should use them instead of building your own solution.