Tweets and followers fetching app with Google App

Question

I'm trying to build an app in Python with Google App Engine that fetches followers of specific accounts and then their tweets. I'm basing it on this template and changing it to adapt it to what I need.

The issue at the moment is that when I try to fetch followers, I get an DeadlineExceededError due to the Twitter API waiting time.

I have found this post on how to fix the same problem and I think that in my case the best solution would be to use backends, but I noticed that they are deprecated.

Does someone know how I can achieve the same result without the deprecated module?

Answer 1

Backends (modules) have been deprecated in favor of Services:

https://cloud.google.com/appengine/docs/flexible/python/an-overview-of-app-engine

For the Service you want to be able to handle requests longer than 60 seconds, set it to Manual Scaling. Then, a request can run for up to 24 hours (or until you shut it down). See:

https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#instance_scaling

Of course, your costs may go up with long running instances and request.

Answer 2

You have a couple options that you can use for long-running tasks:

• Use GAE Task Queues: GAE provides push and pull queues which allow you to do work asynchronously outside of the individual request.
• Use Cloud Pub/Sub: A type of pull queue, this would allow your App Engine app to publish a message every time you wanted fetch followers or fetch tweets. The subscriber would then take the message from the queue, perform a long-running task, and then put the result into some datastore.
• Use GAE Services: This would allow you to create a background service and manually scale it to run as long as you need.