App Engine >60 seconds backend processing?

Question

I have an appengine app which receives pubsub messages.

I am processing the data which might take more than 60 seconds.

Since the request handler is limited for 60 seconds, I acknowledge the pubsub straight away and then do the processing.

Though this does not help - the messages I send are retried again and again (I see in the logger) even though the processing was done.

here is my code:

public class Handler extends HttpServlet {


@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException  {


    // Acknowledge the message by returning a success code
    resp.setStatus(HttpServletResponse.SC_OK);
    resp.getWriter().close();

    JsonParser parser = JacksonFactory.getDefaultInstance().createJsonParser(req.getInputStream());
    parser.skipToKey("message");

    PubsubMessage message = parser.parseAndClose(PubsubMessage.class);
    log.info("request was received " +  message.getMessageId());
    String payload = new String(message.decodeData(), "UTF-8");

    doSomthingWithPayload(payload); // this takes more than 60 seconds
}

}

I tried using task queue, but since my data is very large I get Task size too large

questions:

1. is this the right way to acknowledge the pubsub?
2. can the process take more than 60 seconds after acknowledging?
   • if yes - why is my pubsub message resent again and again?
   • if not: what can I do as a work around - while I cannot use task queue?

Answer 1

You need to let your servlet thread go ASAP. In your code you writing response and than doing payload processing on servlet thread. Instead immediately after you received request you should parse and store payload to datastore, initiate async processing (via task queue), set status HttpServletResponse.SC_OK and return from doPost. If you will use datastore to store your payload you will not hit task size limit because you will be able to put in task only datastore record ID, not all payload.