How to use memcache in dataflow?

Question

I would like to use Memcache in dafalow ParDo? any ideas how?I can't use existing memcahse lib as they belong to appengine and are not serializable. Rohit

Answer 1

My guess is, you have a private variable in your DoFn of type MemcacheServiceImpl (if my guess is wrong, please edit your question to include the code of your DoFn).

Indeed, Dataflow serializes your DoFn's when you submit the pipeline and de-serializes them on the workers. The proper way to handle this is to make the variable transient, and initialize it lazily:

class MyDoFn extends DoFn<..., ...> {
  private transient MemcacheService memcache;
  private MemcacheService getMemcache() {
    if (memcache == null) {
      memcache = MemcacheServiceFactory.getMemcacheService();
      ...
    }
  }

  public void process(...) {
    ...use getMemcache()...
  }
}

Also note that to access AppEngine APIs, including Memcache, from a non-AppEngine environment, you should use the Remote API.