Memcached: auto-discovery python support on AWS El

问题:

I started to use AWS Elasticache with my django web app.

I started by setting the cache location to the unique endpoint using the auto-discovery feature, but it doesn't seems to work.

I'm using pylibmc (1.2.2) and django-pylibmc-sasl (0.2.4) to connect to memcached from python.

Does the auto-discovery feature work on these clients? How can I enable it?

回答1:

Quick answer

Yes for django: django-elasticache

Long Answer

ElastiCache provides memcached interface so there are three solution of using it:

1. Memcached configured with location = Configuration Endpoint.

In this case your application will randomly connect to nodes in cluster and cache will be used with not optimal way. At some moment you will be connected to first node and set item. Minute later you will be connected to another node and will not able to get this item.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': 'cache.gasdbp.cfg.use1.cache.amazonaws.com:11211',
    }
}

2. Memcached configured with all nodes.

It will work fine, memcache client will separate items between all nodes and will balance loading on client side. You will have problems only after adding new nodes or delete old nodes. In this case you should add new nodes manually and don't forget update your app after all changes on AWS.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': [
            'cache.gqasdbp.0001.use1.cache.amazonaws.com:11211',
            'cache.gqasdbp.0002.use1.cache.amazonaws.com:11211',
        ]
    }
}

3. Use django-elasticache.

It will connect to cluster and retrieve ip addresses of all nodes and configure memcached to use all nodes.

CACHES = {
    'default': {
        'BACKEND': 'django_elasticache.memcached.ElastiCache',
        'LOCATION': 'cache-c.draaaf.cfg.use1.cache.amazonaws.com:11211',
    }
}

Difference between setup with nodes list (django-elasticache) and connection to only one configuration Endpoint (using dns routing) you can see on this graph:

回答2:

I used the PyLibMC binding which doesn't seems to support auto-discovery.

The Memcached backend built-in with Django and used in the documentation is working well with the unique endpoint provided by Elasticache.

Now everything's running fine and I improved a lot my response time with Memcached.

回答3:

I wrote a python client for aws elasticache, you can try it.

Installation:

pip install python_memcached hash_ring
pip install elasticache_pyclient

Simple usage:

>>> from elasticache_pyclient import MemcacheClient
>>> mc = MemcacheClient('test.lwgyhw.cfg.usw2.cache.amazonaws.com:11211')
>>> mc.set('foo', 'bar')
True
>>> mc.get('foo')
'bar'

This package call python_memcached to do the actual memcache operation, so it has the exactly same functions as python_memcached, for more options, you can reference python_memcached.

Here is the elasticache_pyclient home page: https://github.com/yupeng820921/elasticache_pyclient