Async Redis pooling using libevent

Question

I want get as much as possible from Redis + Hiredis + libevent.

I'm using following code (without any checks to be short)

#include <stdlib.h>
#include <event2/event.h>
#include <event2/http.h>
#include <event2/buffer.h>
#include <hiredis/hiredis.h>
#include <hiredis/async.h>
#include <hiredis/adapters/libevent.h>

typedef struct reqData {
  struct evhttp_request* req;
  struct evbuffer* buf;
} reqData;

struct event_base* base;
redisAsyncContext* c;

void get_cb(redisAsyncContext* context, void* r, void* data) {
  redisReply* reply = r;
  struct reqData* rd = data;

  evbuffer_add_printf(rd->buf, "%s", reply->str);
  evhttp_send_reply(rd->req, HTTP_OK, NULL, rd->buf);

  evbuffer_free(rd->buf);
  redisAsyncDisconnect(context);
}

void cb(struct evhttp_request* req, void* args) {
  struct evbuffer* buf;
  buf = evbuffer_new();

  reqData* rd = malloc(sizeof(reqData));
  rd->req = req;
  rd->buf = buf;

  c = redisAsyncConnect("0.0.0.0", 6380);
  redisLibeventAttach(c, base);

  redisAsyncCommand(c, get_cb, rd, "GET name");
}

int main(int argc, char** argv) {
  struct evhttp* http;
  struct evhttp_bound_socket* sock;

  base = event_base_new();
  http = evhttp_new(base);
  sock = evhttp_bind_socket_with_handle(http, "0.0.0.0", 8080);

  evhttp_set_gencb(http, cb, NULL);

  event_base_dispatch(base);

  evhttp_free(http);
  event_base_free(base);
  return 0;
}

To compile, use gcc -o main -levent -lhiredis main.c assuming libevent, redis and hiredis in system.

I curious when I need to do redisAsyncConnect? In main() once or (as example shows) in every callback. Is there anything I can do to boost performance?

I'm getting about 6000-7000 req/s. Using ab to benchmark this, stuff complicates when trying big numbers (e.g. 10k reqs) - it cannot complete benchmark and freezes. Doing the same thing but in blocking way the results are 5000-6000 req/s.

I've extended max file open by limit -n 10000. I'm using Mac OS X Lion.

Answer 1

It is of course much better to open the Redis connection once, and try to reuse it as far as possible.

With the provided program, I suspect the benchmark freezes because the number of free ports in the ephemeral port range is exhausted. Each time a new connection to Redis is opened and closed, the corresponding socket spends some time in TIME_WAIT mode (this point can be checked using the netstat command). The kernel cannot recycle them fast enough. When you have too many of them, no further client connection can be initiated.

You also have a memory leak in the program: the reqData structure is allocated for each request, and never deallocated. A free is missing in get_cb.

Actually, there are 2 possible sources of TIME_WAIT sockets: the ones used for Redis, and the ones opened by the benchmark tool to connect to the server. Redis connections should be factorized in the program. The benchmark tool must be configured to use HTTP 1.1 and keepalived connections.

Personally, I prefer to use siege over ab to run this kind of benchmark. ab is considered as a naive tool by most people interested in benchmarking HTTP servers.

On my old Linux PC, the initial program, run against siege in benchmark mode with 50 keepalived connections, results in:

Transaction rate:            3412.44 trans/sec
Throughput:                     0.02 MB/sec

When we remove completely the call to Redis, only returning a dummy result, we get:

Transaction rate:            7417.17 trans/sec
Throughput:                     0.04 MB/sec

Now, let's modify the program to factorize the Redis connection, and naturally benefit from pipelining. The source code is available here. Here is why we get:

Transaction rate:            7029.59 trans/sec
Throughput:                     0.03 MB/sec

In other words, by removing the systematic connection/disconnection events, we can achieve twice the throughput. The performance with the Redis call is not so far than the performance we get without any Redis call.

To further optimize, you could consider using a unix domain socket between your server and Redis, and/or pool the dynamically allocated objects to reduce CPU consumption.

UPDATE:

To experiment with a unix domain socket, it is straightforward: you just have to activate the support in Redis itself by updating the configuration file:

# Specify the path for the unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
#
unixsocket /tmp/redis.sock
unixsocketperm 755

and then replace the connection function:

c = redisAsyncConnect("0.0.0.0", 6379);

by:

c = redisAsyncConnectUnix("/tmp/redis.sock");

Note: here, hiredis async does a good job at pipelining the commands (provided the connection is permanent), so the impact will be low.