Redis Mass Insertion

Question

I am considering using Redis' protocol to do mass insertions as described here: http://redis.io/topics/mass-insert Before I get busy write code to handle this I just want to be sure that I am clear on what is required by Redis to make this work.

The above link suggests that to call the SET operation [SET myKey Value myValue] using mass insertion I need to create a command that can be done in either several lines in a file or a single quoted string.

Assuming I don't want to use the SET command instead I want to use the SADD command to add to a set. Is what I have here valid for a quoted string format?

"*4\r\n$4\r\nSADD\r\n$2\r\n80\r\n$5\r\n1,2,34\r\n"

Essentially what I am storing is a key: 80 whose value is 1,2,34

In the end what I want the capability to do is this:

Key     Value
80      1,2,34
90      4,8,34

Get the intersection (SINTER) and/or union (SUNION) of the two sets. i.e. SINTER = 34 or SUNION = 1,2,4,8,34

Any helpful information you can provide is appreciated. I just want to make sure I'm on the right path this.

Answer 1

I would say you are on the path of premature optimization (which is usually not the best one).

Any scripting language with a Redis client supporting pipelining should be able to push at least 50K commands/s to the Redis server. The code will be straightforward, with no need to manually encode the Redis protocol. Granted, the massive insert trick is faster, but do you really need it?

Now, if you still want to stick to massive insert, you need to encode a proper Redis command. The example you provided is wrong for several reasons:

• the number of arguments is wrong (your example should start by *3)

• the length of the last argument is wrong (1,2,34 length is 6 bytes not 5).

• in the SADD command, you need one argument per item of the set (i.e. for Redis 1,2,34 will be a single item, not three).

The proper command would rather be something like this:

"*5\r\n$4\r\nSADD\r\n$2\r\n80\r\n$1\r\n1\r\n$1\r\n2\r\n$2\r\n34\r\n"

The Redis protocol is described here: http://redis.io/topics/protocol