Redis — best way to store a large map (dictionary)

Question

What I need to do is to store a one-to-one mapping. The dataset consists of a large number of key-value pairs of the same kind (10M+). For example, one could use a single instance of HashMap object in Java for storing such data.

The first way to do this is to store lots of key-value pairs, like this:

SET map:key1 value1
...
SET map:key900000 value900000
GET map:key1

The second option is to use a single "Hash":

HSET map key1 value
...
HSET map key900000 value900000
HGET map key1

Redis Hashes have some convenient commands (HMSET, HMGET, HGETALL, etc.), and they don't pollute the keyspace, so this looks like a better option. However, are there any performance or memory considerations when using this approach?

Answer 1

Yes, as Itamar Haber says you should look at redis memory optimization guide. But also you should keep in mind such things (in few lines):

1. Prefer HSET besides of KEYS. Redis consume much of memory just on key space manegement. In simple (and rough) 1 HSET with 1,000,000 keys consume up to 10x less memory then 1,000,000 keys with one value.
2. Keep HSET size less then hash-max-zipmap-entries and valid hash-max-zipmap-value if memory is main target. Be sure to understand, what hash-max-zipmap-entries and hash-max-zipmap-value means. Also take some time to read about ziplist.
3. While you actualy do not mant to handle hash-max-zipmap-entries with 10M+ keys (to slow access in this keys) you should break one HSET in some slots. For example you set hash-max-zipmap-entries as 10,000. So to store 10M+ keys your need 1000+ HSET keys with 10,000 each. For rough example - crc32(key) % maxHsets.
4. Read about strings in redis and use KEY names (in HSET) lenght based on real memory management for this structure. In simple - keeping key lenght under 7 bytes you spend 16 bytes per key, but 8 bytes key spends 48 bytes each. Why? Read about simple dynamic strings.

It may be usefull to read about:

• Redis Memory Optimization (from sripathikrishnan)
• Comments about internal ziplist structure.
• Storing hundreds of millions of simple key-value pairs in Redis (Instagram)