Reason for Redis `dir` path changing dynamically

Question

We are facing an issue with redis, where the 'dir' path for the redis is getting set without any notice.

Resulting in the following error (while writing to redis).

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Information we have to support this.

- Redis Config file

A look at our redis.conf (located at /etc/redis/redis.conf) hint us that the dir path value is...

# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /var/lib/redis

- Redis INFO Result.

{"redis_version"=>"2.8.17", "redis_git_sha1"=>"00000000", "redis_git_dirty"=>"0", "redis_build_id"=>"e2de0295da7f0ee1", "redis_mode"=>"standalone", "os"=>"Linux 3.16.0-4-amd64 x86_64", "arch_bits"=>"64", "multiplexing_api"=>"epoll", "gcc_version"=>"4.9.2", "process_id"=>"6860", "run_id"=>"4ecd0ff19798a88793e7a3ce281f398ba9aa2f5a", "tcp_port"=>"6379", "uptime_in_seconds"=>"99525", "uptime_in_days"=>"1", "hz"=>"10", "lru_clock"=>"11741717", "config_file"=>"/etc/redis/redis.conf", "connected_clients"=>"2", "client_longest_output_list"=>"0", "client_biggest_input_buf"=>"0", "blocked_clients"=>"0", "used_memory"=>"784840", "used_memory_human"=>"766.45K", "used_memory_rss"=>"3624960", "used_memory_peak"=>"784840", "used_memory_peak_human"=>"766.45K", "used_memory_lua"=>"33792", "mem_fragmentation_ratio"=>"4.62", "mem_allocator"=>"jemalloc-3.6.0", "loading"=>"0", "rdb_changes_since_last_save"=>"0", "rdb_bgsave_in_progress"=>"0", "rdb_last_save_time"=>"1454582288", "rdb_last_bgsave_status"=>"ok", "rdb_last_bgsave_time_sec"=>"0", "rdb_current_bgsave_time_sec"=>"-1", "aof_enabled"=>"0", "aof_rewrite_in_progress"=>"0", "aof_rewrite_scheduled"=>"0", "aof_last_rewrite_time_sec"=>"-1", "aof_current_rewrite_time_sec"=>"-1", "aof_last_bgrewrite_status"=>"ok", "aof_last_write_status"=>"ok", "total_connections_received"=>"12", "total_commands_processed"=>"4651", "instantaneous_ops_per_sec"=>"0", "rejected_connections"=>"0", "sync_full"=>"0", "sync_partial_ok"=>"0", "sync_partial_err"=>"0", "expired_keys"=>"0", "evicted_keys"=>"0", "keyspace_hits"=>"0", "keyspace_misses"=>"0", "pubsub_channels"=>"0", "pubsub_patterns"=>"0", "latest_fork_usec"=>"206", "role"=>"master", "connected_slaves"=>"0", "master_repl_offset"=>"0", "repl_backlog_active"=>"0", "repl_backlog_size"=>"1048576", "repl_backlog_first_byte_offset"=>"0", "repl_backlog_histlen"=>"0", "used_cpu_sys"=>"57.78", "used_cpu_user"=>"25.71", "used_cpu_sys_children"=>"0.00", "used_cpu_user_children"=>"0.00", "db0"=>"keys=2851,expires=0,avg_ttl=0"}

Key point

 redis.info['config_file'] 

 => /etc/redis/redis.conf

Process id of Redis

 redis.info['process_id']

 => 6860

- Redis Config

And then I inspected redis config and...

redis.config "get" ,"dir"

=> ["dir", "/etc/ssh"]
## ideally should be /var/lib/redis

Things, I did to fix the above error.

redis.config "set" ,"dir","/var/lib/redis"
=> "OK" 

redis.config "get", "dir"
=> ["dir", "/var/lib/redis"]

This sort of fix it (the above issue). But for temporary purpose only because.....

After couple of hours I noticed this.

redis.config "get" ,"dir"
=> ["dir", "/etc/ssh"]

Inspecting the redis config again

- Redis Info

{"redis_version"=>"2.8.17", "redis_git_sha1"=>"00000000", "redis_git_dirty"=>"0", "redis_build_id"=>"e2de0295da7f0ee1", "redis_mode"=>"standalone", "os"=>"Linux 3.16.0-4-amd64 x86_64", "arch_bits"=>"64", "multiplexing_api"=>"epoll", "gcc_version"=>"4.9.2", "process_id"=>"6860", "run_id"=>"4ecd0ff19798a88793e7a3ce281f398ba9aa2f5a", "tcp_port"=>"6379", "uptime_in_seconds"=>"109032", "uptime_in_days"=>"1", "hz"=>"10", "lru_clock"=>"11751224", "config_file"=>"/etc/redis/redis.conf", "connected_clients"=>"2", "client_longest_output_list"=>"0", "client_biggest_input_buf"=>"0", "blocked_clients"=>"0", "used_memory"=>"788984", "used_memory_human"=>"770.49K", "used_memory_rss"=>"3629056", "used_memory_peak"=>"808952", "used_memory_peak_human"=>"789.99K", "used_memory_lua"=>"33792", "mem_fragmentation_ratio"=>"4.60", "mem_allocator"=>"jemalloc-3.6.0", "loading"=>"0", "rdb_changes_since_last_save"=>"82", "rdb_bgsave_in_progress"=>"0", "rdb_last_save_time"=>"1454591521", "rdb_last_bgsave_status"=>"ok", "rdb_last_bgsave_time_sec"=>"0", "rdb_current_bgsave_time_sec"=>"-1", "aof_enabled"=>"0", "aof_rewrite_in_progress"=>"0", "aof_rewrite_scheduled"=>"0", "aof_last_rewrite_time_sec"=>"-1", "aof_current_rewrite_time_sec"=>"-1", "aof_last_bgrewrite_status"=>"ok", "aof_last_write_status"=>"ok", "total_connections_received"=>"14", "total_commands_processed"=>"6966", "instantaneous_ops_per_sec"=>"0", "rejected_connections"=>"0", "sync_full"=>"0", "sync_partial_ok"=>"0", "sync_partial_err"=>"0", "expired_keys"=>"0", "evicted_keys"=>"0", "keyspace_hits"=>"0", "keyspace_misses"=>"0", "pubsub_channels"=>"0", "pubsub_patterns"=>"0", "latest_fork_usec"=>"135", "role"=>"master", "connected_slaves"=>"0", "master_repl_offset"=>"0", "repl_backlog_active"=>"0", "repl_backlog_size"=>"1048576", "repl_backlog_first_byte_offset"=>"0", "repl_backlog_histlen"=>"0", "used_cpu_sys"=>"62.96", "used_cpu_user"=>"27.97", "used_cpu_sys_children"=>"0.00", "used_cpu_user_children"=>"0.00", "db0"=>"keys=2902,expires=0,avg_ttl=0"}

Key point.

 redis.info["process_id"]
 => 6860

 redis.info['config_file']
 => /etc/redis/redis.conf

So,

1. Redis Server was never restarted. (process_id says a lot about this).

2. Redis is picking the correct config file i.e. /etc/redis/redis.conf.

Question.

So, If point 1, and 2 are true what is causing redis to set the config dir value to /etc/ssh after some hours? ( I have reset the config 5-6 times since yesterday.)

Note:

Also restarted the server (yesterday) in a hope that the issue would be resolved permanently. But with no luck.

Note:

Responding to @Keith answer

Yes! the server is perfectly behind a restrictive firewall.

Thought?

Answer 1

I recently saw an attack that utilized this as a strategy. Is your server behind a restrictive firewall?

REF: http://antirez.com/news/96