redis-cli redirected to 127.0.0.1

Question

I started Redis cluster on PC1, then connected it on PC2. When needed to redirect to another cluster node, it shows Redirected to slot [7785] located at 127.0.0.1, but should show Redirected to slot [7785] located at [IP of PC1, like 192.168.1.20], then it shows an error. What is happening? What can I do?

The output:

[admin@localhost ~]$ redis-cli -c -h 192.168.1.20 -p 30001
192.168.1.20:30001> get foo
-> Redirected to slot [12182] located at 127.0.0.1:30003
Could not connect to Redis at 127.0.0.1:30003: Connection refused
Could not connect to Redis at 127.0.0.1:30003: Connection refused
not connected>

Output of redis-cli -h 192.168.1.20 -p 30001 cluster nodes:

5f6d6f1319318233917aba92b6ab0e244b3260d7 127.0.0.1:30004 slave 4c7b046ecaeb2dc689cbad21ee3466fb43b48fb9 0 14639
84410573 4 connected
e04d5b461cb6a2b48cb2a607e2140b7c1d32af25 127.0.0.1:30006 slave 3fc25c3851f7a9afd09b60739434118c25cd9243 0 14639
84410473 6 connected
3fc25c3851f7a9afd09b60739434118c25cd9243 127.0.0.1:30003 master - 0 1463984410573 3 connected 10923-16383
4c7b046ecaeb2dc689cbad21ee3466fb43b48fb9 127.0.0.1:30001 myself,master - 0 0 1 connected 0-5460
7383830ac84f199db346da3112b5aaf9e124d3cf 127.0.0.1:30005 slave 1eeeb51522aed364fcf9623d6045fa3df2748579 0 14639
84410573 5 connected
1eeeb51522aed364fcf9623d6045fa3df2748579 127.0.0.1:30002 master - 0 1463984410473 2 connected 5461-10922

Answer 1

That is because all your Redis IP addresses have updated to 127.0.0.1, and they believe other Redis are located in 127.0.0.1 too. That's not wrong if nodes in a cluster just communicate with each other, but definitely improper when a connection from other host want to know about the cluster.

In that situation, your client asked a Redis for a key it's not in charge and the Redis told the client to redirect to 127.0.0.1:30003. The client misunderstood it and tried to connect the port 30003 in its localhost, and certainly found nothing.

To fix it, try to send cluster meet with the right IP to each Redis in the cluster. I've made an experiment like this

# initial, Redis doesn't know its IP before a meet
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d :7000 myself,master - 0 0 0 connected

# meet from 127.0.0.1, and their IP addresses updated to 127.0.0.1
127.0.0.1:7000> cluster meet 127.0.0.1 7001
OK
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d 127.0.0.1:7000 myself,master - 0 0 0 connected
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 127.0.0.1:7001 master - 0 1463987186714 1 connected

# send another meet, use the eth0 IP other than lo
127.0.0.1:7000> cluster meet 172.31.28.76 7001
OK
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d 127.0.0.1:7000 myself,master - 0 0 0 connected
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 172.31.28.76:7001 master - 0 1463987192672 1 connected

# connect to :7001, its cluster nodes are what we expect
127.0.0.1:7001> cluster nodes
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 172.31.28.76:7001 myself,master - 0 0 1 connected
8af9e47cb96f3bd8fff3800c38da11601157605d 172.31.28.76:7000 master - 0 1463987203631 0 connected
# send another meet to fix
127.0.0.1:7001> cluster meet 172.31.28.76 7000
OK

# back to :7000, its address updated
127.0.0.1:7000> cluster nodes
8af9e47cb96f3bd8fff3800c38da11601157605d 172.31.28.76:7000 myself,master - 0 0 0 connected
2c3d9b6c29f21ecd846f42bcfb238099d88b57df 172.31.28.76:7001 master - 0 1463987210539 1 connected

In your case you may send multiple cluster meet commands to each Redis to ensure its IP updated at all its peers.

Answer 2

Hey could you try binding your redis cluster instance to server's IP

Update your redis.conf to add

bind 172.31.28.76

PS- Update IP as required