Redis Sentinel is a dedicated process to automate and simplify the Redis replication failover and switchover.
In this article, we will start with 3 Nodes. We will have two Redis instances on two different nodes – 1 master and 1 replica (or slave). Sentinel will be co-located on those 2 nodes, plus an additional node on one of our web servers.
Normally you would co-locate a Redis instance on the web/application server and access it via localhost or through a UNIX socket file. This is the straightforward way to incorporate Redis into the application.
For a scalable and highly available setup app, Redis should be deployed in a centralized approach, or into a different tier called a cache tier. This allows Redis instances to work together as a dedicated cache provider for the applications, decoupling the applications from the local Redis dependencies.
For this deployment we are setting up 3 Debian 11 VM instances.
Before deploying Redis Sentinel, we have to deploy a Redis replication consisting of two or more Redis Server instances. Let’s start by installing Redis on both servers, Redis-01 and redis-02:
1 – 192.168.0.30 – App- Web Server + Sentinel
2 – 192.168.0.31 – Redis-01 (replica) Redis Server + Sentinel
3 – 192.168.0.32 – Redis-02 (replica) Redis Server + Sentinel
apt install redis net-tools
Edit /etc/redis/redis.conf:
For redis-01 (master):
bind 127.0.0.1 192.168.0.31
protected-mode no
supervised systemd
masterauth Zssy56G21Zx
masteruser redisuser
user redisuser +@all on >Zssy56G21Zx
For redis-02 (replica):
bind 127.0.0.1 192.168.0.32
protected-mode no
supervised systemd
replicaof 192.168.0.31 6379
masterauth Zssy56G21Zx
masteruser redisuser
user redisuser +@all on >Zssy56G21Zx
- bind: List all the IP addresses that you want Redis to listen to. For Sentinel to work properly, Redis must be reachable remotely. Therefore we have to list out the interface the Sentinel will communicate with.
- protected-mode: This must be set to “no” to allow Redis to serve remote connections. This is required for Sentinel as well.
- supervised: We use the default systemd unit files provided by the installer package. For Ubuntu 20.04, it uses systemd as the service manager so we specify systemd here.
- replicaof: This is only for the slave node. For the original topology, we will make redis2 as the replica and redis1 as the master.
- masterauth: The password for user masteruser.
- masteruser: The username of the master user.
- user: We create the master user here. The user shall have no limit (+@all) and a password. This user will be used for Redis to manage replication and failover by Sentinel.
Restart Redis to apply the changes and enable it on boot:
sudo systemctl enable redis-server
Verify that Redis is running on port 6379 on both interfaces. The following example is the output from redis2:
sudo netstat -tulpn | grep -i redis
tcp 0 0 192.168.0.32:6379 0.0.0.0:* LISTEN 1246/redis-server 1
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1246/redis-server 1
tcp 0 0 127.0.0.1:26379 0.0.0.0:* LISTEN 1128/redis-sentinel
tcp6 0 0 ::1:26379 :::* LISTEN 1128/redis-sentinel
Verify the replication is working. On redis-01:
redis-cli info replication
Output:
Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.32,port=6379,state=online,offset=10621,lag=1
master_replid:c649cbc9ffab2b6ef7adae7df90a1d5d00913987
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:10754
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:10754
Pay attention to the role, connected_slaves and slave{i} keys. This indicates that redis1 is the master. Also, note that a replica can be a master of another replica – this is also known as chained replication.
While on the redis2:
redis-cli info replication
Output:
Replication
role:slave
master_host:192.168.0.31
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:52453
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:c649cbc9ffab2b6ef7adae7df90a1d5d00913987
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:52453
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:52453
Pay attention to the role, master_host, master_link_status and master_repl_offset. The replication delay between these two nodes can be determined by the master_repl_offset value on both servers.
Redis Sentinel Deployment
Redis Sentinel is basically the same redis-server process running with the “–sentinel” flag and different configuration files and ports. For production usage, it is strongly recommended to have at least 3 Sentinel instances for an accurate observation when performing the automatic failover. Therefore, we will install Sentinel on those 2 Redis nodes that we have, plus one of our web servers, 192.168.44.70 (shown in the architecture diagram).
Install the redis-sentinel package on the selected web server (Sentinel is already installed on our Redis hosts):
apt install redis-sentinel
Stop redis sentinel to make changes
systemctl stop redis-sentinel.service
By default, the Sentinel configuration file is located at /etc/redis/sentinel.conf. Make sure the following configuration lines are set:
App server, 192.168.0.30:
bind 192.168.0.30
port 26379
sentinel monitor mymaster 192.168.0.31 6379 2
sentinel auth-pass mymaster Zssy56G21Zx
sentinel auth-user mymaster redisuser
sentinel down-after-milliseconds mymaster 10000
redis-01, 192.168.0.31:
bind 192.168.0.31
port 26379
sentinel monitor mymaster 192.168.0.31 6379 2
sentinel auth-pass mymaster Zssy56G21Zx
sentinel auth-user mymaster redisuser
sentinel down-after-milliseconds mymaster 10000
redis-02, 192.168.0.32:
bind 192.168.0.32
port 26379
sentinel monitor mymaster 192.168.0.31 6379 2
sentinel auth-pass mymaster Zssy56G21Zx
sentinel auth-user mymaster redisuser
sentinel down-after-milliseconds mymaster 10000
Restart the redis-sentinel daemon to apply the changes:
sudo systemctl restart redis-sentinel
sudo systemctl enable redis-sentinel
Make sure redis-sentinel is running on port 26379. On redis2, you should see something like this:
netstat -tupln | grep redis
Output:
tcp 0 0 192.168.0.31:26379 0.0.0.0:* LISTEN 1170/redis-sentinel
tcp 0 0 192.168.0.31:6379 0.0.0.0:* LISTEN 881/redis-server 12
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 881/redis-server 12
Verify if the Sentinel is observing our Redis replication link by looking at the log file, /var/log/redis/redis-sentinel.log. Make sure you see the following lines:
tail -f /var/log/redis/redis-sentinel.log
Output:
553:X 01 Dec 2021 11:55:36.378 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
553:X 01 Dec 2021 11:55:36.379 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
553:X 01 Dec 2021 11:55:36.379 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=553, just started
553:X 01 Dec 2021 11:55:36.379 # Configuration loaded
553:X 01 Dec 2021 11:55:36.380 * Running mode=sentinel, port=26379.
553:X 01 Dec 2021 11:55:36.380 # Sentinel ID is 1fc2ddf0a56cb12f8cfe91da98d643fde80c7f29
553:X 01 Dec 2021 11:55:36.380 # +monitor master mymaster 192.168.0.31 6379 quorum 2
553:X 01 Dec 2021 11:55:37.399 * +sentinel sentinel 1cf43ff56ead25a832eb0f5ecd9a1e9ae4cdbbc1 192.168.0.31 26379 @ mymaster 192.168.0.31 6379
553:X 01 Dec 2021 11:55:37.525 # +new-epoch 3
553:X 01 Dec 2021 11:55:38.084 * +sentinel sentinel 5d27f1340e5a04dc6703c60d7b60e7c6a121b44c 192.168.0.32 26379 @ mymaster 192.168.0.31 6379
We can get more information on the Sentinel process by using the redis-cli and connect to the Sentinel port 26379. From the app server, run:
redis-cli -h 192.168.0.32 -p 26379 sentinel masters
Output:
1) "name"
2) "mymaster"
3) "ip"
4) "192.168.0.31"
5) "port"
6) "6379"
7) "runid"
8) "ebad0adccca0d81eb09c119b5a44fd5043994d11"
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "274"
19) "last-ping-reply"
20) "274"
21) "down-after-milliseconds"
22) "30000"
23) "info-refresh"
24) "4035"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "1038344"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "1"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "180000"
39) "parallel-syncs"
40) "1"
Check the “num-slaves” value which is 1 and “num-other-sentinels” value which is 2, indicating that we have a total number of 3 Sentinel nodes (one for this node + two other nodes).
Failover Testing
We can now test the failover by simply shutting down the Redis service on redis1(master):
sudo systemctl stop redis-server
After 10 seconds (down-after-milliseconds value), you should see the following output in the /var/log/redis/redis-sentinel.log file:
tail -f /var/log/redis/redis-sentinel.log
Output
553:X 01 Dec 2021 12:41:31.440 # +elected-leader master mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:31.440 # +failover-state-select-slave master mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:31.523 # +selected-slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:31.523 * +failover-state-send-slaveof-noone slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:31.586 * +failover-state-wait-promotion slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:32.371 # +promoted-slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:32.371 # +failover-state-reconf-slaves master mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:32.413 # +failover-end master mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:41:32.413 # +switch-master mymaster 192.168.0.31 6379 192.168.0.32 6379
553:X 01 Dec 2021 12:41:32.413 * +slave slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379
553:X 01 Dec 2021 12:41:42.454 # +sdown slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379
553:X 01 Dec 2021 12:43:17.878 # -sdown slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379
Now, the slave, 192.168.0.32, has been promoted to a master. Once our old master (redis-01) comes back online, you should see something like this reported by Sentinel:
553:X 01 Dec 2021 12:41:32.413 * +slave slave 192.168.0.31:6379 192.168.0.31 6379 @ mymaster 192.168.0.32 6379
The above indicates the old master has been converted to slave and now replicating from the current master, redis2. We can confirm this by checking the replication info on redis2:
redis-cli info replication
Output
Replication
role:master
connected_slaves:0
master_replid:ddf2e69e46513233a51b835babdb18823d6b4a7b
master_replid2:e30c6877b48ef5408cb3f1212d040a271d4d3e92
master_repl_offset:626689
second_repl_offset:611794
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:626689
If we want to promote redis1 to a master status again, we can simply bring down redis2 or use the Sentinel failover command as below:
redis-cli -h 192.168.0.30 -p 26379 sentinel failover mymaster
Output:
553:X 01 Dec 2021 12:54:03.444 # +switch-master mymaster 192.168.0.32 6379 192.168.0.31 6379
553:X 01 Dec 2021 12:54:03.444 * +slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379
553:X 01 Dec 2021 12:54:13.653 * +convert-to-slave slave 192.168.0.32:6379 192.168.0.32 6379 @ mymaster 192.168.0.31 6379
Check:
redis-01
redis-cli info replication
Output
Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.32,port=6379,state=online,offset=795039,lag=1
master_replid:2b5a609a0d28227480c9126d95a915a870002811
master_replid2:ddf2e69e46513233a51b835babdb18823d6b4a7b
master_repl_offset:795039
second_repl_offset:766456
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:635864
repl_backlog_histlen:159176