How to Setup Redis For High Availability with Sentinel in CentOS 8 – Part 2

[‘

n

Redis provides high availability via Redis Sentinel distributed system. Sentinel helps to monitor Redis instances, detect failures and will do roles switches automatically thus enabling a Redis deployment to resist any kind of failures.

n

It features monitoring of Redis instances (master and replicas), supports notification of other services/processes or the system administrator via a script, automatic failover to promote a replica to a master when the master goes down and provides configuration for clients to discover the current master offering a particular service.

n

This article demonstrates how to set up Redis for high availability with Redis Sentinel in CentOS 8, including configuring sentinels, checking the setup status and testing a Sentinel failover.

n

Prerequisite:

n

    n

  1. How To Setup Redis Replication (with Cluster-Mode Disabled) in CentOS 8 – Part 1
  2. n

n

Test Environment Setup

n

Master Server and Sentinel1: 10.42.0.247rnRedis Replica1 and Sentinel2: 10.42.0.21rnRedis Replica2 and Sentinel3: 10.42.0.34rn

n

Redis Sentinel Setup Logical Diagram
Redis Sentinel Setup Logical Diagram

n

According to the Redis Sentinel documentation, one needs at least three Sentinel instances for a robust deployment. Considering our set up above, if the master fails, Sentinels2 and Sentinel3 will agree about the failure and will be able to authorize a failover, making client operations able to continue.

n

Step 1: Starting and Enabling Redis Sentinel Service

n

1. On CentOS 8, the Redis Sentinel service is installed alongside the Redis server (which we already did in the Redis Replication Setup).

n

To start the Redis sentinel service and enable it to automatically start at system boot, use the following systemctl commands. Also, confirm that it is up and running by checking its status (do this on all the nodes):

n

# systemctl start redis-sentinelrn# systemctl enable redis-sentinelrn# systemctl status redis-sentinelrn

n

Start Redis Sentinel Service
Start Redis Sentinel Service

n

Step 2: Configuring Redis Sentinel on All Redis Nodes

n

2. In this section, we explain how to configure Sentinel on all our nodes. The Sentinel service has a similar configuration format as the Redis server. To configure it, use the /etc/redis-sentinel.conf self-documented configuration file.

n

First, create a backup of the original file and open it for editing.

n

# cp /etc/redis-sentinel.conf /etc/redis-sentinel.conf.origrn# vi /etc/redis-sentinel.confrn

n

3. By default, Sentinel listens on port 26379, verify this on all the instances. Note that you have to leave the bind parameter commented out (or set to 0.0.0.0).

n

port 26379rn

n

Set Sentinel Listen Interface and Port
Set Sentinel Listen Interface and Port

n

4. Next, tell Sentinel to monitor our master, and to consider it in the “Objectively Down” state only if at least 2 quorum sentinels agree. You can replace “mymaster” with a custom name.

n

#On Master Server and Sentinel1rnsentinel monitor mymaster 127.0.0.1 6379 2rnrn#On Replica1 and Sentinel2rnsentinel monitor mymaster 10.42.0.247 6379 2rnrn#On Replica1 and Sentinel3rnsentinel monitor mymaster 10.42.0.247 6379 2rn

n

Set Redis Master to Monitor
Set Redis Master to Monitor

n

Important: The sentinel monitor statement MUST be placed before the sentinel auth-pass statement to avoid the error “No such master with the specified name.” when restarting the sentinel service.

n

5. If the Redis master to monitor has a password set (in our case the master has), provide the password so that Sentinel instance can authenticate with the protected instance.

n

 rnsentinel auth-pass mymaster [emailxa0protected]rn

n

Set Master Auth Password
Set Master Auth Password

n

6. Then set the number of milliseconds the master (or any attached replica or sentinel) should be unreachable to consider it in the “Subjectively Down” state.

n

The following configuration means that the master will be considered failing as soon as we don’t receive any reply from our pings within 5 seconds (1 second is equivalent to 1000 milliseconds).

n

sentinel down-after-milliseconds mymaster 5000rn

n

Set Down Time for Master
Set Down Time for Master

n

7. Next, set the failover timeout in milliseconds which defines many things (read the documentation of the parameter in the configuration file).

n

sentinel failover-timeout mymaster 180000rn

n

Set Fail Over Timeout
Set Fail Over Timeout

n

8. Then set the number of replicas that can be reconfigured to use the new master after a failover at the same time. Since we have two replicas, we will set one replica as the other will be promoted to the new master.

n

sentinel parallel-syncs mymaster 1rn

n

Set Number of Parallel Sync Replicas
Set Number of Parallel Sync Replicas

n

Note that the configuration files on Redis Replica1 and Sentinel2, and Reddis Replica1 and Sentinel2 should be identical.

n

9. Next, restart the Sentinel services on all nodes to apply the recent changes.

n

# systemctl restart redis-sentinelrn

n

10. Next, open port 26379 in the firewall on all nodes to enable the Sentinel instances to start talking, receive connections from the other Sentinel instances, using the firewall-cmd.

n

# firewall-cmd --zone=public --permanent --add-port=26379/tcprn# firewall-cmd --reloadrn

n

11. All the replicas will be automatically discovered. Importantly, Sentinel will update the configuration automatically with additional information about replicas. You can confirm this by opening the Sentinel configuration file for each instance and look through it.

n

For example, when you look at the end of the master’s configuration file, you should see the known-sentinels and known-replica statements as shown in the following screenshot.

n

Auto Generated Config in Master
Auto Generated Config in Master

n

It’s should be the same case on replica1 and replica2.

n

Auto Generated Config in Replica1
Auto Generated Config in Replica1

n

Auto Generated Config in Replica2
Auto Generated Config in Replica2

n

Note that the Sentinel configuration is also rewritten/updated every time a replica is promoted to master status during a failover and every time a new Sentinel is discovered in the setup.

n

Step 3: Check the Redis Sentinel Setup Status

n

12. Now check the Sentinel status/information on the master, using the info sentinel command as follows.

n

# redis-cli -p 26379 info sentinelrn

n

From the output of the command as seen in the following screenshot, we have two replicas/slaves and three sentinels.

n

Check Sentinel Info on Master
Check Sentinel Info on Master

n

13. To shows detailed information about the master (called mymaster), use the sentinel master command.

n

# redis-cli -p 26379 sentinel master mymasterrn

n

Show Detailed Info About Sentinel Master
Show Detailed Info About Sentinel Master

n

14. To shows detailed information about the slaves and sentinels, use the sentinel slaves command and sentinel sentinels command respectively.

n

# redis-cli -p 26379 sentinel slaves mymasterrn# redis-cli -p 26379 sentinel sentinels mymasterrn

n

15. Next, ask the address of the master by name from the slave instances using the sentinel get-master-addr-by-name command as follows.

n

The output should be the IP address and port of the current master instance:

n

# redis-cli -p 26379 sentinel get-master-addr-by-name mymasterrn

n

Get the Address of Master by Name on Slaves
Get the Address of Master by Name on Slaves

n

Step 4: Test the Sentinel Failover

n

16. Finally, let’s test automatic failover in our Sentinel setup. On the Redis/Sentinel master, make the Redis master (running on port 6379) to sleep for 60 seconds. Then query the address of the current master on the replicas/slaves as follows.

n

# redis-cli -p 6379rn127.0.0.1:6379> AUTH [emailxa0protected]rn127.0.0.1:6379>  debug sleep 60rn# redis-cli -p 26379 sentinel get-master-addr-by-name mymasterrn# redis-cli -p 26379 sentinel get-master-addr-by-name mymasterrn

n

From the output for the query, the new master is now replica/slave2 with IP address 10.42.0.34 as seen in the following screenshot.

n

Test Redis Sentinel Failover
Test Redis Sentinel Failover

n

You can get more information from the Redis Sentinel documentation. But if you have any thoughts to share or queries, the feedback form below is your gateway to us.

n

In the next and last part of this series, we will look at how to set up a Redis Cluster in CentOS 8. It will be an independent article from the first two.

n

‘]