How to Setup a Redis Cluster in CentOS 8 – Part 3

[‘

n

Redis Cluster is a builtin Redis feature that supports automatic sharding, replication and high availability which was previously implemented using Sentinels. It is designed for two major purposes: one is to automatically split your dataset among multiple instances and secondly to provide some degree of availability during partitions, to continue operations when some instances (especially masters) fail or are not able to communicate with the majority of nodes in the cluster.

n

However, the cluster stops to operate in the event of larger failures (e.g when the majority of master instances are unavailable). Also, if a master and slave fail at the same time, the cluster cannot continue normal operations (though the workaround is to add more nodes or create an asymmetry in the cluster, to auto-change the cluster layout).

n

According to the Redis cluster documentation, the “minimal cluster” that works as expected requires to contain at least 3 master nodes. But the most suitable setup for high availability should have at least 6 nodes with three masters and three slaves, each master having a slave.

n

Important: Redis Cluster also has some limitations which are lack of support for NATted environments as well those where IP addresses or TCP ports are remapped for instance under Docker. Additionally, not every client library supports it.

n

This article shows how to set up a Redis Cluster (with Cluster-Mode Disabled) in CentOS 8. It includes how to install Redis, configure the cluster nodes, create a cluster and test the cluster failover.

n

Note: For this guide, we will use fresh/empty Redis instances to run the cluster mode. The cluster mode will not work with some configurations done in the first two guides of our Redis Series, particularly it doesn’t work when the replica of the parameter is being used.

n

Prerequisites:

n

    n

  1. Servers with CentOS 8 Installation
  2. n

n

Test Environment Setup

n

Redis Master1: 10.42.0.247rnRedis Master2: 10.42.0.197rnRedis Master3: 10.42.0.132rnrnRedis Slave1: 10.42.0.200rnRedis Slave2: 10.42.0.21rnRedis Slave3: 10.42.0.34rn

n

Redis Cluster Logical Diagram
Redis Cluster Logical Diagram

n

Our setup has 3 read/write master nodes and 3 read-only replica nodes, each master having one replica, so three shards contain all of the cluster’s data in each node. An application API or CLI client can write only to the master nodes but read from any node in the cluster.

n

Step 1: Installing Redis on All Nodes

n

1. Log into all the instances via SSH, then run the following command to install the Redis module using the DNF package manager as shown.

n

# dnf module install redisrn

n

2. Next, start the Redis service, enable it to automatically start at system boot and check its status to verify that it is running (verify the service on all the 6 instances):

n

# systemctl start redisrn# systemctl enable redisrn# systemctl status redisrn

n

Step 2: Configuring Redis Instances on all Nodes

n

3. This section describes how to configure the Redis cluster nodes. Remember to perform the configurations here on all the nodes.

n

Use the /etc/redis.conf configuration file to configure the Redis server. As a recommended practice, create a backup of the original file before editing it using a command-line text editor of your choice.

n

# cp /etc/redis.conf /etc/redis.conf.origrn# vi /etc/redis.confrn

n

4. Next, find the following configuration parameters and edit their values as shown. The bind parameter sets the interface of the Redis server will listen on, set its value to the instance LAN IP. Remove the 127.0.0.1 because we realized leaving it there slows down the process of cluster creation, particularly the stage of joining the cluster.

n

bind  10.42.0.247rn

n

Then set the protected-mode to no to allow connections from the other instances on the cluster.

n

protected-mode norn

n

The port parameter defines the port the Redis server will listen on for connections, the default is 6379. This is the data port for communicating with clients.

n

port 6379rn

n

Set Redis Listen Interface and Port
Set Redis Listen Interface and Port

n

5. The next set of parameters will enable the cluster mode and set some of its useful features. The cluster-enabled parameter, when set to yes, activates the cluster mode.

n

cluster-enabled yesrn

n

Next, the cluster-config-file parameter sets the name of a cluster node’s cluster configuration file (e.g nodes-6379.conf). The file is created in the working directory (default is /var/lib/redis defined using the dir parameter) and is not user editable.

n

cluster-config-file nodes-6379.confrn

n

The next useful cluster option is cluster-node-timeout, it is used to set the maximum amount of time in milliseconds an instance can be unavailable for it to be considered in a failure state. A value of 15000 is equivalent to 15 seconds.

n

cluster-node-timeout 15000rn

n

Set Cluster Node Timeout
Set Cluster Node Timeout

n

6. We also need to enable Redis persistence on disk. We can use one of the persistence modes, that is the Append Only File (AOF): it logs (in the file appendonly.aof created under the working directory) every write operation successfully received by the server. The data will be played during the server startup to reconstruct the original dataset.

n

To enable it, set the appendonly parameter to yes.

n

appendonly yesrn

n

Set Persistence Options
Set Persistence Options

n

7. After making all the changes, restart the Redis service on all the nodes to apply the recent changes.

n

# systemctl restart redisrn

n

8. At this point, every cluster node should now have an ID. You can check this in the logfile located at /var/log/redis/redis.log.

n

# cat /var/log/redis/redis.logrn

n

Check Cluster Node Log File
Check Cluster Node Log File

n

9. Next, open port 6397 and 16379 on all the instances. The later port is used for the cluster bus (a node-to-node communication channel using a binary protocol). This is a basic requirement for the Redis cluster TCP connections.

n

# firewall-cmd --zone=public --permanent --add-port=6379/tcp rn# firewall-cmd --zone=public --permanent --add-port=16379/tcp rn# firewall-cmd --reloadrn

n

Step 3: Creating the Redis Cluster

n

10. To create the cluster, use the redis-cli command-line client as follows. The --cluster create enables cluster creation and --cluster-replicas 1 means create one replica per master.

n

For our setup which has 6 nodes, we will have 3 masters and 3 slaves.

n

Note that the first 6 nodes will be considered masters (M) and the next three will be considered slaves (S). The first slave i.e 10.42.0.200:6379 replicates the first master i.e 10.42.0.247:6379, the second slave replicates the second master, in that order.

n

The following command is formatted in a way that the outcome will represent our logical setup above.

n

# redis-cli --cluster create 10.42.0.247:6379 10.42.0.197:6379 10.42.0.132:6379 10.42.0.200:6379 10.42.0.21:6379 10.42.0.34:6379 --cluster-replicas 1rn

n

Create Redis Cluster
Create Redis Cluster

n

11. Once the cluster creation is successful, run the following command on any host (specify its IP address using the -h flag) to list all cluster nodes.

n

# redis-cli -h 10.42.0.247 -p 6379 cluster nodesrn

n

You should be able to see all the cluster nodes, with the slaves indicating their masters, as shown in the following screenshot.

n

Check All Cluster Nodes on Any Node
Check All Cluster Nodes on Any Node

n

The different fields are in this order: node ID, IP address:port, flags, last ping sent, last pong received, configuration epoch, link-state, slots (for masters).

n

Step 4: Testing Redis Cluster Failover

n

12. In this section, we will demonstrate how to test a cluster failover. First, let’s take note of the masters.

n

# redis-cli -h 10.42.0.247 -p 6379 cluster nodes  | grep masterrn

n

List Redis Cluster Masters
List Redis Cluster Masters

n

Also, take note of the Redis slaves.

n

# redis-cli -h 10.42.0.247 -p 6379 cluster nodes  | grep slavern

n

List All Redis Cluster Slaves
List All Redis Cluster Slaves

n

13. Next, let’s stop the Redis service on one of the master nodes e.g 10.42.0.197 and check all master nodes in the cluster.

n

# systemctl stop redisrn# redis-cli -h 10.42.0.247 -p 6379 cluster nodes | grep masterrn

n

From the following screenshot, you can see that the node 10.42.0.197:6367 is in fail state and its slave 10.42.0.21:6379 has been promoted to master status.

n

Check Cluster Failover Status
Check Cluster Failover Status

n

14. Now let’s start the Redis service once again on the failed node and check all the masters in the cluster.

n

# systemctl start redisrn# redis-cli -h 10.42.0.247 -p 6379 cluster nodes  | grep masterrn

n

Check All Redis Cluster Master Status
Check All Redis Cluster Master Status

n

Also, check the cluster slaves to confirm that the failed master is now a slave.

n

# redis-cli -h 10.42.0.247 -p 6379 cluster nodes  | grep slavern

n

Check All Redis Cluster Slaves
Check All Redis Cluster Slaves

n

Step 5: Testing Data Replication Across the Redis Cluster

n

15. This last section explains how to verify cluster data replication. We will create a key and value on one of the masters, then try to read it from all the cluster nodes as follows. Use the -c switch to enable cluster support under redis-cli utility and access data in cluster mode.

n

# redis-cli -c -h 10.42.0.247 -p 6379 set name 'TecMint.com'rn# redis-cli -c -h 10.42.0.247 -p 6379 get namern# redis-cli -c -h 10.42.0.21 -p 6379 get namern# redis-cli -c -h 10.42.0.132 -p 6379 get namern# redis-cli -c -h 10.42.0.200 -p 6379 get namern# redis-cli -c -h 10.42.0.197 -p 6379 get namern# redis-cli -c -h 10.42.0.34 -p 6379 get namern

n

Verify Redis Cluster Data Replication
Verify Redis Cluster Data Replication

n

The bottom line is the Redis Cluster is the preferred way to get automatic sharding, replication, and high availability. There are many other well-documented configuration parameters in the rest of the /etc/redis.conf file, you can find more information in the official documentation: Redis cluster tutorial and Redis cluster specification.

n

This brings us to the end of the three-part Redis tutorial series. The feedback form below can be used to post questions or comments.

n

‘]