The Nuts and Bolts of Redis Cluster

Redis is an in-memory data structure store that can be used as a database, cache, and message broker. With cluster-mode enabled, it provides better availability and scalability, making it a popular choice for many applications. To understand the basics of Redis cluster, I recommend reading the official Redis cluster 101 tutorial, which gives a clear overview and some basic usages.

In this article, we will focus on more practical topics, such as setting up Redis clusters in Docker, Golang library support, and discussing some limitations of Redis clusters.

While the official tutorial mentioned above covers a way to manually start a Redis cluster locally, it would be more efficient if we could recreate it using container technologies and streamline the process. Docker Compose comes to the rescue, allowing us to manage multiple containers easily.

By searching for docker, redis, and cluster, you will likely find a Docker image from Bitnami, which we will be using in this tutorial. The Docker Hub page for this image can be found here.

The Docker Hub page for this image is at here.

$ docker pull bitnami/redis-cluster:latest

Docker Compose is a tool for managing or composing multiple containers. You describe multiple containers in one file and can easily manage them through a single CLI entrance, which is docker-compose or docker compose.

The Bitnami image provided is for a single Redis node, and we need multiple nodes to create a cluster. To do this, we will use Docker Compose to create multiple containers, manage the network, and set up persistent storage.

According to the official tutorial, a Redis cluster should have at least six nodes (three master nodes and three replicas). We can lay out the containers in a docker-compose.yml file.

Here is how the containers are laid out in the docker-compose.yml file:

version: "2"
services:
  redis-node-0:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-0:/redis/data
    environment:
      - "ALLOW_EMPTY_PASSWORD=yes"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

  redis-node-1:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-1:/redis/data
    environment:
      - "ALLOW_EMPTY_PASSWORD=yes"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

  redis-node-2:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-2:/redis/data
    environment:
      - "ALLOW_EMPTY_PASSWORD=yes"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

  redis-node-3:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-3:/redis/data
    environment:
      - "ALLOW_EMPTY_PASSWORD=yes"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

  redis-node-4:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-4:/redis/data
    environment:
      - "ALLOW_EMPTY_PASSWORD=yes"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

  redis-node-5:
    image: docker.io/bitnami/redis-cluster:6.2
    volumes:
      - redis-cluster_data-5:/redis/data
    environment:
      - "ALLOW_EMPTY_PASSWORD=yes"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"

  redis-cluster-init:
    image: docker.io/bitnami/redis-cluster:6.2
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
      - redis-node-5
    environment:
      - "REDIS_CLUSTER_REPLICAS=1"
      - "REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5"
      - "REDIS_CLUSTER_CREATOR=yes"

volumes:
  redis-cluster_data-0:
    driver: local
  redis-cluster_data-1:
    driver: local
  redis-cluster_data-2:
    driver: local
  redis-cluster_data-3:
    driver: local
  redis-cluster_data-4:
    driver: local
  redis-cluster_data-5:
    driver: local

Once the docker-compose.yml file is set up, we can start the services using docker-compose up -d, stop them with docker-compose stop, and stop and remove them using docker-compose down.

In our current configuration, none of the Redis nodes are accessible from the outside, not even from the host. This is related to how nodes in a Redis cluster communicate with each other.

"Wouldn't it be super easy if we just exposed the port 6379 of every Redis node service", you may ask. Well, that's where we're running into the limitations of this approach, this is mostly related to how nodes in a Redis cluster communicate with each other.

The current mode we make the Docker Compose containers run in is the host mode, which means containers can access each other through their hostnames like redis-node-0. But the host or any other machine from outside the network cannot access them at all. If we were to expose the port 6379 of every Redis node, and map them to a port available on the host machine, e.g. 7000, 7001, 7002, 7003, 7004 and 7005 (6 nodes that is), it might not work the way we expected. According to the official tutorial, Redis under cluster mode listens on 2 ports other than just 1, one being what we're already familiar with, which is used for the client to connect to. The other one is used to sync data and track status between the nodes. So this is one of the reasons why simply exposing one port isn't enough.

As mentioned above, in host mode, containers connect with each other through hostnames. When our client randomly connects to one of the nodes and request to fetch a value for a specific key, if the key is unfortunately sharded elsewhere, and need a redirection to the actual node that stores the key, the node we randomly landed should return the address of the desired node. This address, however, is where the issues arise. Because nodes are aware of each other in the docker network using each other's hostname, which is the internal hostname of the docker network the client from outside cannot successfully reach. So the redirect address the node returns is not useful to us, and we lose a way to know where the right node we should be landing.

One solution to this issue is to include your client within the Docker network. This allows the client to access the Redis nodes using their hostnames and ensures that the redirect address provided by the nodes is valid.

To do this, add a new service to the docker-compose.yml file. For example, if your client is a Golang service, you can add the following configuration:

golang-cli:
  image: golang:1.16.3
  container_name: redis-cluster-golang-cli
  entrypoint: /bin/sh
  volumes:
    - ../golib:/mnt/project/
  tty: true

After starting all the services, you can attach to the golang-cli console using the command:

docker exec -it redis-cluster-golang-cli /bin/bash

-it or --interactive flag means we are getting into the interactive mode that will keep stdin open so that we can type things. redis-cluster-golang-cli is the container name, you can also use the container id. /bin/bash is the shell we're using, /bin/sh is also fine if bash isn't available in the image.

A popular Golang library with Redis cluster support is go-redis. This library handles redirection for us, so we don't have to worry about the redirect address provided by the nodes.

The library makes it simple to work with Redis clusters. In most cases, you treat it as a normal client that handles clustering in the background. Here is an example of how to use the go-redis library:

// You can either provide 1 or more node addresses since
// the library will find the rest of the cluster itself.
rdb := redis.NewClusterClient(&redis.ClusterOptions{
        Addrs:     []string{"redis-node-0:6379"},
    })

// Treat it as a normal Redis client
err := rdb.Set(ctx, "key", "value", 0).Err()
val, err := rdb.Get(ctx, "key").Result()
if err != nil {
    panic(err)
}
fmt.Println("key", val)

However, there are some limitations, especially when it comes to batch commands. For example, using the del command to delete multiple keys may not work. Redis Cluster does not support multiple-key commands if these keys don't belong to the same slot. Many other commands that touch multiple keys have this restriction as well.

In summary, Redis clusters provide improved scalability and availability, making them an exciting option to explore with Docker and Golang. With the right setup and library support, you can take advantage of these features while overcoming some of the limitations that come with Redis clusters.