High Availability DRBD on RHEL 8

6 min readDec 28, 2021

DRBD®– software is a distributed replicated storage system for the Linux platform. It is implemented as a kernel driver, several userspace management applications, and some shell scripts. DRBD Heartbeat cluster is a good Active/Passive cluster solution for small-scale applications using two servers in active and passive mode. This means, only one server will work at a time while keeping the other server as a backup with real-time data updates. DRBD is a kernel-level service that will replicate your block devices with the second server. So all the data that is required for working your application must need to place in that hard disk partition. Also, make sure that both of your servers need the same amount of free space.

Heartbeat is a service that will manage the IP high availability and other services in your servers. You can also integrate CRM along with heartbeat for your big cluster projects.

Lab Environment:

Below are my environment specifications:

Prerequisites:

Operating System RHEL 8
Red Hat Developer Subscription (register here)

Setup DRBD

Let’s take a look at the steps required to set up DRBD on RHEL 8. The steps look something like the following:

Prepare systems, execute on all nodes

Create LVM on Secondary Disk

# mokutil --sb-state
# lsblk
# pvcreate /dev/sdb
# vgcreate drbdpool /dev/sdb
# lvcreate -n drbddata -l100%FREE drbdpool

# subscription-manager register
# subscription-manager attach --auto

Enable repo and make system update

# subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms
# dnf repolist
# dnf update

Setup Cluster Nodes

Install pacemaker and other dependencies

# dnf install pcs pacemaker fence-agents-all pcp-zeroconf psmisc policycoreutils-python-utils chrony -y
# timedatectl set-ntp yes
# systemctl restart chronyd
# timedatectl
# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --reload

Setup Cluster

# echo "redhat" | passwd --stdin hacluster
# systemctl enable --now pcsd
# systemctl status pcsd
# pcs host auth rhel-ha1 rhel-ha2

Start and verify cluster, execute on one of the nodes:

[root@rhel-ha1 ~]# pcs cluster setup mycluster --start rhel-ha1 rhel-ha2
[root@rhel-ha1 ~]# pcs cluster enable --all
[root@rhel-ha1 ~]# systemctl enable corosync && systemctl enable pacemaker
[root@rhel-ha1 ~]# corosync-cfgtool -s
[root@rhel-ha1 ~]# corosync-cmapctl | grep members
[root@rhel-ha1 ~]# pcs status corosync

*Disable Fencing (optional)

[root@rhel-ha1 ~]# pcs property set stonith-enabled=false
[root@rhel-ha1 ~]# pcs status

Setup DRBD

Install DRBD 9.0 from ELREPO

# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm -y
# dnf repolist
# dnf search drbd
# dnf install kmod-drbd90.x86_64 drbd90-utils.x86_64 -y

Configure DRBD to use port 7789, so allow that port from each host to the other.

# lsmod | grep -i drbd
# modprobe drbd
# echo drbd > /etc/modules-load.d/drbd.conf
# semanage permissive -a drbd_t
# firewall-cmd --add-port=7789/tcp --permanent
# firewall-cmd --reload
# reboot
# lsmod | grep -i drbd

All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf. By convention, /etc/drbd.d/global_common.conf contains the global and common sections of the DRBD configuration, whereas the .res files contain one resource section each.

# cat /etc/drbd.conf
# mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.bak
# cat << EOF > /etc/drbd.d/global_common.conf
global {
 usage-count no;
}
common {
 net {
 protocol C;
 }
}
EOF

Configure DRBD resource

A per-resource configuration file is usually named /etc/drbd.d/<resource>.res. Any DRBD resource you define must be named by specifying a resource name in the configuration. In this example, the name of our resource as resource0

# cat << EOF > /etc/drbd.d/resource0.res
resource resource0 {
 protocol C;
 on rhel-ha1 {
   address 192.168.1.105:7789;
   device /dev/drbd0;
   disk /dev/drbdpool/drbddata;
   meta-disk internal;
 }
 on rhel-ha2 {
   address 192.168.1.104:7789;
   device /dev/drbd0;
   disk /dev/drbdpool/drbddata;
   meta-disk internal;
 }
}
EOF

Create and Enable DRBD Resource

Each of the following steps must be completed on all the cluster nodes.
# drbdadm create-md resource0
# systemctl enable --now drbd

The next step consists of drbd configuration includes enabling resource0 resources in order to finish allocating both disk and network resources for its operation. drbdadm status output should now contain information similar to the following related to Linux Disk Replication.

# drbdadm up resource0
# drbdadm status resource0Configure DRBD primary server
# drbdadm primary resource0 --force 
# cat /proc/drbd

create a file system on our DRBD device to be able to perform read-write operations. This example will create ext4 file system on /dev/drbd0. This step is required on any one of the Cluster nodes and the changes will be replicated across as our DRBD is already running

# mkfs.ext4 /dev/drbd0
# mkdir -p /share
# mount /dev/drbd0 /share
# touch /share/DRBD{1..5}
# umount /shareCreate a mount point with same name on another cluster nodes
# mkdir /share

Configure the Cluster for the DRBD device

Our first resource will be a unique IP address that the cluster can bring up on either node. Regardless of where any cluster service(s) are running, end users need a consistent address to contact them on

# pcs resource create Virtual_IP IPaddr2 ip=192.168.1.110 cidr_netmask=24 op monitor interval=30s

Next create a heartbeat resource for DRBD Cluster File System, then add DRBD Storage as Master/Slave using Clone resource in the KVM Cluster.

# pcs resource create DRBD_Data ocf:linbit:drbd drbd_resource=resource0 promotable promoted-max=1 promoted-node-max=1 clone-max=3 clone-node-max=1 notify=true
# pcs resource create DRBD_FS ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/share fstype=ext4
# pcs status

Configure Resource Constraint

# pcs constraint colocation add DRBD_FS with DRBD_Data-clone INFINITY with-rsc-role=Master
# pcs constraint order promote DRBD_Data-clone then start DRBD_FS 
# pcs constraint order stop DRBD_FS then demote DRBD_Data-clone score=500
# pcs constraint location DRBD_Data-clone prefers rhel-ha1.localdomain=100
# pcs constraint location DRBD_Data-clone prefers rhel-ha2.localdomain=50
# pcs constraint location Virtual_IP prefers rhel-ha1.localdomain=50Restart resource on the cluster
# pcs resource cleanup
# pcs status

Verify resource constraint

Verify Failover DRBD

On rhel-ha1 node, make this cluster node as standby to verify DRBD Failover. Check cluster using pcs status command. As expected our DRBD resource has switched to a different KVM Cluster node and is started on rhel-ha2

On rhel-ha2 node, verify content on /share are mounting and available and not lost

Lets bring back our standby cluster node to active state, using pcs node unstandby command. Once the cluster node is active, drbd0 device was able to re-connect to the respective cluster node. Hence the DRBD failover is working as expected.