High Availability DRBD on RHEL 8
DRBD®– software is a distributed replicated storage system for the Linux platform. It is implemented as a kernel driver, several userspace management applications, and some shell scripts. DRBD Heartbeat cluster is a good Active/Passive cluster solution for small-scale applications using two servers in active and passive mode. This means, only one server will work at a time while keeping the other server as a backup with real-time data updates. DRBD is a kernel-level service that will replicate your block devices with the second server. So all the data that is required for working your application must need to place in that hard disk partition. Also, make sure that both of your servers need the same amount of free space.
Heartbeat is a service that will manage the IP high availability and other services in your servers. You can also integrate CRM along with heartbeat for your big cluster projects.
Lab Environment:
Below are my environment specifications:
Prerequisites:
- Operating System RHEL 8
- Red Hat Developer Subscription (register here)
Setup DRBD
Let’s take a look at the steps required to set up DRBD on RHEL 8. The steps look something like the following:
Prepare systems, execute on all nodes
Create LVM on Secondary Disk
# mokutil --sb-state
# lsblk
# pvcreate /dev/sdb
# vgcreate drbdpool /dev/sdb
# lvcreate -n drbddata -l100%FREE drbdpool
Register server to RHSM
# subscription-manager register
# subscription-manager attach --auto
Enable repo and make system update
# subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms
# dnf repolist
# dnf update
Setup Cluster Nodes
Install pacemaker and other dependencies
# dnf install pcs pacemaker fence-agents-all pcp-zeroconf psmisc policycoreutils-python-utils chrony -y
# timedatectl set-ntp yes
# systemctl restart chronyd
# timedatectl
# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --reload
Setup Cluster
# echo "redhat" | passwd --stdin hacluster
# systemctl enable --now pcsd
# systemctl status pcsd
# pcs host auth rhel-ha1 rhel-ha2
Start and verify cluster, execute on one of the nodes:
[root@rhel-ha1 ~]# pcs cluster setup mycluster --start rhel-ha1 rhel-ha2
[root@rhel-ha1 ~]# pcs cluster enable --all
[root@rhel-ha1 ~]# systemctl enable corosync && systemctl enable pacemaker
[root@rhel-ha1 ~]# corosync-cfgtool -s
[root@rhel-ha1 ~]# corosync-cmapctl | grep members
[root@rhel-ha1 ~]# pcs status corosync
*Disable Fencing (optional)
[root@rhel-ha1 ~]# pcs property set stonith-enabled=false
[root@rhel-ha1 ~]# pcs status
Setup DRBD
Install DRBD 9.0 from ELREPO
# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm -y
# dnf repolist
# dnf search drbd
# dnf install kmod-drbd90.x86_64 drbd90-utils.x86_64 -y
Configure DRBD to use port 7789, so allow that port from each host to the other.
# lsmod | grep -i drbd
# modprobe drbd
# echo drbd > /etc/modules-load.d/drbd.conf
# semanage permissive -a drbd_t
# firewall-cmd --add-port=7789/tcp --permanent
# firewall-cmd --reload
# reboot
# lsmod | grep -i drbd
All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf.
By convention, /etc/drbd.d/global_common.conf
contains the global and common sections of the DRBD configuration, whereas the .res files contain one resource section each.
# cat /etc/drbd.conf
# mv /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.bak
# cat << EOF > /etc/drbd.d/global_common.conf
global {
usage-count no;
}
common {
net {
protocol C;
}
}
EOF
Configure DRBD resource
A per-resource configuration file is usually named /etc/drbd.d/<resource>.res
. Any DRBD resource you define must be named by specifying a resource name in the configuration. In this example, the name of our resource as resource0
# cat << EOF > /etc/drbd.d/resource0.res
resource resource0 {
protocol C;
on rhel-ha1 {
address 192.168.1.105:7789;
device /dev/drbd0;
disk /dev/drbdpool/drbddata;
meta-disk internal;
}
on rhel-ha2 {
address 192.168.1.104:7789;
device /dev/drbd0;
disk /dev/drbdpool/drbddata;
meta-disk internal;
}
}
EOF
Create and Enable DRBD Resource
Each of the following steps must be completed on all the cluster nodes.
# drbdadm create-md resource0
# systemctl enable --now drbd
The next step consists of drbd configuration includes enabling resource0
resources in order to finish allocating both disk and network resources for its operation. drbdadm
status output should now contain information similar to the following related to Linux Disk Replication.
# drbdadm up resource0
# drbdadm status resource0Configure DRBD primary server
# drbdadm primary resource0 --force
# cat /proc/drbd
create a file system on our DRBD device to be able to perform read-write operations. This example will create ext4
file system on /dev/drbd0.
This step is required on any one of the Cluster nodes and the changes will be replicated across as our DRBD is already running
# mkfs.ext4 /dev/drbd0
# mkdir -p /share
# mount /dev/drbd0 /share
# touch /share/DRBD{1..5}
# umount /shareCreate a mount point with same name on another cluster nodes
# mkdir /share
Configure the Cluster for the DRBD device
Our first resource will be a unique IP address that the cluster can bring up on either node. Regardless of where any cluster service(s) are running, end users need a consistent address to contact them on
# pcs resource create Virtual_IP IPaddr2 ip=192.168.1.110 cidr_netmask=24 op monitor interval=30s
Next create a heartbeat resource for DRBD Cluster File System, then add DRBD Storage as Master/Slave using Clone resource in the KVM Cluster.
# pcs resource create DRBD_Data ocf:linbit:drbd drbd_resource=resource0 promotable promoted-max=1 promoted-node-max=1 clone-max=3 clone-node-max=1 notify=true
# pcs resource create DRBD_FS ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/share fstype=ext4
# pcs status
Configure Resource Constraint
# pcs constraint colocation add DRBD_FS with DRBD_Data-clone INFINITY with-rsc-role=Master
# pcs constraint order promote DRBD_Data-clone then start DRBD_FS
# pcs constraint order stop DRBD_FS then demote DRBD_Data-clone score=500
# pcs constraint location DRBD_Data-clone prefers rhel-ha1.localdomain=100
# pcs constraint location DRBD_Data-clone prefers rhel-ha2.localdomain=50
# pcs constraint location Virtual_IP prefers rhel-ha1.localdomain=50Restart resource on the cluster
# pcs resource cleanup
# pcs status
Verify resource constraint
Verify Failover DRBD
On rhel-ha1 node, make this cluster node as standby to verify DRBD Failover. Check cluster using pcs status
command. As expected our DRBD resource has switched to a different KVM Cluster node and is started on rhel-ha2
On rhel-ha2 node, verify content on /share are mounting and available and not lost
Lets bring back our standby cluster node to active state, using pcs node unstandby
command. Once the cluster node is active, drbd0
device was able to re-connect to the respective cluster node. Hence the DRBD failover is working as expected.
References:
- https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Clusters_from_Scratch/index.html
- https://www.golinuxcloud.com/
- https://blog.hostseo.com/drbd-heartbeat-active-passive-high-availability-cluster/
#DRBD #pacemaker #RHEL #HighAvailability