ETCD Backup and Restore in Kubernetes

In this blog, we will discuss the importance of ETCD backup and restore in Kubernetes and walk you through the process using Kubernetes documentation as a guide.

Synopsis:

Why Backing Up ETCD is Essential
How to Backup etcd Using etcdctl

Taking the etcd Snapshot
Verifying the Snapshot
Restoring etcd from a Snapshot

Why Backing Up ETCD is Essential

ETCD, the distributed key-value store used by Kubernetes, stores all cluster data, including configurations, service states, and much more. Backing up ETCD data is vital for several reasons:

Disaster Recovery: In case of hardware failures, data corruption, or other unforeseen events, having a backup of ETCD ensures you can recover your Kubernetes cluster to a previous state.
Upgrades and Migrations: Backing up ETCD is crucial before upgrading or migrating your cluster. If something goes wrong during the process, you can revert to the previous state.
Data Loss Prevention: Protects against data loss due to human errors or misconfigurations.

Now, let’s dive into the process of backing up and restoring ETCD in Kubernetes.

How to Backup etcd Using etcdctl

Prerequisites:

Kubernetes cluster
ETCDCTL utility

You can install ETCDCTL using the command

				
					sudo apt install etcd-client

Gathering Required Information

To take an etcd snapshot, you need the following pieces of information:

etcd endpoint (e.g., https://127.0.0.1:2379)
CA certificate file
Server certificate file
Server key file

You can obtain these details in two ways:

Method 1: From the etcd Static Pod Manifest

You can find these details in the etcd static pod manifest file located at /etc/kubernetes/manifests/etcd.yaml. You can access the files directly from the pod manifest.

Method 2: Describing the etcd Pod

Alternatively, you can describe the etcd pod running in the kube-system namespace using the following commands:

				
					kubectl get po -n kube-system
kubectl describe pod etcd-master-node -n kube-system

Replace etcd-master-node with the name of your etcd pod.

Taking the etcd Snapshot

With the required information at hand, you can take an etcd snapshot using the following command:

				
					ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert= \
  --cert= \
  --key= \
  snapshot save

Here’s what the command looks like with actual values:

				
					ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/backup/etcd.db

Execute this command to create a backup. Upon successful execution, you will receive a message confirming the snapshot’s location:
#Snapshot saved at /opt/backup/etcd.db

Verifying the Snapshot

To verify the snapshot, use the following command:

				
					ETCDCTL_API=3 etcdctl --write-out=table snapshot status /opt/backup/etcd.db

This will display information about the snapshot, including the hash, revision, total number of keys, and total size.

Here is an example output.

				
					+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| a3047677 |    51305 |       1088 |     6.8 MB |
+----------+----------+------------+------------+

Execute this command to create a backup.

Restoring etcd from a Snapshot

Now, let’s explore how to restore etcd from a snapshot backup. If you have the backup file located at /opt/backup/etcd.db, you can execute the following command:

				
					ETCDCTL_API=3 etcdctl snapshot restore /opt/backup/etcd.db

This command will restore etcd using the snapshot backup.

Using a Specific Data Directory

If you want to use a specific data directory for the restore, you can specify the location using the --data-dir flag as shown below:

				
					ETCDCTL_API=3 etcdctl --data-dir /opt/etcd snapshot restore /opt/backup/etcd.db

This is useful when you want to control where the etcd data will be stored during the restore process.

Conclusion

Regular backups and restores of etcd are essential for maintaining the stability and recoverability of your Kubernetes cluster. By following these steps, you can ensure that your etcd data is securely backed up and ready for recovery if needed.

Remember to schedule regular backups to keep your cluster’s data secure and ensure smooth operations.