Etcd Tutorial

Preface: This tutorial focuses exclusively on the Etcd v3 protocol. Throughout Etcd's history, two protocols have been employed: v2 and v3. However, v2 is considered outdated and not recommended for production environments. Furthermore, Etcd v2 and v3 have distinct data storage structures. As a result, you cannot use Etcd v2 to read data from Etcd v3 or use an Etcd v2 snapshot to restore data in Etcd v3. This document is tailored for Etcd instances containing v3 data.

How to Deploy an Etcd Cluster?

Etcd, a widely used distributed key-value store, plays a pivotal role in Kubernetes and is employed by numerous internet corporations worldwide. For both backend engineers and operations personnel, becoming proficient in handling errors and mastering this middleware is crucial. In this section, we will explore how to deploy an Etcd cluster using container tools such as "Docker" and "Docker Compose." Let's get started!

I have chosen an Ubuntu virtual machine as the working platform, specifically version 20.04.2 LTS. I assume you have already installed Docker and Docker Compose on your machine. If not, you can refer to the official Docker and Docker Compose documentation for installation instructions.

https://docs.docker.com/desktop/install/linux-install/

1. Etcd Directory Structure

 1$ tree
 2.
 3├── docker-compose.yaml
 4├── etcd
 5├── etcd.conf
 6└── etcd-ssl
 7    ├── ca.crt
 8    ├── ca.key
 9    ├── ca.srl
10    ├── client.crt
11    ├── client.csr
12    ├── client.key
13    ├── server.cnf
14    ├── server.crt
15    ├── server.csr
16    └── server.key

As you may know, we load our Etcd cluster using Docker Compose. Therefore, we need to create a docker-compose.yaml file to describe our Etcd cluster. Additionally, we require a customized etcd.conf file because the following section will involve configuring our Etcd cluster. The "etcd" and "etcd-ssl" directories are used to store Etcd data and certificates for client authentication.

2. Complete the docker-compose.yaml File

 1version: "3"
 2services:
 3  etcd:
 4    image: quay.io/coreos/etcd:v3.5.9
 5    network_mode: "host"
 6    container_name: etcd_container
 7    command: /usr/local/bin/etcd --config-file /etcd.conf
 8    volumes:
 9      - ./etcd:/var/lib/etcd
10      - ./etcd.conf:/etcd.conf:ro
11      - ./etcd-ssl:/etcd/etcd-ssl/:ro
12    restart: always
13    ulimits:
14      nofile:
15        soft: 1048576
16        hard: 1048576

1. We are using the official Etcd image from quay.io, specifically version v3.5.9.
2. The network_mode: "host" configuration allows the Etcd cluster to communicate with each other, enabling the use of the host's IP address for connecting to the Etcd cluster.
3. The restart: always setting ensures that the Etcd cluster remains operational, fully utilizing Etcd's high availability mechanisms.

3. Customize the etcd.conf File

 1#---for basic running---#
 2#[basic]
 3name: k8s-etcd01
 4data-dir: /var/lib/etcd
 5enable-v2: true
 6listen-client-urls: https://0.0.0.0:2379
 7listen-peer-urls: http://0.0.0.0:2380
 8
 9#[cluster]
10initial-advertise-peer-urls: http://10.0.24.14:2380
11advertise-client-urls: https://10.0.24.14:2379
12# for multi-cluster: k8s-etcd01=http://10.0.24.14:2380,k8s-etcd02=http://10.146.80.49:2380,k8s-etcd03=http://10.146.80.52:2380
13initial-cluster: k8s-etcd01=http://10.0.24.14:2380
14initial-cluster-state: existing
15initial-cluster-token: etcd-cluster
16
17#---for cluster election---#
18initial-election-tick-advance: true
19heartbeat-interval: 500
20election-timeout: 3000
21
22#---for performance tuner---#
23quota-backend-bytes: 8589934592
24auto-compaction-mode: 'periodic'
25auto-compaction-retention: '1'
26max-request-bytes: 10485760
27snapshot-count: 50000
28max-snapshots: 5
29max-wals: 5
30
31#---for client TLS encryption---#
32client-transport-security:
33  # Path to the client server TLS cert file.
34  cert-file: /etcd/etcd-ssl/server.crt
35  # Path to the client server TLS key file.
36  key-file: /etcd/etcd-ssl/server.key
37  # Enable client cert authentication.
38  # client-cert-auth: false
39  # Path to the client server TLS trusted CA cert file.
40  trusted-ca-file: /etcd/etcd-ssl/ca.crt
41  # Client TLS using generated certificates
42  # auto-tls: false

The table below provides explanations for each parameter in the etcd.conf file:

4. Generate Certificates for Client Authentication

Enabling client authentication for the Etcd cluster is essential for preventing unauthorized access. Therefore, we need to generate certificates for client authentication. For efficiency reasons, in private environments, peer authentication is not necessary, as it may impact performance.

Follow these commands to generate client certificates using OpenSSL:

 1# Generate ca.crt
 2$ openssl genrsa -out ca.key 4096
 3$ openssl req -new -x509 -key ca.key -out ca.crt -subj "/CN=etcd-ca"
 4
 5# Generate server.crt
 6$ vim server.cnf
 7[req]
 8distinguished_name = req_distinguished_name
 9req_extensions = v3_req
10[req_distinguished_name]
11[ v3_req ]
12subjectAltName = @alt_names
13[alt_names]
14DNS.1 = xxx.xxx.xxx.xxx
15IP.1 = xxx.xxx.xxx.xxx
16IP.2 = xxx.xxx.xxx.xxx
17IP.3 = xxx.xxx.xxx.xxx
18$ openssl genrsa -out server.key 2048
19$ openssl req -new -key server.key -out server.csr -subj "/CN=etcd-server" -config server.cnf
20$ openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -extensions v3_req -extfile server.cnf
21
22# Generate client.crt
23$ openssl genrsa -out client.key 2048
24$ openssl req -new -key client.key -out client.csr -subj "/CN=etcd-client"
25$ openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365

5. Optimize the Etcd Environment

To achieve optimal performance as an Etcd machine, it's essential to optimize both the Etcd configuration and the host's hardware environment. Here are some recommended parameters to adjust before starting your Etcd cluster:

1. File Descriptors:
   • Linux treats everything as a file, including regular files, directories, sockets, pipes, and more.
   • A file descriptor is a unique integer identifier used by a process to access open files or other resources. It acts as a handle that the process uses to read, write, or manipulate the resource.
   • There are three standard file descriptors: 0 (stdin), 1 (stdout), and 2 (stderr).
   • You can view a process's current open file count with lsof -p <pid> | wc -l.
   • How to view a process's open file limit? cat /proc/<pid>/limits | grep "Max open files"
   • How to view user's current file descriptor limit(per user)? ulimit -n
   • To change the file descriptor limit for your user, modify /etc/security/limits.conf or create a new file under /etc/security/limits.d/, with the following content:

     1* soft nofile 1048576
     2* hard nofile 1048576
     

   • You can also adjust file descriptor limits in systemd service files or Docker Compose YAML files, for example,add the following lines to the [Service] section:

     1LimitNOFILE=new_limit
     2LimitNPROC=new_limit
     

2. Max Open Files Count:
   • The "max open files count" or "file descriptor limit" is a system-wide limit on the number of file descriptors a process can have open simultaneously.
   • This limit is in place to prevent a single process from consuming excessive system resources by opening too many files or network connections.
   • It is especially important for server applications that need to handle many concurrent clients, as well as for system daemons and background processes.
   • You can view the current file descriptor limit for a process using the ulimit -Hn command.
   • To change the system-wide file descriptor limit, modify /etc/sysctl.conf, by adding the following line:

     1fs.file-max = new_limit
     

   Remember that increasing file descriptor limits should be done carefully and with consideration for system resources. Setting limits too high can potentially lead to resource exhaustion, so it's essential to strike a balance between accommodating your application's needs and maintaining system stability.
3. TCP Keepalive:
   • Enable TCP keepalive to prevent idle connections from timing out. Modify net.ipv4.tcp_keepalive_time, net.ipv4.tcp_keepalive_probes, and net.ipv4.tcp_keepalive_intvl, for example:

     1net.ipv4.tcp_keepalive_time = 60
     2net.ipv4.tcp_keepalive_probes = 3
     3net.ipv4.tcp_keepalive_intvl = 10
     

4. I/O Scheduler: Choose an I/O scheduler that best suits your workload. For example, for SSDs, you can use the deadline or mq-deadline scheduler, and for HDDs, the cfq scheduler. To check the current scheduler, use cat /sys/block/<device>/queue/scheduler, and to change it, use echo <scheduler> > /sys/block/<device>/queue/scheduler.
5. CPU: Etcd performance benefits from having a fast CPU, so choose a machine with multiple cores and high clock speeds.
6. Memory: Etcd's performance is directly affected by available memory. Ensure you have enough RAM to accommodate your etcd workload.
7. Storage: Etcd performance can be I/O bound, so consider using faster storage solutions like SSDs to reduce latency and increase overall performance.
8. Bandwidth: Ensure your network has sufficient bandwidth to handle the expected etcd traffic.
9. Latency: Minimize network latency between etcd nodes to enhance communication speed.

Please note that specific values for these parameters depend on your workload, hardware, and network environment. Perform load testing and benchmarking after making changes to assess their impact on Etcd's performance and stability. Additionally, ensure proper backups and conduct changes in a controlled test environment before applying them to production systems.

6. Bootstrapping the Etcd Cluster

 1$ docker-compose up
 2$ docker-compose ps
 3     Name                   Command               State   Ports
 4---------------------------------------------------------------
 5etcd_container   /usr/local/bin/etcd --conf ...   Up
 6# Check logs
 7$ docker-compose logs -f
 8...
 9
10# Check Etcd cluster status
11ETCDCTL_API=3 etcdctl --endpoints=https://10.0.24.14:2379 --cacert=./etcd-ssl/ca.crt --cert=./etcd-ssl/client.crt --key=./etcd-ssl/client.key endpoint status -w table
12+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
13|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
14+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
15| https://10.0.24.14:2379 | c8e267dee39d90f6 |   3.5.9 |   20 kB |      true |      false |         6 |         38 |                 38 |        |
16+-------------------------+
17
18------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Etcd Basic Commands

Here are some basic Etcd commands you can use:

Etcd Common Issues And How To Fix Them

1. Disaster Recovery

Taking a Snapshot of Etcd Data

You can take a snapshot of the etcd data while it's running using the following command:

1$ ETCDCTL_API=3 etcdctl --endpoints=https://10.0.24.14:2379 --cacert=./etcd-ssl/ca.crt --cert=./etcd-ssl/client.crt --key=./etcd-ssl/client.key snapshot save ./snapshot.db
2{"level":"info","ts":"2023-09-06T00:20:31.982857+0800","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"./snapshot.db.part"}
3{"level":"info","ts":"2023-09-06T00:20:31.992423+0800","logger":"client","caller":"v3@v3.5.9/maintenance.go:212","msg":"opened snapshot stream; downloading"}
4{"level":"info","ts":"2023-09-06T00:20:31.992476+0800","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://10.0.24.14:2379"}
5{"level":"info","ts":"2023-09-06T00:20:32.003568+0800","logger":"client","caller":"v3@v3.5.9/maintenance.go:220","msg":"completed snapshot read; closing"}
6{"level":"info","ts":"2023-09-06T00:20:32.008874+0800","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://10.0.24.14:2379","size":"20 kB","took":"now"}
7{"level":"info","ts":"2023-09-06T00:20:32.008952+0800","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"./snapshot.db"}
8Snapshot saved at ./snapshot.db

The command will produce output indicating the snapshot has been saved.

Restoring Etcd Data from a Snapshot

To restore etcd data from a snapshot, ensure that the following parameters in your restore command match the ones in your etcd.conf file, such as initial-cluster-token:

1$ ETCDCTL_API=3 etcdctl --endpoints=https://10.0.24.14:2379 --cacert=./etcd-ssl/ca.crt --cert=./etcd-ssl/client.crt --key=./etcd-ssl/client.key snapshot restore --skip-hash-check ./snapshot.db --initial-cluster=k8s-etcd01=http://10.0.24.14:2380 --initial-cluster-token=etcd-cluster --initial-advertise-peer-urls=http://10.0.24.14:2380 --name k8s-etcd01  --data-dir=./data
2Deprecated: Use `etcdutl snapshot restore` instead.
3
42023-09-06T00:25:53+08:00	info	snapshot/v3_snapshot.go:248	restoring snapshot	{"path": "./snapshot.db", "wal-dir": "data/member/wal", "data-dir": "./data", "snap-dir": "data/member/snap", "stack": "go.etcd.io/etcd/etcdutl/v3/snapshot.(*v3Manager).Restore\n\tgo.etcd.io/etcd/etcdutl/v3@v3.5.9/snapshot/v3_snapshot.go:254\ngo.etcd.io/etcd/etcdutl/v3/etcdutl.SnapshotRestoreCommandFunc\n\tgo.etcd.io/etcd/etcdutl/v3@v3.5.9/etcdutl/snapshot_command.go:147\ngo.etcd.io/etcd/etcdctl/v3/ctlv3/command.snapshotRestoreCommandFunc\n\tgo.etcd.io/etcd/etcdctl/v3/ctlv3/command/snapshot_command.go:129\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.1.3/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.1.3/command.go:897\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.Start\n\tgo.etcd.io/etcd/etcdctl/v3/ctlv3/ctl.go:107\ngo.etcd.io/etcd/etcdctl/v3/ctlv3.MustStart\n\tgo.etcd.io/etcd/etcdctl/v3/ctlv3/ctl.go:111\nmain.main\n\tgo.etcd.io/etcd/etcdctl/v3/main.go:59\nruntime.main\n\truntime/proc.go:250"}
52023-09-06T00:25:53+08:00	info	membership/store.go:141	Trimming membership information from the backend...
62023-09-06T00:25:53+08:00	info	membership/cluster.go:421	added member	{"cluster-id": "7cadd2cbf250eee7", "local-member-id": "0", "added-peer-id": "c8e267dee39d90f6", "added-peer-peer-urls": ["http://10.0.24.14:2380"]}
72023-09-06T00:25:53+08:00	info	snapshot/v3_snapshot.go:269	restored snapshot	{"path": "./snapshot.db", "wal-dir": "data/member/wal", "data-dir": "./data", "snap-dir": "data/member/snap"}

Alternatively, you can restore specific keys or directories from a snapshot:

1$ etcdctl snapshot restore <snapshot-file> --data-dir <data-directory> --restore-from-key <key>
2# or
3$ etcdctl snapshot restore /path/to/snapshot.db --data-dir /var/lib/etcd --restore-from-key /mydir/mykey

After adjusting etcd's data directory in the docker-compose.yaml file, start the restored etcd cluster:

1$ docker-compose up -d
2$ ETCDCTL_API=3 etcdctl --endpoints=https://10.0.24.14:2379 --cacert=./etcd-ssl/ca.crt --cert=./etcd-ssl/client.crt --key=./etcd-ssl/client.key endpoint status -w table
3+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
4|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
5+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
6| https://10.0.24.14:2379 | c8e267dee39d90f6 |   3.5.9 |   20 kB |      true |      false |         2 |          4 |                  4 |        |
7+-------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

2. Issue: "etcdserver: mvcc: database space exceeded"

Related links: https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/

If you encounter the "etcdserver: mvcc: database space exceeded" issue, you can follow these steps to resolve it:

 1$ ETCDCTL_API=3 etcdctl --endpoints localhost:2379 alarm list/disarm
 2$ ETCDCTL_API=3 etcdctl --endpoints localhost:2379 endpoint status -w table
 3$ ETCDCTL_API=3 etcdctl --endpoints localhost:2379 compact
 4$ ETCDCTL_API=3 etcdctl --endpoints localhost:2379 defrag
 5
 6# Check the etcd revision to identify keys with the most revisions
 7$ ETCDCTL_API=3 etcdctl --endpoints localhost:2379 endpoint status -w json
 8$ ETCDCTL_API=3 etcdctl --endpoints localhost:2379 get "" --prefix --keys-only | grep -v ^$ | awk -F '/'  '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr
 9
10# Modify the docker-compose.yaml file and restart the etcd cluster
11$ vim docker-compose.yaml

In the docker-compose.yaml file, ensure that you specify "--quota-backend-bytes 8589934592" and any other settings for your etcd cluster. Then restart the cluster:

1$ docker-compose up -d

Let's delve into the etcd's revision mechanism:

In etcd, the revision represents a continually increasing integer value assigned to each modification made to the etcd key-value store. It serves as a global version number that tracks the historical changes within the etcd cluster. The revision number is incremented each time there is a modification, such as creating, updating, or deleting a key-value pair in etcd. Each modification operation increments the revision by one.

The etcd revision number carries several crucial implications:

1. Consistency and Order: The revision number ensures the consistency and order of operations within the etcd cluster. It guarantees that operations applied at a higher revision number have occurred after operations at lower revision numbers.
2. Read Operations: When conducting a read operation on etcd, you can specify a revision number to retrieve the state of the key-value store at a specific point in time. This enables you to access a consistent snapshot of the data at a particular revision.
3. Change Detection: By monitoring the revision number, you can identify changes or updates to the etcd key-value store. You can periodically fetch the latest revision number and compare it to a previously recorded value to ascertain if any modifications have taken place.
4. Database Compaction: Etcd offers a compaction mechanism that allows you to remove older or redundant revisions from the database. By compacting the database, you can reclaim storage space and enhance performance. The revision number plays a pivotal role in the compaction process.

3. Adding a New Etcd Cluster Member

To add a new etcd cluster member, follow these steps:

• Configure the new node's etcd.conf file with the following essential parameters:

  1. name: A unique name for the new node.
  2. data-dir: The directory where etcd will store its data.
  3. initial-advertise-peer-urls: The URL at which the new node advertises itself to other members.
  4. listen-peer-urls: The URLs on which the new node listens for communication with other etcd nodes.
  5. listen-client-urls: The URLs on which the new node listens for client requests.
  6. advertise-client-urls: The URLs that clients should use to connect to the new node.
  7. initial-cluster: A list of the initial etcd cluster members and their associated initial-advertise-peer-urls, which should match those in other nodes' etcd.conf.
  8. initial-cluster-state: Set it to "existing" if you're adding a new member to an existing cluster.

• Use etcdctl to add the new member:

1$ ETCDCTL_API=3 etcdctl --endpoints=https://ETCDCLUSTER --cacert=./etcd-ssl/ca.crt --cert=./etcd-ssl/client.crt --key=./etcd-ssl/client.key member add ETCDNAME --peer-urls=https://NEW-NODE:2380

• Start the new node and check its status:

1$ docker-compose up -d
2$ ETCDCTL_API=3 etcdctl --endpoints=https://ETCDCLUSTER --cacert=./etcd-ssl/ca.crt --cert=./etcd-ssl/client.crt --key=./etcd-ssl/client.key endpoint status -w table

Interacting With Kubernetes's Etcd

Installing etcdhelper

To interact with Kubernetes's etcd, you can install etcdhelper using the following steps:

1# Clone the etcdhelper repository
2$ git clone https://github.com/openshift/origin.git
3
4# Set up an alias for etcdhelper
5$ vim ~/.zshrc
6alias ectl='ETCDCTL_API=3 etcdhelper -endpoint=172.31.27.61:2379 -cacert /home/k8s/code/etcd-prod-27-61/cert/ca.pem -cert /home/k8s/code/etcd-prod-27-61/cert/etcd.pem -key /home/k8s/code/etcd-prod-27-61/cert/etcd.key'

Accessing Kubernetes's Etcd

You can use etcdhelper to access Kubernetes's etcd and retrieve data:

1$ ectl get /registry/secrets/test/harbor-kubernetes && echo

Please note that Kubernetes stores data in the following order: /registry/{resource_name}/{namespace}/{resource_instance}. Secrets' content is base64 encoded before being stored in etcd, so you need to decode it to retrieve the actual content.

Additional Information

To explore etcd's API paths, you can use the provided script:

1#!/bin/bash
2# Get Kubernetes keys from etcd
3export ETCDCTL_API=3
4keys=`ETCDCTL_API=3 etcdctl --endpoints=172.31.27.61:2379 --cacert /etc/ssl/etcd/ssl
5
6/ca.pem --cert /etc/ssl/etcd/ssl/member-ip-172-31-27-61.cn-north-1.compute.internal.pem --key /etc/ssl/etcd/ssl/member-ip-172-31-27-61.cn-north-1.compute.internal-key.pem get  /registry --prefix -w json|python -m json.tool|grep key|cut -d ":" -f2|tr -d '"'|tr -d ","`
7for x in $keys;do
8  echo $x|base64 -d|sort
9done

This script helps you retrieve and decode Kubernetes-related keys stored in etcd.

Please keep in mind that the most reliable and accurate source for tutorials is the official documentation (https://etcd.io/docs/v3.5/). If you have any questions, please consult the official documentation first. Wishing you the best of luck!