Etcd调优和性能测试

2021-01-19 约 7922 字预计阅读 16 分钟本文总阅读量次

针对etcd的性能指标：延迟(latency)和吞吐量(throughput)，进行Etcd参数调优和对比测试

说明

etcd 提供稳定的，持续的高性能。两个定义性能的因素：延迟(latency)和吞吐量(throughput)。延迟是完成操作的时间。吞吐量是在某个时间期间之内完成操作的总数量。当 etcd 接收并发客户端请求时，通常平均延迟随着总体吞吐量增加而增加。在通常的云环境，比如 Google Compute Engine (GCE) 标准的 n-4 或者 AWS 上相当的机器类型，一个三成员 etcd 集群在轻负载下可以在低于1毫秒内完成一个请求，并在重负载下可以每秒完成超过 30000 个请求。

etcd 使用 Raft 一致性算法来在成员之间复制请求并达成一致。一致性性能，特别是提交延迟，受限于两个物理约束：网络IO延迟和磁盘IO延迟。完成一个etcd请求的最小时间是成员之间的网络往返时延(Round Trip Time / RTT)，加需要提交数据到持久化存储的 fdatasync 时间。在一个数据中心内的 RTT 可能有数百毫秒。在美国典型的 RTT 是大概 50ms, 而在大陆之间可以慢到400ms. 旋转硬盘(注：指传统机械硬盘)的典型 fdatasync 延迟是大概 10ms。对于 SSD 硬盘, 延迟通常低于 1ms. 为了提高吞吐量, etcd 将多个请求打包在一起并提交给 Raft。这个批量策略让 etcd 在重负载时获得高吞吐量.

有其他子系统影响到 etcd 的整体性能。每个序列化的 etcd 请求必须通过 etcd 的 boltdb支持的(boltdb-backed) MVCC 存储引擎,它通常需要10微秒来完成。etcd 定期递增快照它最近实施的请求，将他们和之前在磁盘上的快照合并。这个过程可能导致延迟尖峰(latency spike)。虽然在SSD上这通常不是问题，在HDD上它可能加倍可观察到的延迟。而且，进行中的压缩可以影响 etcd 的性能。幸运的是，压缩通常无足轻重，因为压缩是错开的，因此它不和常规请求竞争资源。RPC 系统，gRPC，为 etcd 提供定义良好，可扩展的 API，但是它也引入了额外的延迟，尤其是本地读取。

etcd 安装

二进制方式安装

参考版本地址这里选择ETCD_VER=v3.4.13版本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

ETCD_VER=v3.4.13

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
# choose GITHUB_URL
DOWNLOAD_URL=${GITHUB_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

/tmp/etcd-download-test/etcd --version
/tmp/etcd-download-test/etcdctl version

etcd 启动

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# start a local etcd server
/tmp/etcd-download-test/etcd

## 指定etcd name
/tmp/etcd-download-test/etcd --name=etcdtest 

## 指定data-dir
/tmp/etcd-download-test/etcd --name=etcdtest --data-dir=/var/lib/etcd 

## 指定 params
/tmp/etcd-download-test/etcd --name=etcdtest --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000 --auto-compaction-retention=1

etcd 键值测试

1
2
3
# write,read to etcd
/tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 put foo bar
/tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 get foo

etcd 参数优化

快照

etcd_snapshot_count 5000

数据快照触发数量，etcd处理指定的次数的事务提交后，生产数据快照

历史数据压缩频率

etcd_compaction_retention 1

由于ETCD数据存储多版本数据，随着写入的主键增加历史版本需要定时清理，　默认的历史数据是不会清理的，数据达到2G就不能写入，必须要清理压缩历史数据才能继续写入；

所以根据业务需求，在上生产环境之前就提前确定，历史数据多长时间压缩一次；　我们的生产环境现在升级后是默认一小时压缩一次数据。这样可以极大的保证集群稳定，减少内存和磁盘占用

时间参数

etcd_heartbeat_interval 200 etcd_election_timeout 2000

客户端连接后的心跳间隔（毫秒）集群选举的超时时间（毫秒）

磁盘IO优先级，在全部etcd节点执行

ionice -c2 -n0 -p pgrep etcd

etcd配置，环境变量方式

vi /etc/etcd.env

1
2
3
4
5
6
vi /etc/etcd.env

ETCD_SNAPSHOT_COUNT=5000
ETCD_HEARTBEAT_INTERVAL=200
ETCD_ELECTION_TIMEOUT=2000
ETCD_AUTO_COMPACTION_RETENTION=1

etcd配置，命令行参数方式

1
etcd --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000 --auto-compaction-retention=1

benchmark安装

etcd/tools/benchmark 是etcd官方benchmark测试工具

安装命令如下，

get方式

1
2
3
4
# set myproject go env
export GOPATH=/home/wangb/go_projects
go get go.etcd.io/etcd/tools/benchmark
ls $GOPATH/bin

示例：

1
2
3
4
5

$ go get go.etcd.io/etcd/tools/benchmark
# GOPATH should be set
$ ls $GOPATH/bin
benchmark

编译方式

如果上面get方式不能成功，则下载etcd源码，进行编译

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# 使用go mod

# 进入项目根目录，如go_projects/src/etcd-3.4.13

export GO111MODULE=on
# export GO111MODULE=off
# go env -w GOPROXY=https://goproxy.cn,direct

#进入项目目录
#go mod init godev

# 下载依赖
go mod tidy

# 生成项目vendor
go mod vendor

1
2
3
# etcd-3.4.13/tools/benchmark
cd tools/benchmark
go build -o benchmark

示例：

1
2
3
4
5
6
7
8
9
(base) [root@yuyuan211 /home/wangb/go_projects/src/etcd-3.4.13/tools/benchmark]# go build -o benchmark
(base) [root@yuyuan211 /home/wangb/go_projects/src/etcd-3.4.13/tools/benchmark]# ll
total 19080
-rwxr-xr-x. 1 root root 19525385 Jan 18 11:09 benchmark
drwxr-xr-x. 2 root root      278 Aug 25 03:11 cmd
-rw-r--r--. 1 root root      675 Aug 25 03:11 doc.go
-rw-r--r--. 1 root root      784 Aug 25 03:11 main.go
-rw-r--r--. 1 root root      284 Aug 25 03:11 README.md

benchmark指标

性能指标说明：

延时
吞吐量

title	description
Performance	Understanding performance: latency & throughput

benchmark测试

启动etcd

1
/tmp/etcd-download-test/etcd --name=etcdtest --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000 --auto-compaction-retention=1

启动打印

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114

(base) [root@yuyuan211 /home/wangb/etcd-test]# /tmp/etcd-download-test/etcd --name=etcdtest --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000 --auto-compaction-retention=1
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-01-19 14:34:35.907827 I | etcdmain: etcd Version: 3.4.13
2021-01-19 14:34:35.908025 I | etcdmain: Git SHA: ae9734ed2
2021-01-19 14:34:35.908089 I | etcdmain: Go Version: go1.12.17
2021-01-19 14:34:35.908116 I | etcdmain: Go OS/Arch: linux/amd64
2021-01-19 14:34:35.908144 I | etcdmain: setting maximum number of CPUs to 32, total number of available CPUs is 32
2021-01-19 14:34:35.908186 W | etcdmain: no data-dir provided, using default data-dir ./etcdtest.etcd
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-01-19 14:34:35.912732 I | embed: name = etcdtest
2021-01-19 14:34:35.912790 I | embed: data dir = etcdtest.etcd
2021-01-19 14:34:35.912833 I | embed: member dir = etcdtest.etcd/member
2021-01-19 14:34:35.912854 I | embed: heartbeat = 200ms
2021-01-19 14:34:35.912873 I | embed: election = 2000ms
2021-01-19 14:34:35.912891 I | embed: snapshot count = 5000
2021-01-19 14:34:35.912944 I | embed: advertise client URLs = http://localhost:2379
2021-01-19 14:34:35.925333 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
raft2021/01/19 14:34:35 INFO: 8e9e05c52164694d switched to configuration voters=()
raft2021/01/19 14:34:35 INFO: 8e9e05c52164694d became follower at term 0
raft2021/01/19 14:34:35 INFO: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
raft2021/01/19 14:34:35 INFO: 8e9e05c52164694d became follower at term 1
raft2021/01/19 14:34:35 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-01-19 14:34:35.928869 W | auth: simple token is not cryptographically signed
2021-01-19 14:34:35.933119 I | etcdserver: starting server... [version: 3.4.13, cluster version: to_be_decided]
2021-01-19 14:34:35.933413 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
raft2021/01/19 14:34:35 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2021-01-19 14:34:35.935531 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2021-01-19 14:34:35.939938 I | embed: listening for peers on 127.0.0.1:2380
raft2021/01/19 14:34:37 INFO: 8e9e05c52164694d is starting a new election at term 1
raft2021/01/19 14:34:37 INFO: 8e9e05c52164694d became candidate at term 2
raft2021/01/19 14:34:37 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
raft2021/01/19 14:34:37 INFO: 8e9e05c52164694d became leader at term 2
raft2021/01/19 14:34:37 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
2021-01-19 14:34:37.328580 I | etcdserver: setting up the initial cluster version to 3.4
2021-01-19 14:34:37.330234 N | etcdserver/membership: set the initial cluster version to 3.4
2021-01-19 14:34:37.330360 I | embed: ready to serve client requests
2021-01-19 14:34:37.330537 I | etcdserver: published {Name:etcdtest ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2021-01-19 14:34:37.330575 I | etcdserver/api: enabled capabilities for version 3.4
2021-01-19 14:34:37.332810 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2021-01-19 14:36:58.994204 I | etcdserver: start to snapshot (applied: 5001, lastsnap: 0)
2021-01-19 14:36:58.999539 I | etcdserver: saved snapshot at index 5001
2021-01-19 14:36:59.000747 I | etcdserver: compacted raft log at 1
2021-01-19 14:37:02.580676 I | etcdserver: start to snapshot (applied: 10002, lastsnap: 5001)
2021-01-19 14:37:02.585886 I | etcdserver: saved snapshot at index 10002
2021-01-19 14:37:02.587930 I | etcdserver: compacted raft log at 5002
2021-01-19 14:37:02.879022 I | etcdserver: start to snapshot (applied: 15005, lastsnap: 10002)
2021-01-19 14:37:02.889233 I | etcdserver: saved snapshot at index 15005
2021-01-19 14:37:02.893700 I | etcdserver: compacted raft log at 10005
2021-01-19 14:37:03.030899 I | etcdserver: start to snapshot (applied: 20042, lastsnap: 15005)
2021-01-19 14:37:03.047952 I | etcdserver: saved snapshot at index 20042
2021-01-19 14:37:03.051905 I | etcdserver: compacted raft log at 15042
2021-01-19 14:37:03.208428 I | etcdserver: start to snapshot (applied: 25335, lastsnap: 20042)
2021-01-19 14:37:03.220643 I | etcdserver: saved snapshot at index 25335
2021-01-19 14:37:03.226455 I | etcdserver: compacted raft log at 20335
2021-01-19 14:37:03.381115 I | etcdserver: start to snapshot (applied: 30503, lastsnap: 25335)
2021-01-19 14:37:03.399205 I | etcdserver: saved snapshot at index 30503
2021-01-19 14:37:03.405633 I | etcdserver: compacted raft log at 25503
2021-01-19 14:37:03.552010 I | etcdserver: start to snapshot (applied: 35970, lastsnap: 30503)
2021-01-19 14:37:03.573982 I | etcdserver: saved snapshot at index 35970
2021-01-19 14:37:03.574904 I | etcdserver: compacted raft log at 30970
2021-01-19 14:37:03.749731 I | etcdserver: start to snapshot (applied: 41678, lastsnap: 35970)
2021-01-19 14:37:03.778794 I | etcdserver: saved snapshot at index 41678
2021-01-19 14:37:03.793623 I | etcdserver: compacted raft log at 36678
2021-01-19 14:37:03.935394 I | etcdserver: start to snapshot (applied: 46839, lastsnap: 41678)
2021-01-19 14:37:03.964458 I | etcdserver: saved snapshot at index 46839
2021-01-19 14:37:03.977165 I | etcdserver: compacted raft log at 41839
2021-01-19 14:37:04.182597 I | etcdserver: start to snapshot (applied: 51885, lastsnap: 46839)
2021-01-19 14:37:04.206669 I | etcdserver: saved snapshot at index 51885
2021-01-19 14:37:04.207931 I | etcdserver: compacted raft log at 46885
2021-01-19 14:37:04.387361 I | etcdserver: start to snapshot (applied: 57144, lastsnap: 51885)
2021-01-19 14:37:04.416557 I | etcdserver: saved snapshot at index 57144
2021-01-19 14:37:04.435562 I | etcdserver: compacted raft log at 52144
2021-01-19 14:37:04.598093 I | etcdserver: start to snapshot (applied: 62521, lastsnap: 57144)
2021-01-19 14:37:04.626258 I | etcdserver: saved snapshot at index 62521
2021-01-19 14:37:04.641498 I | etcdserver: compacted raft log at 57521
2021-01-19 14:37:04.773653 I | etcdserver: start to snapshot (applied: 67553, lastsnap: 62521)
2021-01-19 14:37:04.788911 I | etcdserver: saved snapshot at index 67553
2021-01-19 14:37:04.791871 I | etcdserver: compacted raft log at 62553
2021-01-19 14:37:04.959274 I | etcdserver: start to snapshot (applied: 72572, lastsnap: 67553)
2021-01-19 14:37:04.972930 I | etcdserver: saved snapshot at index 72572
2021-01-19 14:37:04.979931 I | etcdserver: compacted raft log at 67572
2021-01-19 14:37:05.130776 I | etcdserver: start to snapshot (applied: 77731, lastsnap: 72572)
2021-01-19 14:37:05.163934 I | etcdserver: saved snapshot at index 77731
2021-01-19 14:37:05.164320 I | etcdserver: compacted raft log at 72731
2021-01-19 14:37:05.341994 I | etcdserver: start to snapshot (applied: 82808, lastsnap: 77731)
2021-01-19 14:37:05.362133 I | etcdserver: saved snapshot at index 82808
2021-01-19 14:37:05.362556 I | etcdserver: compacted raft log at 77808
2021-01-19 14:37:05.534155 I | etcdserver: start to snapshot (applied: 87972, lastsnap: 82808)
2021-01-19 14:37:05.552320 I | etcdserver: saved snapshot at index 87972
2021-01-19 14:37:05.560299 I | etcdserver: compacted raft log at 82972
2021-01-19 14:37:05.708062 I | etcdserver: start to snapshot (applied: 93127, lastsnap: 87972)
2021-01-19 14:37:05.733762 I | etcdserver: saved snapshot at index 93127
2021-01-19 14:37:05.744657 I | etcdserver: compacted raft log at 88127
2021-01-19 14:37:05.915552 I | etcdserver: start to snapshot (applied: 98166, lastsnap: 93127)
2021-01-19 14:37:05.936166 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000001389.snap successfully
2021-01-19 14:37:05.936297 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000002712.snap successfully
2021-01-19 14:37:05.936417 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000003a9d.snap successfully
2021-01-19 14:37:05.936540 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000004e4a.snap successfully
2021-01-19 14:37:05.936641 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-00000000000062f7.snap successfully
2021-01-19 14:37:05.936752 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000007727.snap successfully
2021-01-19 14:37:05.936873 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000008c82.snap successfully
2021-01-19 14:37:05.936984 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-000000000000a2ce.snap successfully
2021-01-19 14:37:05.937092 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-000000000000b6f7.snap successfully
2021-01-19 14:37:05.937197 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-000000000000caad.snap successfully
2021-01-19 14:37:05.937311 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-000000000000df38.snap successfully
2021-01-19 14:37:05.937419 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-000000000000f439.snap successfully
2021-01-19 14:37:05.937532 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-00000000000107e1.snap successfully
2021-01-19 14:37:05.937663 I | pkg/fileutil: purged file etcdtest.etcd/member/snap/0000000000000002-0000000000011b7c.snap successfully
2021-01-19 14:37:05.944586 I | etcdserver: saved snapshot at index 98166

参数说明

clients：Number of clients：客户端数量
conns：Number of connections，http连接数量，多个客户端可复用1个连接
total ：Total number of put requests，requests请求数量，即所有客户端的请求总数量，默认值10000

write测试

With this configuration, etcd can approximately write:

Number of keys	Key size in bytes	Value size in bytes	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	Average server RSS
10,000	8	256	1	1	leader only	1359	0.7ms	24 MB
100,000	8	256	100	1000	leader only	27507	36ms	75MB
100,000	8	256	100	1000	all members	27206	36.3ms	89MB

说明：这里只有一个etcd节点，所以表格第3行的集群raft测试结果参考意义不大。

Sample commands are:

1
2
3
4
5
6
7
HOST_1=http://127.0.0.1:2379
HOST_2=http://127.0.0.1:2379
HOST_3=http://127.0.0.1:2379
# include benchmark bin path
current=`pwd`
export PATH=$PATH:$current

1
2
3
4
5
6
7
8
9
# write to leader
benchmark --endpoints=${HOST_1} --target-leader --conns=1 --clients=1 \
    put --key-size=8 --sequential-keys --total=10000 --val-size=256
benchmark --endpoints=${HOST_1} --target-leader  --conns=100 --clients=1000 \
    put --key-size=8 --sequential-keys --total=100000 --val-size=256

# write to all members
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
    put --key-size=8 --sequential-keys --total=100000 --val-size=256

read测试

Linearizable read requests go through a quorum of cluster members for consensus to fetch the most recent data. Serializable read requests are cheaper than linearizable reads since they are served by any single etcd member, instead of a quorum of members, in exchange for possibly serving stale data. etcd can read:

Number of requests	Key size in bytes	Value size in bytes	Number of connections	Number of clients	Consistency	Average read QPS	Average latency per request
10,000	8	256	1	1	Linearizable	1110	0.9ms
10,000	8	256	1	1	Serializable	1251	0.8ms
100,000	8	256	100	1000	Linearizable	9532	0.1044s
100,000	8	256	100	1000	Serializable	11354	0.0875s

说明由于测试etcd为单节点，所以Linearizable和Serializable特性测试结果差别不大，参考意义不大。

Sample commands are:

1
2
3
4
5
6
7
HOST_1=http://127.0.0.1:2379
HOST_2=http://127.0.0.1:2379
HOST_3=http://127.0.0.1:2379

current=`pwd`
export PATH=$PATH:$current

read前，先write一个测试key

1
2
3
4
YOUR_KEY=foo
/tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 put $YOUR_KEY bar

/tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 get $YOUR_KEY

测试命令

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Single connection read requests
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=1 --clients=1 \
    range $YOUR_KEY --consistency=l --total=10000
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=1 --clients=1 \
    range $YOUR_KEY --consistency=s --total=10000

# Many concurrent read requests
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
    range $YOUR_KEY --consistency=l --total=100000
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
    range $YOUR_KEY --consistency=s --total=100000

read-only 时间参数设置优化

read测试时的etcd打印信息

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

2021-01-18 14:54:58.315985 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (128.237385ms) to execute
2021-01-18 14:54:58.316748 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (136.434995ms) to execute
2021-01-18 14:54:58.317021 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (125.138823ms) to execute
2021-01-18 14:54:58.327063 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (113.659252ms) to execute
2021-01-18 14:54:58.327171 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (140.480071ms) to execute
2021-01-18 14:54:58.328320 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (138.142424ms) to execute
2021-01-18 14:54:58.329457 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (136.980041ms) to execute
2021-01-18 14:54:58.330026 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (139.674614ms) to execute
2021-01-18 14:54:58.330674 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (137.950461ms) to execute
2021-01-18 14:54:58.330710 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (151.589201ms) to execute
2021-01-18 14:54:58.338877 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (149.623303ms) to execute
2021-01-18 14:54:58.339042 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (148.882374ms) to execute

上面的read测试时的etcd打印信息，会一直输出告警打印信息（包括输出到系统日志中），因为read-only range request > 100ms，导致性能降低

代码中默认值设置为 100ms

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
//  v3.3 -> v3.4.14
const (
    warnApplyDuration = 100 * time.Millisecond
)


//  v3.4-master
const (
DefaultWarningApplyDuration  = 100 * time.Millisecond
)

在etcd-v3.4最新版本(master)添加了参数优化设置，而v3.4.14以前，包括v3.4.14和etcd-v3.3没有办法消除该告警信息打印

同时etcd-3.4版本的模块目录有所调整，所以需要跟踪etcd版本

跟踪etcd版本，是否新增了配置变量ExperimentalWarningApplyDuration，而最新版本不再使用变量WarnApplyDuration，改为WarningApplyDuration

WarningApplyDuration 修改参考

1
2
3
4
srvcfg := etcdserver.ServerConfig{
    WarningApplyDuration:        cfg.ExperimentalWarningApplyDuration,
}

benchmark测试条件

We encourage running the benchmark test when setting up an etcd cluster for the first time in a new environment to ensure the cluster achieves adequate performance; cluster latency and throughput can be sensitive to minor environment differences.

磁盘IO 参数调优

Linux 中 etcd 的磁盘优先级可以使用 ionice 配置：

1
2
3
4
5
6
7
-c  class  指定调度类型，0代表none，1代表real time,2代表best effort, 3代表idle

-n  classdata   指定优先级  real time和best effor可以使用0-7

-p  pid    查看或改变已经运行的进程的调度类型和优先级。

-t  忽略设置指定优先级的错误信息

查询命令

1
ionice  -p `pgrep etcd`

原有配置

1
2
3
(base) [root@yuyuan211 ~]# ionice  -p `pgrep etcd`
unknown: prio 4

设置命令

Best Effort策略，优先级为0（优先级最高）

1
2
3
4
# best effort, highest priority
ionice -c2 -n0 -p `pgrep etcd`

ionice  -p `pgrep etcd`

优化配置

1
2
3
4
5
6
(base) [root@yuyuan211 ~]# ionice -c2 -n0 -p `pgrep etcd`
(base) [root@yuyuan211 ~]#
(base) [root@yuyuan211 ~]# ionice  -p `pgrep etcd`
best-effort: prio 0
(base) [root@yuyuan211 ~]#

优化后测试

说明

为了在相同环境下对比测试，把之前测试数据文件etcdtest.etcd全部删除
read测试时，测试键值key，可以不预置，这样read测试为极限值

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
(base) [root@yuyuan211 /home/wangb/etcd-test]# ll
total 19068
-rwxr-xr-x. 1 root root 19525385 Jan 18 11:36 benchmark
drwxr-xr-x. 3 root root      151 Jan 18 11:37 etcd-download-test
drwx------. 3 root root       28 Jan 18 13:46 etcdtest.etcd
(base) [root@yuyuan211 /home/wangb/etcd-test]#
(base) [root@yuyuan211 /home/wangb/etcd-test]#
(base) [root@yuyuan211 /home/wangb/etcd-test]#
(base) [root@yuyuan211 /home/wangb/etcd-test]# rm -rf etcdtest.etcd
(base) [root@yuyuan211 /home/wangb/etcd-test]#

write测试

Sample commands are:

1
2
3
4
5
6
7
HOST_1=http://127.0.0.1:2379
HOST_2=http://127.0.0.1:2379
HOST_3=http://127.0.0.1:2379
# include benchmark bin path
current=`pwd`
export PATH=$PATH:$current

1
2
3
4
5
6
7
8
9
# write to leader
benchmark --endpoints=${HOST_1} --target-leader --conns=1 --clients=1 \
    put --key-size=8 --sequential-keys --total=10000 --val-size=256
benchmark --endpoints=${HOST_1} --target-leader  --conns=100 --clients=1000 \
    put --key-size=8 --sequential-keys --total=100000 --val-size=256

# write to all members
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
    put --key-size=8 --sequential-keys --total=100000 --val-size=256

With this configuration, etcd can approximately write:

Number of keys	Key size in bytes	Value size in bytes	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request
10,000	8	256	1	1	leader only	1357	0.7ms
100,000	8	256	100	1000	leader only	28232	35.1ms
100,000	8	256	100	1000	all members	27620	35.9ms

read测试

测试命令

1
2
3
4
5
6
7
HOST_1=http://127.0.0.1:2379
HOST_2=http://127.0.0.1:2379
HOST_3=http://127.0.0.1:2379
# include benchmark bin path
current=`pwd`
export PATH=$PATH:$current

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Single connection read requests
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=1 --clients=1 \
    range $YOUR_KEY --consistency=l --total=10000
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=1 --clients=1 \
    range $YOUR_KEY --consistency=s --total=10000

# Many concurrent read requests
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
    range $YOUR_KEY --consistency=l --total=100000
benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
    range $YOUR_KEY --consistency=s --total=100000

Number of requests	Key size in bytes	Value size in bytes	Number of connections	Number of clients	Consistency	Average read QPS	Average latency per request
10,000	8	256	1	1	Linearizable	1272	0.8ms
10,000	8	256	1	1	Serializable	1432	0.7ms
100,000	8	256	100	1000	Linearizable	13108	0.0758s
100,000	8	256	100	1000	Serializable	16088	0.0617s

对比结果

write对比

Number of keys	Key size in bytes	Value size in bytes	Number of connections	Number of clients	Target etcd server	Average write QPS	Average latency per request	Tunning
10,000	8	256	1	1	leader only	1359	0.7ms	false
10,000	8	256	1	1	leader only	1382	0.7ms	true
100,000	8	256	100	1000	leader only	27507	36ms	false
100,000	8	256	100	1000	leader only	28381	34.8ms	true
100,000	8	256	100	1000	all members	27206	36.3ms	false
100,000	8	256	100	1000	all members	27855	35.6ms	true

read对比

Number of requests	Key size in bytes	Value size in bytes	Number of connections	Number of clients	Consistency	Average read QPS	Average latency per request	Tunning
10,000	8	256	1	1	Linearizable	1110	0.9ms	false
10,000	8	256	1	1	Linearizable	1272	0.8ms	true
10,000	8	256	1	1	Serializable	1251	0.8ms	false
10,000	8	256	1	1	Serializable	1432	0.7ms	true
100,000	8	256	100	1000	Linearizable	9532	0.1044s	false
100,000	8	256	100	1000	Linearizable	13108	0.0758s	true
100,000	8	256	100	1000	Serializable	11354	0.0875s	false
100,000	8	256	100	1000	Serializable	16088	0.0617s	true

结论

磁盘IO参数可以优化etcd性能，write和read接口指标改善
快照和数据压缩参数，可以减少etcd的内存和磁盘占用量

附录

Linearizability

Linearizability (also known as Atomic Consistency or External Consistency) is a consistency level between strict consistency and sequential consistency.

For linearizability, suppose each operation receives a timestamp from a loosely synchronized global clock. Operations are linearized if and only if they always complete as though they were executed in a sequential order and each operation appears to complete in the order specified by the program. Likewise, if an operation’s timestamp precedes another, that operation must also precede the other operation in the sequence.

For example, consider a client completing a write at time point 1 (t1). A client issuing a read at t2 (for t2 > t1) should receive a value at least as recent as the previous write, completed at t1. However, the read might actually complete only by t3. Linearizability guarantees the read returns the most current value. Without linearizability guarantee, the returned value, current at t2 when the read began, might be “stale” by t3 because a concurrent write might happen between t2 and t3.

etcd does not ensure linearizability for watch operations. Users are expected to verify the revision of watch responses to ensure correct ordering.

etcd ensures linearizability for all other operations by default. Linearizability comes with a cost, however, because linearized requests must go through the Raft consensus process. To obtain lower latencies and higher throughput for read requests, clients can configure a request’s consistency mode to serializable, which may access stale data with respect to quorum, but removes the performance penalty of linearized accesses' reliance on live consensus.

线性化（也称为原子一致性或外部一致性）是严格一致性和顺序一致性之间的一致性级别。

对于线性化，假设每个操作从松散同步的全局时钟接收一个时间戳。当且仅当操作总是像按顺序执行一样完成，并且每个操作似乎按程序指定的顺序完成时，才线性化操作。同样，如果一个操作的时间戳先于另一个操作，那么该操作也必须先于序列中的另一个操作。

例如，假设客户机在时间点1（t1）完成写入。在t2发出read（对于t2>t1）的客户机应至少收到与在t1完成的上一次写入相同的最新值。然而，读取实际上可能只在t3之前完成。线性化保证读取返回最新值。如果没有线性化保证，返回值（读取开始时t2处的当前值）可能会被t3“过时”，因为t2和t3之间可能发生并发写入。

etcd不能确保监视操作的线性化。用户需要验证监视响应的修订，以确保正确排序。

默认情况下，etcd可确保所有其他操作的线性化。然而，线性化是有代价的，因为线性化的请求必须经过协商一致的过程。为了获得较低的延迟和较高的读请求吞吐量，客户机可以将请求的一致性模式配置为可串行化，这可能会访问有关仲裁的过时数据，但消除了线性化访问依赖实时一致性的性能损失。

测试记录

write测试

表格第1行测试数据

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1} --target-leader --conns=1 --clients=1 \
>     put --key-size=8 --sequential-keys --total=10000 --val-size=256
 10000 / 10000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 7s

Summary:
  Total:        7.3535 secs.
  Slowest:      0.0048 secs.
  Fastest:      0.0003 secs.
  Average:      0.0007 secs.
  Stddev:       0.0003 secs.
  Requests/sec: 1359.9057

Response time histogram:
  0.0003 [1]    |
  0.0008 [7950] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0012 [1535] |∎∎∎∎∎∎∎
  0.0016 [240]  |∎
  0.0021 [152]  |
  0.0025 [53]   |
  0.0030 [33]   |
  0.0034 [20]   |
  0.0039 [10]   |
  0.0043 [3]    |
  0.0048 [3]    |

Latency distribution:
  10% in 0.0006 secs.
  25% in 0.0006 secs.
  50% in 0.0006 secs.
  75% in 0.0007 secs.
  90% in 0.0010 secs.
  95% in 0.0012 secs.
  99% in 0.0022 secs.
  99.9% in 0.0037 secs.

(base) [root@yuyuan211 /home/wangb/etcd-test]#


(base) [root@yuyuan211 /home/wangb/etcd-test]# ps aux |grep etcd
root      29017  1.9  0.0 10616300 24752 pts/10 Sl+  13:46   0:14 /tmp/etcd-download-test/etcd --name=etcdtest --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000

表格第2行测试数据

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1} --target-leader  --conns=100 --clients=1000 \
>     put --key-size=8 --sequential-keys --total=100000 --val-size=256
INFO: 2021/01/18 14:00:00 parsed scheme: "endpoint"

INFO: 2021/01/18 14:00:00 ccResolverWrapper: sending new addresses to cc: [{http://localhost:2379  <nil> 0 <nil>}]
 100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 3s

Summary:
  Total:        3.5391 secs.
  Slowest:      0.0837 secs.
  Fastest:      0.0035 secs.
  Average:      0.0351 secs.
  Stddev:       0.0109 secs.
  Requests/sec: 28255.6626

Response time histogram:
  0.0035 [1]    |
  0.0116 [1]    |
  0.0196 [1298] |∎
  0.0276 [25892]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0356 [34394]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0436 [15557]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0517 [14631]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0597 [5577] |∎∎∎∎∎∎
  0.0677 [1873] |∎∎
  0.0757 [428]  |
  0.0837 [348]  |

Latency distribution:
  10% in 0.0235 secs.
  25% in 0.0272 secs.
  50% in 0.0318 secs.
  75% in 0.0424 secs.
  90% in 0.0504 secs.
  95% in 0.0550 secs.
  99% in 0.0665 secs.
  99.9% in 0.0812 secs.

(base) [root@yuyuan211 /home/wangb/etcd-test]#

(base) [root@yuyuan211 /home/wangb/etcd-test]# ps aux |grep etcd
root      29017  6.6  0.2 10687728 75552 pts/10 Sl+  13:46   0:56 /tmp/etcd-download-test/etcd --name=etcdtest --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000

表格第3行测试数据

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
>     put --key-size=8 --sequential-keys --total=100000 --val-size=256

INFO: 2021/01/18 14:11:16 ccResolverWrapper: sending new addresses to cc: [{http://127.0.0.1:2379  <nil> 0 <nil>}]
 100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 3s

Summary:
  Total:        3.5622 secs.
  Slowest:      0.0836 secs.
  Fastest:      0.0123 secs.
  Average:      0.0353 secs.
  Stddev:       0.0109 secs.
  Requests/sec: 28072.8070

Response time histogram:
  0.0123 [1]    |
  0.0194 [1105] |∎
  0.0266 [23001]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0337 [30319]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0408 [13738]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0480 [17523]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0551 [9420] |∎∎∎∎∎∎∎∎∎∎∎∎
  0.0622 [3625] |∎∎∎∎
  0.0693 [506]  |
  0.0765 [597]  |
  0.0836 [165]  |

Latency distribution:
  10% in 0.0235 secs.
  25% in 0.0268 secs.
  50% in 0.0319 secs.
  75% in 0.0437 secs.
  90% in 0.0501 secs.
  95% in 0.0548 secs.
  99% in 0.0649 secs.
  99.9% in 0.0791 secs.



(base) [root@yuyuan211 /home/wangb/etcd-test]# ps aux |grep etcd
root      29017  6.2  0.2 10687728 89612 pts/10 Sl+  13:46   1:41 /tmp/etcd-download-test/etcd --name=etcdtest --heartbeat-interval=200 --election-timeout=2000 --snapshot-count=5000

read测试

测试数据 Single connection read requests

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
(base) [root@yuyuan211 /home/wangb/etcd-test]# # Single connection read requests
(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=1 --clients=1 \
>     range $YOUR_KEY --consistency=l --total=10000
bench with linearizable range
 10000 / 10000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 9s

Summary:
  Total:        9.0010 secs.
  Slowest:      0.0064 secs.
  Fastest:      0.0004 secs.
  Average:      0.0009 secs.
  Stddev:       0.0005 secs.
  Requests/sec: 1110.9827

Response time histogram:
  0.0004 [1]    |
  0.0010 [9102] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0016 [432]  |∎
  0.0022 [76]   |
  0.0028 [144]  |
  0.0034 [101]  |
  0.0040 [54]   |
  0.0046 [36]   |
  0.0052 [29]   |
  0.0058 [21]   |
  0.0064 [4]    |

Latency distribution:
  10% in 0.0007 secs.
  25% in 0.0007 secs.
  50% in 0.0008 secs.
  75% in 0.0008 secs.
  90% in 0.0009 secs.
  95% in 0.0014 secs.
  99% in 0.0039 secs.
  99.9% in 0.0055 secs.
(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=1 --clients=1 \
>     range $YOUR_KEY --consistency=s --total=10000
bench with serializable range
 10000 / 10000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 7s

Summary:
  Total:        7.9876 secs.
  Slowest:      0.0095 secs.
  Fastest:      0.0004 secs.
  Average:      0.0008 secs.
  Stddev:       0.0005 secs.
  Requests/sec: 1251.9368

Response time histogram:
  0.0004 [1]    |
  0.0013 [9485] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0022 [161]  |
  0.0031 [207]  |
  0.0040 [83]   |
  0.0050 [48]   |
  0.0059 [11]   |
  0.0068 [3]    |
  0.0077 [0]    |
  0.0086 [0]    |
  0.0095 [1]    |

Latency distribution:
  10% in 0.0006 secs.
  25% in 0.0006 secs.
  50% in 0.0007 secs.
  75% in 0.0007 secs.
  90% in 0.0008 secs.
  95% in 0.0014 secs.
  99% in 0.0037 secs.
  99.9% in 0.0054 secs.
(base) [root@yuyuan211 /home/wangb/etcd-test]#

测试数据 Many concurrent read requests

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
(base) [root@yuyuan211 /home/wangb/etcd-test]# # Many concurrent read requests
(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
>     range $YOUR_KEY --consistency=l --total=100000
bench with linearizable range
INFO: 2021/01/18 14:54:38 parsed scheme: "endpoint"
INFO: 2021/01/18 14:54:38 ccResolverWrapper: sending new addresses to cc: [{http://127.0.0.1:2379  <nil> 0 <nil>}]
 100000 / 100000 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 10s

Summary:
  Total:        10.4905 secs.
  Slowest:      1.1881 secs.
  Fastest:      0.0004 secs.
  Average:      0.1044 secs.
  Stddev:       0.1186 secs.
  Requests/sec: 9532.4514

Response time histogram:
  0.0004 [1]    |
  0.1192 [67345]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.2380 [21111]        |∎∎∎∎∎∎∎∎∎∎∎∎
  0.3567 [6188] |∎∎∎
  0.4755 [3418] |∎∎
  0.5943 [1527] |
  0.7130 [232]  |
  0.8318 [123]  |
  0.9505 [48]   |
  1.0693 [5]    |
  1.1881 [2]    |

Latency distribution:
  10% in 0.0074 secs.
  25% in 0.0189 secs.
  50% in 0.0634 secs.
  75% in 0.1471 secs.
  90% in 0.2604 secs.
  95% in 0.3664 secs.
  99% in 0.5222 secs.
  99.9% in 0.7631 secs.







(base) [root@yuyuan211 /home/wangb/etcd-test]# benchmark --endpoints=${HOST_1},${HOST_2},${HOST_3} --conns=100 --clients=1000 \
>     range $YOUR_KEY --consistency=s --total=100000
bench with serializable range
INFO: 2021/01/18 14:54:49 parsed scheme: "endpoint"
INFO: 2021/01/18 14:54:49 ccResolverWrapper: sending new addresses to cc: [{http://127.0.0.1:2379  <nil> 0 <nil>}]
 100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 8s

Summary:
  Total:        8.8068 secs.
  Slowest:      1.0424 secs.
  Fastest:      0.0005 secs.
  Average:      0.0875 secs.
  Stddev:       0.0995 secs.
  Requests/sec: 11354.9122

Response time histogram:
  0.0005 [1]    |
  0.1047 [67942]        |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.2089 [21128]        |∎∎∎∎∎∎∎∎∎∎∎∎
  0.3131 [7264] |∎∎∎∎
  0.4172 [2300] |∎
  0.5214 [797]  |
  0.6256 [367]  |
  0.7298 [154]  |
  0.8340 [33]   |
  0.9382 [12]   |
  1.0424 [2]    |

Latency distribution:
  10% in 0.0052 secs.
  25% in 0.0137 secs.
  50% in 0.0518 secs.
  75% in 0.1290 secs.
  90% in 0.2170 secs.
  95% in 0.2829 secs.
  99% in 0.4543 secs.
  99.9% in 0.6792 secs.

read测试时的etcd打印信息

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
09072ms) to execute
2021-01-18 14:54:58.315985 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (128.237385ms) to execute
2021-01-18 14:54:58.316748 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (136.434995ms) to execute
2021-01-18 14:54:58.317021 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (125.138823ms) to execute
2021-01-18 14:54:58.327063 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (113.659252ms) to execute
2021-01-18 14:54:58.327171 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (140.480071ms) to execute
2021-01-18 14:54:58.328320 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (138.142424ms) to execute
2021-01-18 14:54:58.329457 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (136.980041ms) to execute
2021-01-18 14:54:58.330026 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (139.674614ms) to execute
2021-01-18 14:54:58.330674 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (137.950461ms) to execute
2021-01-18 14:54:58.330710 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (151.589201ms) to execute
2021-01-18 14:54:58.338877 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (149.623303ms) to execute
2021-01-18 14:54:58.339042 W | etcdserver: read-only range request "key:\"foo\" serializable:true " with result "range_response_count:1 size:30" took too long (148.882374ms) to execute

目录

Etcd调优和性能测试

说明

etcd 安装

二进制方式安装

etcd 启动

etcd 键值测试

etcd 参数优化

快照

历史数据压缩频率

时间参数

磁盘IO优先级，在全部etcd节点执行

etcd配置，环境变量方式

etcd配置， 命令行参数方式

benchmark安装

get方式

编译方式

benchmark指标

benchmark测试

启动etcd

参数说明

write测试

read测试

测试命令

read-only 时间参数设置优化

benchmark测试条件

磁盘IO 参数调优

查询命令

设置命令

优化后测试

说明

write测试

read测试

测试命令

对比结果

结论

附录

Linearizability

测试记录

write测试

read测试

etcd配置，命令行参数方式