文章目录
- Docker下部署Consul集群和ACL权限配置
- 规划与准备
- 搭建Consul集群
- 创建server1配置文件,启动server1节点
- 创建server2配置文件,启动server2节点
- 创建server3配置文件,启动server3节点
- 加入集群
- 验证Consul集群选举机制
- 配置join参数,节点自动加入集群
- server节点退出集群
- 节点自动加入集群
- 节点之间加入通讯密钥
- 增加ACL token权限配置
- 启用ACL,配置master token
- 配置agent token
Docker下部署Consul集群和ACL权限配置
规划与准备
本次计划部署的consul集群有3个节点,都是server类型
容器IP | 节点 | 类型 |
---|---|---|
172.17.0.2 | server1 | server |
172.17.0.3 | server2 | server |
172.17.0.4 | server3 | server |
把consul的数据文件都映射到宿主机上,有利于备份数据以及方便以后重构容器。
宿主机建立目录server1、server2、server3,下面分别存放3个consul节点的信息:
1 2 3 4 5 6 7 8 9 10 11 | [root@wuli-centOS7 ~]# mkdir -p /data/consul/server1/config [root@wuli-centOS7 ~]# mkdir -p /data/consul/server1/data [root@wuli-centOS7 ~]# mkdir -p /data/consul/server1/log [root@wuli-centOS7 ~]# mkdir -p /data/consul/server2/config [root@wuli-centOS7 ~]# mkdir -p /data/consul/server2/data [root@wuli-centOS7 ~]# mkdir -p /data/consul/server2/log [root@wuli-centOS7 ~]# mkdir -p /data/consul/server3/config [root@wuli-centOS7 ~]# mkdir -p /data/consul/server3/data [root@wuli-centOS7 ~]# mkdir -p /data/consul/server3/log |
搭建Consul集群
创建server1配置文件,启动server1节点
- 创建server1的配置文件:
1 | [root@wuli-centOS7 ~]# vim /data/consul/server1/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | { "datacenter": "dc1", "bootstrap_expect": 3, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_1", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" } } |
- 启动节点consul_server_1
1 | [root@wuli-centOS7 ~]# docker run -d -p 8510:8500 -v /data/consul/server1/data:/consul/data -v /data/consul/server1/config:/consul/config -e CONSUL_BIND_INTERFACE='eth0' --privileged=true --name=consul_server_1 consul agent -data-dir=/consul/data; |
docker run命令说明:
-
Environment Variable(环境变量):
CONSUL_BIND_INTERFACE=eth0:在容器启动时,自动绑定eth0端口的IP地址 -
docker参数:
-e:将时区信息传入到容器内部
-d:Daemon模式
-p:绑定端口
–privileged:表示以root权限运行
–name:指定实例名称
consul:consul启动命令
启动后,因为配置了bootstrap_expect=3,但只启动了一个server,所以会报错:没有集群领导者
1 2 3 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul monitor 2020-05-03T18:01:40.453Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader" 2020-05-03T18:01:57.802Z [ERROR] agent: Coordinate update error: error="No cluster leader" |
创建server2配置文件,启动server2节点
- 创建server2的配置文件:
1 | [root@wuli-centOS7 ~]# vim /data/consul/server2/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | { "datacenter": "dc1", "bootstrap_expect": 3, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_2", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" } } |
- 启动节点consul_server_2
1 | [root@wuli-centOS7 ~]# docker run -d -p 8520:8500 -v /data/consul/server2/data:/consul/data -v /data/consul/server2/config:/consul/config -e CONSUL_BIND_INTERFACE='eth0' --privileged=true --name=consul_server_2 consul agent -data-dir=/consul/data; |
创建server3配置文件,启动server3节点
- 创建server3的配置文件:
1 | [root@wuli-centOS7 ~]# vim /data/consul/server3/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | { "datacenter": "dc1", "bootstrap_expect": 3, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_3", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" } } |
- 启动节点consul_server_3
1 | [root@wuli-centOS7 ~]# docker run -d -p 8530:8500 -v /data/consul/server3/data:/consul/data -v /data/consul/server3/config:/consul/config -e CONSUL_BIND_INTERFACE='eth0' --privileged=true --name=consul_server_3 consul agent -data-dir=/consul/data; |
加入集群
- 查看节点server2、server3的IP,然后通过join命令把全部节点加入集群
1 2 3 4 | [root@wuli-centOS7 ~]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' consul_server_2 172.17.0.3 [root@wuli-centOS7 ~]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' consul_server_3 172.17.0.4 |
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul join 172.17.0.3 [root@wuli-centOS7 ~]# docker exec consul_server_1 consul join 172.17.0.4 |
- 查看后台日志:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul monitor 2020-05-03T19:04:39.254Z [INFO] agent: (LAN) joining: lan_addresses=[172.17.0.3] 2020-05-03T19:04:39.260Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul_server_2 172.17.0.3 2020-05-03T19:04:39.261Z [INFO] agent: (LAN) joined: number_of_nodes=1 2020-05-03T19:04:39.261Z [INFO] agent.server: Adding LAN server: server="consul_server_2 (Addr: tcp/172.17.0.3:8300) (DC: dc1)" 2020-05-03T19:04:39.265Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul_server_2.dc1 172.17.0.3 2020-05-03T19:04:39.266Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul_server_2.dc1 area=wan 2020-05-03T19:04:40.362Z [ERROR] agent: Coordinate update error: error="No cluster leader" 2020-05-03T19:04:44.019Z [INFO] agent: (LAN) joining: lan_addresses=[172.17.0.4] 2020-05-03T19:04:44.021Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul_server_3 172.17.0.4 2020-05-03T19:04:44.021Z [INFO] agent: (LAN) joined: number_of_nodes=1 2020-05-03T19:04:44.022Z [INFO] agent.server: Adding LAN server: server="consul_server_3 (Addr: tcp/172.17.0.4:8300) (DC: dc1)" 2020-05-03T19:04:44.027Z [INFO] agent.server: Found expected number of peers, attempting bootstrap: peers=172.17.0.2:8300,172.17.0.3:8300,172.17.0.4:8300 2020-05-03T19:04:44.049Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul_server_3.dc1 172.17.0.4 2020-05-03T19:04:44.049Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul_server_3.dc1 area=wan 2020-05-03T19:04:48.088Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2020-05-03T19:04:48.088Z [INFO] agent.server.raft: entering candidate state: node="Node at 172.17.0.2:8300 [Candidate]" term=2 2020-05-03T19:04:48.100Z [INFO] agent.server.raft: election won: tally=2 2020-05-03T19:04:48.101Z [INFO] agent.server.raft: entering leader state: leader="Node at 172.17.0.2:8300 [Leader]" 2020-05-03T19:04:48.101Z [INFO] agent.server.raft: added peer, starting replication: peer=78293668-16a6-1de0-673f-455d594e7447 2020-05-03T19:04:48.101Z [INFO] agent.server.raft: added peer, starting replication: peer=0b6169d8-7acc-ed24-682f-56ffd12b486c 2020-05-03T19:04:48.102Z [INFO] agent.server: cluster leadership acquired 2020-05-03T19:04:48.103Z [INFO] agent.server: New leader elected: payload=consul_server_1 2020-05-03T19:04:48.104Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Voter 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300}" next=1 2020-05-03T19:04:48.107Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 0b6169d8-7acc-ed24-682f-56ffd12b486c 172.17.0.4:8300}" 2020-05-03T19:04:48.112Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300}" 2020-05-03T19:04:48.120Z [INFO] agent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=172.17.0.2:8300 2020-05-03T19:04:48.129Z [INFO] agent.leader: started routine: routine="CA root pruning" 2020-05-03T19:04:48.129Z [INFO] agent.server: member joined, marking health alive: member=consul_server_1 2020-05-03T19:04:48.146Z [INFO] agent.server: member joined, marking health alive: member=consul_server_2 2020-05-03T19:04:48.156Z [INFO] agent.server: member joined, marking health alive: member=consul_server_3 2020-05-03T19:04:48.720Z [INFO] agent: Synced node info |
- 查看成员
1 2 3 4 5 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul members Node Address Status Type Build Protocol DC Segment consul_server_1 172.17.0.2:8301 alive server 1.7.2 2 dc1 <all> consul_server_2 172.17.0.3:8301 alive server 1.7.2 2 dc1 <all> consul_server_3 172.17.0.4:8301 alive server 1.7.2 2 dc1 <all> |
- 查看集群的选举情况,领导者为哪个节点等信息
1 2 3 4 5 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul operator raft list-peers Node ID Address State Voter RaftProtocol consul_server_1 0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 172.17.0.2:8300 leader true 3 consul_server_2 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300 follower true 3 consul_server_3 0b6169d8-7acc-ed24-682f-56ffd12b486c 172.17.0.4:8300 follower true 3 |
- 访问ui,节点正常,标星的节点表示是leader
验证Consul集群选举机制
Consul中只有server节点会参与Raft算法并且作为peer set中的一部分。Raft中的节点总是处于以下三种状态之一: follower、candidate或leader。目前server1是leader,我们下面重启consul_server_1容器,观察consul集群变化情况。
- 重启consul_server_1
1 | [root@wuli-centOS7 ~]# docker restart consul_server_1 |
- 观察server2和server3的日志,consul_server_2后台日志:
1 2 3 4 5 6 7 8 | [root@wuli-centOS7 ~]# docker exec consul_server_2 consul monitor 2020-05-03T19:26:29.564Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul_server_1 has failed, no acks received 2020-05-03T19:26:30.533Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul_server_1.dc1 2020-05-03T19:26:30.533Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul_server_1.dc1 area=wan 2020-05-03T19:26:30.564Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul_server_1 2020-05-03T19:26:30.565Z [INFO] agent.server: Updating LAN server: server="consul_server_1 (Addr: tcp/172.17.0.2:8300) (DC: dc1)" 2020-05-03T19:26:33.542Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=172.17.0.4:8300 leader=172.17.0.2:8300 2020-05-03T19:26:33.565Z [INFO] agent.server: New leader elected: payload=consul_server_3 |
- consul_server_3后台日志:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | [root@wuli-centOS7 ~]# docker exec consul_server_3 consul monitor 2020-05-03T19:26:28.151Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with consul_server_1 failed: dial tcp 172.17.0.2:8301: connect: connection refused 2020-05-03T19:26:30.235Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul_server_1 has failed, no acks received 2020-05-03T19:26:30.420Z [ERROR] agent: Coordinate update error: error="rpc error making call: stream closed" 2020-05-03T19:26:30.536Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul_server_1.dc1 2020-05-03T19:26:30.536Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul_server_1.dc1 area=wan 2020-05-03T19:26:30.729Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul_server_1 2020-05-03T19:26:30.729Z [INFO] agent.server: Updating LAN server: server="consul_server_1 (Addr: tcp/172.17.0.2:8300) (DC: dc1)" 2020-05-03T19:26:32.234Z [WARN] agent.server.memberlist.wan: memberlist: Was able to connect to consul_server_1.dc1 but other probes failed, network may be misconfigured 2020-05-03T19:26:33.536Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=172.17.0.2:8300 2020-05-03T19:26:33.536Z [INFO] agent.server.raft: entering candidate state: node="Node at 172.17.0.4:8300 [Candidate]" term=3 2020-05-03T19:26:33.546Z [INFO] agent.server.raft: election won: tally=2 2020-05-03T19:26:33.546Z [INFO] agent.server.raft: entering leader state: leader="Node at 172.17.0.4:8300 [Leader]" 2020-05-03T19:26:33.546Z [INFO] agent.server.raft: added peer, starting replication: peer=0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 2020-05-03T19:26:33.546Z [INFO] agent.server.raft: added peer, starting replication: peer=78293668-16a6-1de0-673f-455d594e7447 2020-05-03T19:26:33.547Z [INFO] agent.server: cluster leadership acquired 2020-05-03T19:26:33.548Z [INFO] agent.server: New leader elected: payload=consul_server_3 2020-05-03T19:26:33.550Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300}" 2020-05-03T19:26:33.551Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 172.17.0.2:8300}" 2020-05-03T19:26:33.554Z [INFO] agent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=172.17.0.4:8300 2020-05-03T19:26:33.555Z [INFO] agent.leader: started routine: routine="CA root pruning" |
- consul_server_1重启后,查看领导者,此时consul_server_3已被推选为领导者:
1 2 3 4 5 | [root@wuli-centOS7 ~]# docker exec -it consul_server_1 consul operator raft list-peers Node ID Address State Voter RaftProtocol consul_server_3 0b6169d8-7acc-ed24-682f-56ffd12b486c 172.17.0.4:8300 leader true 3 consul_server_1 0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 172.17.0.2:8300 follower true 3 consul_server_2 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300 follower true 3 |
配置join参数,节点自动加入集群
在Consul集群中,一个节点优雅退出,不会影响到集群中其他节点的正常运行,而集群的数据中心会把离开的节点标志为left状态,等到该节点重新加入集群,状态会变为alive,默认情况下,数据中心会保留离开节点信息72小时,72小时候后如果仍然没有加入集群,则会把该节点的信息移除掉。
server节点退出集群
下面演示节点退出的情况:
- consul_server_2优雅退出
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_2 consul leave Graceful leave complete |
- 查看成员状态,
1 2 3 4 5 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul members Node Address Status Type Build Protocol DC Segment consul_server_1 172.17.0.2:8301 alive server 1.7.2 2 dc1 <all> consul_server_2 172.17.0.3:8301 left server 1.7.2 2 dc1 <all> consul_server_3 172.17.0.4:8301 alive server 1.7.2 2 dc1 <all> |
- 查看leader
1 2 3 4 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul operator raft list-peers Node ID Address State Voter RaftProtocol consul_server_3 0b6169d8-7acc-ed24-682f-56ffd12b486c 172.17.0.4:8300 leader true 3 consul_server_1 0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 172.17.0.2:8300 follower true 3 |
查看后台日志,没有报错,访问ui正常,说明配置文件的bootstrap_expect=3,只是在创建集群的时候期待的节点数量,如果达不到就不会初次创建集群,但节点数据量达到3后,集群初次创建成功,后面如果server通过优雅退出,不影响集群的健康情况,集群仍然会正常运行,而优雅退出的集群的状态会标志为“left”。
- 重新启动consul_server_2后,不会自动加入集群,因为配置文件没有start_join和retry_join参数,需要通过命令consul join加入集群,通过consul_server_2或者集群中的其他server发起命令都可以:
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_2 consul join 172.17.0.2 Successfully joined cluster by contacting 1 nodes. |
或者:
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul join 172.17.0.3 Successfully joined cluster by contacting 1 nodes. |
节点自动加入集群
全部server的配置文件添加start_join和retry_join参数,可以在重启后自动集群。
- 参看各个server的ip:
1 2 3 4 5 6 | [root@wuli-centOS7 server4]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' consul_server_1 172.17.0.2 [root@wuli-centOS7 server4]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' consul_server_2 172.17.0.3 [root@wuli-centOS7 server4]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' consul_server_3 172.17.0.4 |
- 分别编辑各个server的配置文件
1 | [root@wuli-centOS7 ~]# vim /data/consul/server1/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | { "datacenter": "dc1", "bootstrap_expect": 3, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_3", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" }, "start_join":[ "172.17.0.2", "172.17.0.3", "172.17.0.4" ], "retry_join":[ "172.17.0.2", "172.17.0.3", "172.17.0.4" ] } |
- 重载配置文件,验证配置是否正确
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul reload Configuration reload triggered |
- 配置正确,以后重启consul server,或者优雅退出后再启动,会自动加入集群,无需使用命令join。
节点之间加入通讯密钥
增加通讯密钥,可以防止其他节点加入集群。步骤如下:
- 使用consul keygen命令生成通讯密钥
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul keygen zVCGFgICqf5MAU61Wd/1wDP1hoQ37rQQFVvMkhzpM1c= |
- 把密钥信息分别写入3个server的配置文件中
1 | [root@wuli-centOS7 ~]# vim /data/consul/server1/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | { "datacenter": "dc1", "bootstrap_expect": 3, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_1", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" }, "encrypt": "zVCGFgICqf5MAU61Wd/1wDP1hoQ37rQQFVvMkhzpM1c=" } |
- consul重新加载配置文件
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul reload Configuration reload triggered |
- 实验中,发现重载配置文件后,其他节点仍然可以加入集群,需要重新启动consul
1 2 3 4 5 6 7 8 | [root@wuli-centOS7 ~]# docker restart consul_server_1 consul_server_1 [root@wuli-centOS7 ~]# docker restart consul_server_2 consul_server_2 [root@wuli-centOS7 ~]# docker restart consul_server_3 consul_server_3 |
后面如果有新的节点,要加入集群中,必须提供encrypt才行。
- 再创建一个节点server4,尝试加入有密钥的集群:
5.1 创建配置配置文件:
1 | [root@wuli-centOS7 ~]# vim /data/consul/server4/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | { "datacenter": "dc1", "bootstrap_expect": 1, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_4", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" } } |
5.2 启动server4
1 2 | [root@wuli-centOS7 ~]# docker run -d -p 8540:8500 -v /data/consul/server4/data:/consul/data -v /data/consul/server4/config:/consul/config -e CONSUL_BIND_INTERFACE='eth0' --privileged=true --name=consul_server_4 consul agent -data-dir=/consul/data; cdf6edee727f92baf2f7d324cb9522644c851f81fe62356fb2fb9aad126eaf13 |
5.3 尝试加入集群,失败:
1 2 3 4 5 6 | [root@wuli-centOS7 ~]# docker exec consul_server_4 consul join 172.17.0.2 Error joining address '172.17.0.2': Unexpected response code: 500 (1 error occurred: * Failed to join 172.17.0.2: Remote state is encrypted and encryption is not configured ) Failed to join any nodes. |
5.4 在配置文件中加入通讯密钥
1 | [root@wuli-centOS7 ~]# vim /data/consul/server4/config/config.json |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | { "datacenter": "dc1", "bootstrap_expect": 1, "data_dir": "/consul/data", "log_file": "/consul/log/", "log_level": "INFO", "node_name": "consul_server_4", "client_addr": "0.0.0.0", "server": true, "ui": true, "enable_script_checks": true, "addresses": { "https": "0.0.0.0", "dns": "0.0.0.0" }, "encrypt": "zVCGFgICqf5MAU61Wd/1wDP1hoQ37rQQFVvMkhzpM1c=" } |
5.5 重载配置文件,然后尝试加入集群,仍然失败,说明添加密钥都需要重启consul才生效
1 2 3 4 5 6 7 8 9 | [root@wuli-centOS7 ~]# docker exec consul_server_4 consul reload Configuration reload triggered [root@wuli-centOS7 ~]# docker exec consul_server_4 consul join 172.17.0.2 Error joining address '172.17.0.2': Unexpected response code: 500 (1 error occurred: * Failed to join 172.17.0.2: Remote state is encrypted and encryption is not configured ) Failed to join any nodes. |
5.6 重启server4,加入集群成功!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | [root@wuli-centOS7 server4]# docker restart consul_server_4 consul_server_4 [root@wuli-centOS7 server4]# docker exec consul_server_4 consul join 172.17.0.2 Successfully joined cluster by contacting 1 nodes. [root@wuli-centOS7 server4]# docker exec consul_server_1 consul members Node Address Status Type Build Protocol DC Segment consul_server_1 172.17.0.2:8301 alive server 1.7.2 2 dc1 <all> consul_server_2 172.17.0.3:8301 alive server 1.7.2 2 dc1 <all> consul_server_3 172.17.0.4:8301 alive server 1.7.2 2 dc1 <all> consul_server_4 172.17.0.5:8301 alive server 1.7.2 2 dc1 <all> [root@wuli-centOS7 server4]# docker exec consul_server_1 consul operator raft list-peers Node ID Address State Voter RaftProtocol consul_server_3 0b6169d8-7acc-ed24-682f-56ffd12b486c 172.17.0.4:8300 leader true 3 consul_server_2 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300 follower true 3 consul_server_1 0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 172.17.0.2:8300 follower true 3 consul_server_4 587e81c7-f1b5-5f19-311c-741c06ca446d 172.17.0.5:8300 follower false 3 |
增加ACL token权限配置
配置master的token,master的token可以自由定义,但为了与其他token格式一致,官方建议使用64位的UUID。consul的配置文件可以有多个,文件后缀名可以是json或者hcl,我们这里使用hcl来演示。
启用ACL,配置master token
Consul的ACL功能需要显示启用,在配置文件中通过设置参数acl.enabled=true即可。
下面演示了两种方法来配置ACL,个人推荐方法二。
方法一:配置acl.enabled=true,然后通过命令consul acl bootstrap生成token,之后把改token作为master的token。
- 添加配置文件acl.hcl:
1 | [root@wuli-centOS7 ~]# vim /data/consul/server4/config/acl.hcl |
1 2 3 4 5 6 7 8 | primary_datacenter = "dc1" acl { enabled = true default_policy = "deny" enable_token_persistence = true tokens { } } |
- 重载配置文件,创建初始token,生成的SecretID就是token
1 2 3 4 5 6 7 8 9 10 11 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul reload Configuration reload triggered [root@wuli-centOS7 ~]# consul acl bootstrap AccessorID: b6676320-dbef-4020-ae69-8a47ae13dcef SecretID: 474dcea7-ee4d-3f11-1af1-a38eb37d3f5d Description: Bootstrap Token (Global Management) Local: false Create Time: 2020-04-28 11:30:42.2992871 +0800 CST Policies: 00000000-0000-0000-0000-000000000001 - global-management |
- 修改配置文件acl.hcl,加入mater token
1 | [root@wuli-centOS7 ~]# vim /data/consul/server4/config/acl.hcl |
1 2 3 4 5 6 7 8 9 | primary_datacenter = "dc1" acl { enabled = true default_policy = "deny" enable_token_persistence = true tokens { master = "474dcea7-ee4d-3f11-1af1-a38eb37d3f5d } } |
- 重启服务,验证
方法二:
- 使用linux的uuidgen命令生成一个64位UUID作为master token
1 2 | [root@wuli-centOS7 ~]# uuidgen dcb93655-0661-4ea1-bfc4-e5744317f99e |
- 编写acl.hcl文件文件
1 | [root@wuli-centOS7 ~]# vim /data/consul/server4/config/acl.hcl |
1 2 3 4 5 6 7 8 9 | primary_datacenter = "dc1" acl { enabled = true default_policy = "deny" enable_token_persistence = true tokens { master = "dcb93655-0661-4ea1-bfc4-e5744317f99e" } } |
修改config.json配置,把bootstrap_expect修改成1
3.重载配置文件,验证是否正确。
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul reload Configuration reload triggered |
- 优雅关闭其他server,重启consul_server_1容器
1 2 3 4 5 | [root@wuli-centOS7 ~]# docker exec consul_server_2 consul leave [root@wuli-centOS7 ~]# docker exec consul_server_3 consul leave [root@wuli-centOS7 ~]# docker exec consul_server_4 consul leave [root@wuli-centOS7 ~]# docker restart consul_server_1 |
-
访问ui,提示需要输入token,输入我们上面的mater token即可
-
此时,使用consul大部分命令,都需要带上token,否则报错:
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul info Error querying agent: Unexpected response code: 403 (Permission denied) |
带上token参数:
1 2 3 4 5 6 7 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul info -token=dcb93655-0661-4ea1-bfc4-e5744317f99e agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 …… |
- 不是在docker启动的consul,可以通过增加环境变量CONSUL_HTTP_TOKEN代替每次命令后带token参数
1 2 3 4 | [root@wuli-centOS7 ~]# vi /etc/profile export CONSUL_HTTP_TOKEN=dcb93655-0661-4ea1-bfc4-e5744317f99e [root@wuli-centOS7 ~]# source /etc/profile |
在docker运行中的consul容器,目前不清楚怎么修改环境变量永久生效,但可以rm移除旧容器,然后在run的时候添加上环境变量,这里就不演示了。
查看consul_server_1容器的全部环境变量
1 2 3 4 5 6 7 | [root@wuli-centOS7 ~]# docker exec consul_server_1 env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=2c2c68c4058f CONSUL_BIND_INTERFACE=eth0 CONSUL_VERSION=1.7.2 HASHICORP_RELEASES=https://releases.hashicorp.com HOME=/root |
查资料CONSUL_BIND_INTERFACE是什么参数
配置agent token
agent token是每个集群都需要的token
没有配置agent token,查看日志报以下警告:
1 2 3 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul monitor -token=dcb93655-0661-4ea1-bfc4-e5744317f99e 2020-05-06T02:40:09.386Z [WARN] agent: Coordinate update blocked by ACLs: accessorID= |
- 使用API生成全权限的token作为agent token,可以根据实际情况分配agent token的权限
1 2 3 4 5 6 | [root@wuli-centOS7 ~]# curl -X PUT \ http://localhost:8510/v1/acl/create \ -H 'X-Consul-Token: dcb93655-0661-4ea1-bfc4-e5744317f99e' \ -d '{"Name": "dc1","Type": "management"}' {"ID":"7f587432-3650-9073-e3f4-445a2463b11f"} |
- 把生成的token写入到配置文件acl.hcl中
1 | [root@wuli-centOS7 ~]# vim /data/consul/server1/config/acl.hcl |
1 2 3 4 5 6 7 8 9 10 | primary_datacenter = "dc1" acl { enabled = true default_policy = "deny" enable_token_persistence = true tokens { master = "dcb93655-0661-4ea1-bfc4-e5744317f99e" agent = "7f587432-3650-9073-e3f4-445a2463b11f" } } |
- 重载配置文件,这里无需重启consul或者容器
1 2 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul reload -token=dcb93655-0661-4ea1-bfc4-e5744317f99e Configuration reload triggered |
- 查看日志,重载配置文件后,已经不报任何警告了
1 2 3 4 | [root@wuli-centOS7 ~]# docker exec consul_server_1 consul monitor -token=dcb93655-0661-4ea1-bfc4-e5744317f99e 2020-05-06T03:00:20.034Z [INFO] agent: Caught: signal=hangup 2020-05-06T03:00:20.034Z [INFO] agent: Reloading configuration... |
- 在集群中acl的配置信息是一致的,所以直接把server1的acl.hcl配置文件复制到其他server节点的配置文件夹下即可
1 2 | [root@wuli-centOS7 config]# cp /data/consul/server1/config/acl.hcl /data/consul/server2/config/ [root@wuli-centOS7 config]# cp /data/consul/server1/config/acl.hcl /data/consul/server3/config/ |
- 重启其他容器,查看集群情况
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | [root@wuli-centOS7 config]# docker restart consul_server_2 consul_server_2 [root@wuli-centOS7 config]# docker restart consul_server_3 consul_server_3 [root@wuli-centOS7 config]# docker exec consul_server_1 consul members -token=dcb93655-0661-4ea1-bfc4-e5744317f99e Node Address Status Type Build Protocol DC Segment consul_server_1 172.17.0.2:8301 alive server 1.7.2 2 dc1 <all> consul_server_2 172.17.0.3:8301 alive server 1.7.2 2 dc1 <all> consul_server_3 172.17.0.4:8301 alive server 1.7.2 2 dc1 <all> [root@wuli-centOS7 config]# docker exec consul_server_1 consul operator raft list-peers -token=dcb93655-0661-4ea1-bfc4-e5744317f99e Node ID Address State Voter RaftProtocol consul_server_1 0214e0b9-04fe-e8ad-c855-b5091dfc8e2e 172.17.0.2:8300 leader true 3 consul_server_2 78293668-16a6-1de0-673f-455d594e7447 172.17.0.3:8300 follower true 3 consul_server_3 0b6169d8-7acc-ed24-682f-56ffd12b486c 172.17.0.4:8300 follower true 3 |
可以看到,由于前面配置了join参数,所以节点会自动加入集群。
7. 分别访问各个server的ui,查看到的token信息是一致的
至此,consul集群搭建和ACL token权限配置完成!
其他联想:
如果第一个节点配置了bootstrap:true或者bootstrap_expect=1,那么自己就会是领导,后面的节点在启动前,就可以在配置文件中加入start_join和retry_join的配置信息,然后启动,自动加入集群;
如果第一个节点配置的是bootstrap_expect=n(n大于1),那么由于没有达到节点数,所以一直不会有leader节点,所以后面的节点在启动前,如果配置了start_join和retry_join就会报错,所以只能在启动后,通过consul join命令来加入集群。