• 0
• 0

mariadb galera集群在Openstack中的应用

技术分享 4个月前 (11-20) 0 999+

关注

本文来自博客园，作者：T-BARBARIANS，博文严禁转载，转载必究！

最近对kolla部署的Openstack做了一些调研，尤其是对mariadb galera集群自身的高可用机制，以及haproxy，proxysql组件对mariadb galera集群的负载均衡方法、支持程度进行了一些分析和探索。我把这次调研的收获总结成了知识型输出，希望可以给有相关需求的同行一些参考。

mariadb cluster概览

mariadb 集群有多种模式，针对不同的业务场景，各有所长。

集群模式	类型	适用场景	延迟	官方支持
MariaDB Galera Cluster	同步复制，强一致性	高频OLTP	低	✅ 是
MariaDB Replication	异步复制，最终一致性	读写分离	中高	✅ 是
MariaDB ColumnStore	列式存储分布式集群	OLAP分析	高	✅ 商业版
MariaDB Xpand	分布式 SQL 数据库	超大规模OLTP	中	✅ 商业版

mariadb galera集群简介

什么是galera集群

Galera 是 MariaDB 官方推荐的 Multi Master，同步复制集群​​方案，主要面向需要​强一致性和​​高可用​​的 OLTP（Online Transaction Processing）场景。

kolla 在 openstack 环境部署时，默认将 galera 作为后端数据库集群方案。官方建议 galera 集群至少需要三个 mariadb 节点。

galera集群同步机制简介

从应用向 galera 集群发起数据请求开始，直至完成事务提交，应用与 galera 集群的交互，以及 galera 集群的同步机制可简单概括为：

sequenceDiagram participant Client as Client <br>（Client_A, Client_B, Client_C） participant mariadb_A as mariadb_A participant mariadb_B as mariadb_B participant mariadb_C as mariadb_C participant GCS as gloabl GCS (Group Communication System)<br>（mariadb_A, mariadb_B, mariadb_C together） Note over Client,mariadb_C: 阶段一：客户端并发请求多个节点执行更新 Client->>mariadb_A: UPDATE users SET name='Tom' WHERE id=1 Client->>mariadb_B: UPDATE users SET name='Jerry' WHERE id=1 Client->>mariadb_C: UPDATE users SET name='Lucy' WHERE id=2 Note over mariadb_A,mariadb_C: 阶段二：各节点生成写集并广播 mariadb_A->>mariadb_A: 生成写集A mariadb_B->>mariadb_B: 生成写集B mariadb_C->>mariadb_C: 生成写集C mariadb_A->>GCS: GCS广播写集A mariadb_B->>GCS: 广播写集B mariadb_C->>GCS: 广播写集C Note over GCS: 阶段三：全局顺序排序（Total Order Isolation，TOI）<br>确保各节点按同一顺序处理并发事务 GCS->>GCS: 分配全局seqno: A=101, B=102, C=103 GCS-->>mariadb_A: 分发全局seqno [A=101, B=102, C=103] GCS-->>mariadb_B: 分发全局seqno [A=101, B=102, C=103] GCS-->>mariadb_C: 分发全局seqno [A=101, B=102, C=103] Note over mariadb_A,mariadb_C: 阶段四：各节点按顺序执行冲突检测 mariadb_A->>mariadb_A: 依据seqno顺序处理 mariadb_A->>mariadb_A: 1.提交A mariadb_A->>mariadb_A: 2.B冲突，拒绝 mariadb_A->>mariadb_A: 3.提交C mariadb_B->>mariadb_B: 与mariadb_A逻辑一致 mariadb_C->>mariadb_C: 与mariadb_A逻辑一致 Note over Client,mariadb_C: 阶段五：返回结果 mariadb_A-->>Client: 事务 A 成功 mariadb_B-->>Client: 事务 B 失败（冲突） mariadb_C-->>Client: 事务 C 成功

• 客户端A、B、C分别同时连接 mariadb_A，mariadb_B，mariadb_C，并发写操作
• 各mariadb节点分别生成对应事务的写集（WriteSet）A，写集B，写集C。例如写集A的内容

  {   "transaction_id": "a1b2c3d4-5678",   "global_seqno": 1024,          // 临时序列号   "source_id": "mariadb_A",      // 发起节点   "timestamp": 1630000000.123,   // 时间戳      "schema": {     "name": "users",            // 表名     "version": 3                // 表结构版本号   },      "rows": [     {       "key": {"id": 1},        // 主键值       "action": "UPDATE",      // 操作类型       "before": {"name": "LiLei"},  // 修改前的值       "after": {"name": "Tom"},     // 修改后的值       "version": 5                  // 行版本号（用于冲突检测），行被修改后递增1     }   ],      "dependencies": [] // 依赖的前序事务seqno }  

  • 写集包含了事务写操作的关键信息
  • 写集用于向集群广播，通告此次sql操作的影响范围。不是原始sql语句，而是哪些数据将被修改的相关信息
• 写集广播。各自mariadb节点使用 Galera 的 GCS（Group Communication System） 将各自写集广播
• 事务全局排序。
  • GCS 使用分布式一致性协议（类似 Paxos，galera称为Virtually Synchronous Replication）在集群协商事务的全局执行顺序
  • 为多个并发事务（A, B, C）分配全局唯一的，严格递增的 seqno，从而达成所有事务的全局执行顺序
• 事务认证（事务冲突检测）与事务提交。
  • 如果当前待提交事务与并发的，已提交的事务无冲突，则提交该事务。否则事务拒绝，并返回 deadlock
  • 各节点按照全局 seqno 按序提交对应事务

在 ​​Galera Cluster​​ 的多主架构中，事务的最终结果​不取决于事务的发起时间​​，而是由 ​​GCS（Group Communication System）的全局排序（Total Ordering）​​ 决定。

galera集群的优缺点

主要优点：

集群模式	类型
Multi-Master架构 ​	所有节点均可处理读写请求
同步复制	采用写集复制，通过组通信协议（GCS）进行同步提交
​强一致性​	​所有事务在提交前必须经过全局认证、排序、冲突检测，最终实现一致性，也意味着所有节点数据对等
无单点故障	任意节点宕机不影响集群整体可用性
新节点自动同步	新节点可以自动加入集群，通过 State Snapshot Transfer (SST) 或 Incremental State Transfer (IST) 同步状态

主要缺点：

集群模式	类型
写入冲突导致回滚​	并发修改相同一行的数据会被检测为冲突，需要应用自行处理
事务延迟依赖最慢节点	所有节点在事务提交前需达成共识，进行全局排序和冲突检测造成事务延迟
写入水平扩展受限 ​	事务广播同步，写性能受限于最慢节点，且每个节点都互为副本，无法通过增加节点提升写吞吐

相同行事务并发更新

• 表结构

MariaDB [test]> desc tmp; +-------------+--------------+------+-----+---------------------+----------------+ | Field       | Type         | Null | Key | Default             | Extra          | +-------------+--------------+------+-----+---------------------+----------------+ | id          | int(11)      | NO   | PRI | NULL                | auto_increment | | data        | varchar(255) | YES  |     | NULL                |                | | created_at  | timestamp    | NO   |     | current_timestamp() |                | | data_origin | varchar(255) | YES  |     | NULL                |                | +-------------+--------------+------+-----+---------------------+----------------+ 4 rows in set (0.003 sec)   

• 并发冲突demo

import uuid import pymysql import threading  DB_HOSTS = ["18.242.143.30", "18.242.143.31", "18.242.143.32"] DB_USER = "root" DB_PASS = "nuage#2345" DB_NAME = "test"  def write_to_db(ip, data, data_origin, row_id):     try:         conn = pymysql.connect(host = ip, user = DB_USER, password = DB_PASS, database = DB_NAME)         cursor = conn.cursor()          # 事务开启         cursor.execute("BEGIN;")          # 更新插入         cursor.execute("""         INSERT INTO tmp (id, data, data_origin)              VALUES (%s, %s, %s)             ON DUPLICATE KEY UPDATE data = VALUES(data), data_origin = VALUES(data_origin);""",              (row_id, data, data_origin))          # 事务提交         conn.commit()          print(f"[{ip}] ✅ Update Success")     except Exception as e:         print(f"[{ip}] ❌ Update Error: {e}")     finally:         conn.close()  thread1 = threading.Thread(target=write_to_db, args=(DB_HOSTS[0], str(uuid.uuid4())[:8], 'value_from_30', 3)) thread2 = threading.Thread(target=write_to_db, args=(DB_HOSTS[1], str(uuid.uuid4())[:8], 'value_from_31', 3)) thread3 = threading.Thread(target=write_to_db, args=(DB_HOSTS[2], str(uuid.uuid4())[:8], 'value_from_32', 3))  # 创建并发线程 thread1.start() thread2.start() thread3.start()  thread1.join() thread2.join() thread3.join() 

• 结果输出

[18.242.143.31] ✅ Update Insert Success [18.242.143.30] ❌ Update Insert Error: (1213, 'Deadlock found when trying to get lock; try restarting transaction') [18.242.143.32] ❌ Update Insert Error: (1213, 'Deadlock found when trying to get lock; try restarting transaction') 

MariaDB [test]> select * from tmp; +----+----------+---------------------+---------------+ | id | data     | created_at          | data_origin   | +----+----------+---------------------+---------------+ |  3 | 04d2ec9a | 2025-08-11 13:58:07 | value_from_31 | +----+----------+---------------------+---------------+ 2 rows in set (0.001 sec) 

或者其它结果：

[18.242.143.32] ✅ Update Insert Success [18.242.143.31] ❌ Update Insert Error: (1213, 'Deadlock found when trying to get lock; try restarting transaction') [18.242.143.30] ❌ Update Insert Error: (1213, 'Deadlock found when trying to get lock; try restarting transaction') 

MariaDB [test]> select * from tmp; +----+----------+---------------------+---------------+ | id | data     | created_at          | data_origin   | +----+----------+---------------------+---------------+ |  3 | f0d31a50 | 2025-08-11 13:58:07 | value_from_32 | +----+----------+---------------------+---------------+ 2 rows in set (0.001 sec) 

• 虽然 galera 支持多节点并发写，但是相同行的并发操作还是会受限。
• 相同行的并发结果随机，取决于谁的事务先到达。但是最后，所有节点一定会按照协商的全局顺序应用并发事务，保证所有节点数据的一致性。

相同行事务串行更新

将线程创建方式从并发改为串行启动：

thread1.start() thread1.join()  thread2.start() thread2.join()  thread3.start() thread3.join() 

• 结果输出

[18.242.143.30] ✅ Update Insert Success [18.242.143.31] ✅ Update Insert Success [18.242.143.32] ✅ Update Insert Success 

不再出现相同行并发事务冲突问题。

不同行事务并发更新

并发更新插入，操作不同行

DB_HOSTS = ["18.242.143.30", "18.242.143.31", "18.242.143.32"]  thread1 = threading.Thread(target=write_to_db, args=(DB_HOSTS[0], str(uuid.uuid4())[:8], 'value_from_30', 3)) thread2 = threading.Thread(target=write_to_db, args=(DB_HOSTS[1], str(uuid.uuid4())[:8], 'value_from_31', 4)) thread3 = threading.Thread(target=write_to_db, args=(DB_HOSTS[2], str(uuid.uuid4())[:8], 'value_from_32', 5)) 

• 结果输出

[18.242.143.32] ✅ Update Insert Success [18.242.143.31] ✅ Update Insert Success [18.242.143.30] ✅ Update Insert Success 

galera三节点集群搭建

• 配置文件举例

[mysqld] bind-address=18.242.143.30      # 监听地址 log_bin=mysql-bin binlog_format=ROW default_storage_engine=InnoDB   # galera只支持InnoDB引擎 innodb_autoinc_lock_mode=2  wsrep_on=ON                                                     # 启用galare插件 wsrep_cluster_name="galera-cluster"                             # galera集群名称 wsrep_cluster_address="gcomm://mariadb-1,mariadb-2,mariadb-3"   # galera集群地址列表 wsrep_provider=/usr/lib64/galera/libgalera_smm.so  wsrep_node_name="mariadb-1"             # 当前节点名称 wsrep_node_address="18.242.143.30"      # 当前节点通信地址  wsrep_sst_method=mariabackup            # sst(State Snapshot Transfer)同步默认方法。不锁表，写入不阻塞 wsrep_sst_auth="sstuser:nuage#2345"     # sst认证凭据  wsrep_slave_threads=4 max_connections=10000 

• 其余两节点配置只需要修改 bind-address ， wsrep_node_name 和 wsrep_node_address
• 当集群是第一次启动，可任选其中一个节点执行：galera_new_cluster
  • galera_new_cluster 生成集群uuid标识，后续加入的节点都使用该ID
  • galera_new_cluster 跳过集群成员检查。常规启动时（systemctl start mariadb），本节点会尝试连接 wsrep_cluster_address 中的其他节点，但是 galera_new_cluster 会跳过该步骤
  • 最先启动的节点成为 Donor 节点（数据源节点）。后续节点加入时，会从该节点触发 SST（State Snapshot Transfer，全量同步）或 IST（Incremental State Transfer，增量同步）
• 主节点启动成功后，其余节点执行：systemctl start mariadb 即可

galera集群节点数规则

mariabd 官方建议 galera 集群至少应由三个节点组成。

galera 集群需要遵循多数派规则：

• 剩余节点数可以构成多数派（剩下的节点数需要大于原始总集群的50%），则该集群可以正常工作
• 剩余节点数不满足多数派，则该集群终止写入能力。

假设集群由两个几点构成：

• 正常：Node1 ↔ Node2 正常通信
• 异常场景：网络故障导致两节点相互失联
  • Node1 认为 Node2 挂了
  • Node2 认为 Node1 挂了

如果没有多数派规则，两个节点都会以为自己是唯一存活节点，于是都可以接受写请求，最终造成两边数据非一致，导致脑裂。所以，为了尽可能避免脑裂，galera 在仲裁规则里规定：剩下的节点必须获得多数派票数，才能成为 Primary，才能继续写，否则终止写入，状态切换为 Non-Primary。有如下集群多数派计算公式和表格：

majority = [节点总数 / 2] + 1  

集群节点数	需要在线的最少节点数（majority）	允许故障数	故障容忍率	说明
1	1	0	0%	孤立节点，非集群
2	2	0	0%	任意一个挂了，剩下的一个失去多数派，集群进入只读状态，即 Non-Primary
3	2	1	33%	允许一个节点故障，剩下两个还能继续写
4	3	1	25%	与3节点集群一样，只允许一个节点故障
5	3	2	40%	允许两个节点故障
6	4	2	33%	与5节点集群一样，只允许两个节点故障

表格反映了一些事实：

• 奇数个节点的允许故障数与 +1 的偶数个节点允许故障数相同
• 奇数个节点的故障容忍率比 +1 的偶数个节点故障容忍率要高

为了避免资源浪费，官方建议，galera 集群建议节点数应为奇数个。并且，三个节点是 galera 最小安全配置。

galera集群扩容

有当前集群信息：

MariaDB [test]> SHOW STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name      | Value | +--------------------+-------+ | wsrep_cluster_size | 3     | +--------------------+-------+ 1 row in set (0.001 sec) 

新增一个 mariadb 节点，从三节点集群扩容至四节点集群。配置：

[mysqld] bind-address=18.242.143.33      # 监听地址 log_bin=mysql-bin binlog_format=ROW default_storage_engine=InnoDB   # galera只支持InnoDB引擎 innodb_autoinc_lock_mode=2  wsrep_on=ON                                                     # 启用galare插件 wsrep_cluster_name="galera-cluster"                             # galera集群名称 wsrep_cluster_address="gcomm://mariadb-1,mariadb-2,mariadb-3"   # galera集群地址列表 wsrep_provider=/usr/lib64/galera/libgalera_smm.so  wsrep_node_name="mariadb-4"             # 当前节点名称 wsrep_node_address="18.242.143.33"      # 当前节点通信地址  wsrep_sst_method=mariabackup            # sst(State Snapshot Transfer)同步默认方法。不锁表，写入不阻塞 wsrep_sst_auth="sstuser:nuage#2345"     # sst认证凭据  wsrep_slave_threads=4 max_connections=10000 

在新增节点执行：systemctl start mariadb 即可。摘取新增节点的关键日志信息：

 # 连接到集群中的其它节点 2025-08-12 14:25:25 0 [Note] WSREP: (21b01b3d-a887, 'tcp://0.0.0.0:4567') connection established to 3afca15d-bd74 tcp://18.242.143.31:4567 2025-08-12 14:25:25 0 [Note] WSREP: (21b01b3d-a887, 'tcp://0.0.0.0:4567') connection established to 3cfa0384-b2c4 tcp://18.242.143.30:4567 2025-08-12 14:25:25 0 [Note] WSREP: (21b01b3d-a887, 'tcp://0.0.0.0:4567') connection established to 134e4245-aba6 tcp://18.242.143.32:4567  # 当前集群状态为 Primary，满足多数派规则 2025-08-12 14:25:26 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 4  # SST同步成功。SST过程开销 14s 2025-08-12 14:25:26 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> ec75dcca-711c-11f0-9611-ef90f88adb5e:918 WSREP_SST: [INFO] Proceeding with SST (20250812 14:25:27.481) 2025-08-12 14:25:39 0 [Note] WSREP: 0.0 (mariadb-3): State transfer to 1.0 (mariadb-4) complete. 2025-08-12 14:25:40 3 [Note] WSREP: SST received  # IST补充未同步事务 2025-08-12 14:25:40 0 [Note] WSREP: ####### IST current seqno initialized to 874 2025-08-12 14:25:40 0 [Note] WSREP: Receiving IST... 0.0% (0/45 events) complete. 2025-08-12 14:25:40 2 [Note] WSREP: IST received: ec75dcca-711c-11f0-9611-ef90f88adb5e:918  # 集群加入成功，新加入节点后续可以参与多数派投票和接受读写 025-08-12 14:25:40 2 [Note] WSREP: Server status change joined -> synced 

新节点成功加入集群后，集群扩容成功

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name      | Value | +--------------------+-------+ | wsrep_cluster_size | 4     | +--------------------+-------+ 1 row in set (0.002 sec) 

mariadb节点正常退出

有当前 galera 集群信息：

MariaDB [test]> SHOW STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name      | Value | +--------------------+-------+ | wsrep_cluster_size | 3     | +--------------------+-------+ 1 row in set (0.002 sec)  # 任意节点的集群状态都是 Primary，代表可读可写 MariaDB [test]> SHOW STATUS LIKE 'wsrep_cluster_status'; +----------------------+---------+ | Variable_name        | Value   | +----------------------+---------+ | wsrep_cluster_status | Primary | +----------------------+---------+ 1 row in set (0.002 sec) 

任意节点通过：systemctl stop mariadb，逐步停止。

• 停止 18.242.143.30 节点

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name      | Value | +--------------------+-------+ | wsrep_cluster_size | 2     | +--------------------+-------+ 1 row in set (0.001 sec)  MariaDB [(none)]> show status like 'wsrep_cluster_status'; +----------------------+---------+ | Variable_name        | Value   | +----------------------+---------+ | wsrep_cluster_status | Primary | +----------------------+---------+ 1 row in set (0.001 sec) 

• 停止 18.242.143.31 节点

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name      | Value | +--------------------+-------+ | wsrep_cluster_size | 1     | +--------------------+-------+ 1 row in set (0.001 sec)  MariaDB [(none)]> MariaDB [(none)]> show status like 'wsrep_cluster_status'; +----------------------+---------+ | Variable_name        | Value   | +----------------------+---------+ | wsrep_cluster_status | Primary | +----------------------+---------+ 1 row in set (0.001 sec) 

节点逐步协商退出，集群的规模逐步减小，直至剩余最后一个节点。因为集群规模也退化为1个节点，也满足 galera 集群多数派运行规则（因为集群规模也缩减至1），所以可以继续对外提供读写服务。

mariadb节点异常退出

3节点集群，异常退出两个 mariadb 节点，得到如下集群状态：

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name      | Value | +--------------------+-------+ | wsrep_cluster_size | 1     | +--------------------+-------+ 1 row in set (0.002 sec)  MariaDB [(none)]> MariaDB [(none)]> show status like 'wsrep_cluster_status'; +----------------------+-------------+ | Variable_name        | Value       | +----------------------+-------------+ | wsrep_cluster_status | non-Primary | +----------------------+-------------+ 1 row in set (0.002 sec) 

galera节点异常退出，导致最后的节点认为集群规模还是3节点，按照 galera 集群多数派运行规则：

• 3节点集群至少需要两个节点正常运行
• 1个节点不再满足大多数（>50%）

所以剩余的唯一的 galera 节点被禁止写，只可读。

galera集群整体停机后的重新恢复

先看文件：/var/lib/mysql/grastate.dat

[root@mariadb-1 ~]# cat /var/lib/mysql/grastate.dat # GALERA saved state version: 2.1 uuid:    ec75dcca-711c-11f0-9611-ef90f88adb5e   # 集群的全局唯一标识符（​​Cluster UUID​​） seqno:   -1                                     # 节点最后提交的事务序列号（​​Sequence Number​​） safe_to_bootstrap: 0                            # 标记节点是否可以​安全引导集群​​（即作为第一个启动节点）。 0：​​禁止​​自动引导；1：​​允许​​引导 

galera整体停机分为两种情况，一种是所有节点按序退出；另一种是所有节点异常宕机。

所有节点按序退出的集群恢复

所有节点按序退出，seqno 会被每个节点更新为最后的事务ID，safe_to_bootstrap 会被最后退出的节点置为 1。正常退出的集群恢复：

• 找到 safe_to_bootstrap 为 1 的节点，执行：galera_new_cluster
• 如果找不到 safe_to_bootstrap 为 1 的节点，找到 seqno 最大的节点，将该节点的 safe_to_bootstrap 置为 1，再执行：galera_new_cluster
• 其它节点执行：systemctl start mariadb 即可

所有节点异常宕机的集群恢复

galera集群所有节点异常宕机，所有节点的 grastate.dat 文件都不会被更新，将出现无引导恢复节点的情形。如果直接通过重启服务：systemctl restart mariadb ，任意节点执行重启，最后都会报错，因为节点启动时会先去连接其它 galera 节点，但此时无一节点存活，启动失败。

2025-08-12 17:34:47 0 [Note] WSREP: gcomm: connecting to group 'galera-cluster', peer 'mariadb-1:,mariadb-2:,mariadb-3:' 2025-08-12 17:34:47 0 [Note] WSREP: (d51c9e68-9e44, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://18.242.143.30:4567 2025-08-12 17:34:47 0 [Note] WSREP: Failed to establish connection: Connection refused 2025-08-12 17:34:47 0 [Note] WSREP: Failed to establish connection: Connection refused 

如果在任意节点修改 grastate.dat 的 safe_to_bootstrap = 1，再执行 galera_new_cluster，存在数据丢失的风险。

因此，这种场景的恢复，需要人工干预：

• 每个 mariabd 节点都需要执行一次：mariadbd --wsrep_recover --user=mysql，目的是得到当前节点停机前的最后一个事务ID。在 mariadb log 文件查看命令行信息
  • 其中一个异常停机节点有信息：

  2025-08-20 11:38:01 0 [Note] Server socket created on IP: '18.242.143.30'. 2025-08-20 11:38:01 0 [Note] WSREP: Recovered position: 80d99892-7d76-11f0-9233-6b6b37ab0652:4 

  • 另一个异常停机节点有信息：

  2025-08-20 13:44:13 0 [Note] Server socket created on IP: '18.242.143.32'. 2025-08-20 13:44:13 0 [Note] WSREP: Recovered position: 80d99892-7d76-11f0-9233-6b6b37ab0652:6 

• 将ID最大节点的 grastate.dat 的 safe_to_bootstrap 设置为 1
• 在该节点执行：galera_new_cluster
• 主节点启动完毕，其它节点依次执行：systemctl start mariadb

当然，也可以通过脚本包装以上命令，实现自动恢复。

galera集群在openstack中的应用

什么是haproxy

​HAProxy​​（High Availability Proxy）是一款开源的高性能 TCP/HTTP 负载均衡器​。

特性	说明
高性能​	单进程，事件驱动（I/O多路复用），支持每秒数十万请求
多协议支持​	TCP（四层）、HTTP（七层）、SSL/TLS、WebSocket、gRPC
​​健康检查​	主动监测后端服务器状态，自动剔除故障节点
负载均衡算法​	轮询（Round Robin）、最小连接（Least Connections）、源 IP 哈希等

kolla haproxy-galera环境节点信息

节点	说明
18.242.143.10	control-1
18.242.143.11	control-2
18.242.143.12	control-3
18.242.143.100	floating ip, control-3

kolla haproxy-galera框架

kolla 部署的 openstack，haproxy 默认 enable，表示将使用 haproxy 代理并负载均衡各种 openstack 应用请求。例如有如下配置：

• haproxy 对 neutron-server 的代理配置：cat /etc/kolla/haproxy/services.d/neutron-server.cfg 。haproxy 对 openstack 核心组件都是使用 http 代理模式，应用请求默认使用轮询分发到各节点

frontend neutron_server_front     mode http     http-request del-header X-Forwarded-Proto     option httplog     option forwardfor     http-request set-header X-Forwarded-Proto https if { ssl_fc }     bind 18.242.143.100:9696  # floating IP     default_backend neutron_server_back  backend neutron_server_back     mode http     server control-60 18.242.143.10:9696 check inter 2000 rise 2 fall 5     server control-61 18.242.143.11:9696 check inter 2000 rise 2 fall 5     server control-62 18.242.143.12:9696 check inter 2000 rise 2 fall 5 

• neutron-server组件本身有关数据库访问的配置

[database] connection = mysql+pymysql://neutron:aYNsc0NKcGKDvhyYEQiiibGp18r6YhJRl9722sSv@18.242.143.100:3306/neutron   # 访问的是 18.242.143.100:3306 connection_recycle_time = 10 max_pool_size = 1 max_retries = -1 

• haproxy 对 mariabd 的代理配置
  • haproxy 不支持 sql 解析
  • haproxy 对 mariadb 代理使用tcp模式
  • kolla openstack，haproxy 将其它 mariadb 节点作为备份节点，所有应用请求均代理到主 mariadb 节点。一主多备模式，无负载均衡
  • kolla openstack，haproxy 检查的端口是：4569，不是 mariadb 默认服务端口3306

frontend mariadb_front     mode tcp     option clitcpka     timeout client 3600s     option tcplog     bind 18.242.143.100:3306    # 监听 vip：18.242.143.100     default_backend mariadb_back  backend mariadb_back     mode tcp                # TCP 代理     option srvtcpka     timeout server 3600s    # 后端空闲3600s后断开连接     option httpchk          # http检查     server control-1 18.242.143.10:3306 check port 4569 inter 2000 rise 2 fall 5            # 被haproxy标记为master。每 2 秒检查一次，2 次成功标记为健康，5 次失败标记为宕机     server control-2 18.242.143.11:3306 check port 4569 inter 2000 rise 2 fall 5 backup     # 被haproxy标记为slave。每 2 秒检查一次，2 次成功标记为健康，5 次失败标记为宕机     server control-3 18.242.143.12:3306 check port 4569 inter 2000 rise 2 fall 5 backup     # 被haproxy标记为slave。每 2 秒检查一次，2 次成功标记为健康，5 次失败标记为宕机 

$ netstat -ntpl | grep 3306 tcp        0      0 18.242.143.100:3306     0.0.0.0:*               LISTEN      513998/haproxy tcp        0      0 18.242.143.10:3306      0.0.0.0:*               LISTEN      3883762/mariadbd 

• 举例说明客户端通过 horizen 发起 vm 实例查询。为了方便展示，图中的两个 haproxy 其实为同一个进程。
  • 客户端通过访问 node-1 的 floating IP 打开 dashboard
  • horizen 向 nova 发起查询，查询指向主节点的 haproxy 代理，haproxy 通过轮询分配了 node-2 的 nova
  • node-2 的 nova 请求网络信息，向 neutron 发起查询请求，请求指向主节点的 haproxy 代理，haproxy 通过轮询分配了 node-3 的 neutron
  • node-3 的 neutron 向 galera 集群发起查询请求，请求指向主节点的 haproxy 代理，haproxy 根据配置，指定向 galera 的 mariadb 主节点发起查询请求
  • node-1 的 mariadb 响应查询

haproxy高可用

haproxy对mariadb的健康检查

haproxy 通过与 mariadb_clustercheck 容器监听的 4569 端口交互，实现对 mariadb 节点的健康检查。

# mariadb监听的是3306和4567端口 $ netstat -ntpl | grep mariadb tcp        0      0 18.242.143.10:4567      0.0.0.0:*               LISTEN      1136220/mariadbd tcp        0      0 18.242.143.10:3306      0.0.0.0:*               LISTEN      1136220/mariadbd $ # kolla部署了mariadb和mariadb_clustercheck容器 $ docker ps | grep mariadb c0f9b3e0876a   quay.io/openstack.kolla/mariadb-server:2024.1-rocky-9              "dumb-init -- kolla_…"   3 days ago    Up 3 days (healthy)                mariadb 8c038cb7f60e   quay.io/openstack.kolla/mariadb-clustercheck:2024.1-rocky-9        "dumb-init --single-…"   5 weeks ago   Up 3 days                          mariadb_clustercheck $ $ netstat -ntpl | grep 4569 tcp        0      0 18.242.143.10:4569      0.0.0.0:*               LISTEN      1136238/socat $ # 端口4569是 mariadb_clustercheck 监听的 $ docker exec -it mariadb_clustercheck lsof -i :4569 -P COMMAND PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME socat     7 root    6u  IPv4 183999368      0t0  TCP control-1:4569 (LISTEN) $ 

synced 状态代表该节点可正常处理读写请求，是一个健康的 mariadb 节点。

当某个 mariadb 节点离线，haproxy 得到状态是：not synced，表示 mariadb 节点异常，从而不会再将流量分配到该节点。

$ curl --noproxy "*" -v http://18.242.143.10:4569 *   Trying 18.242.143.10:4569... * Connected to 18.242.143.10 (18.242.143.10) port 4569 (#0) > GET / HTTP/1.1 > Host: 18.242.143.10:4569 > User-Agent: curl/7.76.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Content-Type: text/plain < Connection: close < Content-Length: 32 < Galera cluster node is not synced. 

haproxy 也提供了 web 展示接口。galera 集群健康时，haproxy 有状态如图：

有 mariadb backup 节点离线时，haproxy 有状态如图：

如果是 mariadb master 节点离线，haproxy 会从 backup 中选择一个作为新 master。

keepalived保障haproxy高可用

keepalived 的配置：

$ cat keepalived/keepalived.conf vrrp_script check_alive {     script "/check_alive.sh"     interval 2     fall 2     rise 10 }  vrrp_instance kolla_internal_vip_51 {     state BACKUP     nopreempt     interface ens3     virtual_router_id 51     priority 1     advert_int 1     virtual_ipaddress {         18.242.143.100 dev ens3     }     authentication {         auth_type PASS         auth_pass gc7dzrgn3kjpRLLVK78aqcgR6aJNeakZzLajGPIz     }     track_script {         check_alive     } } 

check_alive.sh 将 /checks 目录下发现的 check* 脚本循环执行一遍。check_alive_haproxy.sh 通过 socat 检查 haproxy 状态。其中：/var/lib/kolla/haproxy/haproxy.sock 是 haproxy 监听的 unix_socket，用于返回 haproxy 的状态信息。

$ cat keepalived/checks/check_alive_haproxy.sh #!/bin/bash  # This will return 0 when it successfully talks to the haproxy daemon via the socket # Failures return 1  echo "show info" | socat unix-connect:/var/lib/kolla/haproxy/haproxy.sock stdio > /dev/null 

当浮动IP所在节点的 haproxy 离线时，浮动IP切换到集群其它节点，不影响针对 galera 集群的查询

haproxy的不足

haproxy 可以对 galera 集群实现流量负载均衡，但是：

• 一主多备模式，读写流量都在主节点。主节点可能负载过大
• 配置多主模式，读写流量按负载算法分配。多节点并发写，可能会造成 galera 集群写冲突

所以，为了避免冲突，kolla 默认将 haproxy 配置为一主多备模式代理 galera 集群。

haproxy 代理的是四层流量，无法实现 sql 层次的解析。如果想要减轻主节点的读负担，实现读水平扩展，同时解决多节点写的冲突问题，需要使用 sql 级别的负载均衡方案。

proxysql代理的galera集群

什么是proxysql

ProxySQL 是一个高性能的 MySQL/MariaDB 中间层代理，实现 SQL 流量的转发、读写分离和负载均衡。如下是一个 mariadb 主从模式集群的 sql 流量读写分离示意图。

kolla 支持 proxysql 部署，但是 proxysql 的配置较为复杂。

kolla proxysql-galera环境节点信息

kolla proxysql-galera框架

proxysql-galera的框架配置

kolla 部署的 openstack，proxysql 默认 disable。打开 proxysql，将有配置如下：

• neutron-server组件本身有关数据库访问的配置

[database] connection = mysql+pymysql://neutron:aYNsc0NKcGKDvhyYEQiiibGp18r6YhJRl9722sSv@18.242.143.69:3306/neutron connection_recycle_time = 10 max_pool_size = 1 max_retries = -1 

3306端口不再是 haproxy 代理，而是 proxysql 代理。

$ netstat -ntpl | grep 3306 tcp        0      0 18.242.143.62:3306      0.0.0.0:*               LISTEN      267014/mariadbd tcp        0      0 18.242.143.69:3306      0.0.0.0:*               LISTEN      3809645/proxysql 

• proxysql 代理用户配置

  • 每个需要访问数据库的 openstack 组件都是一个用户

  $ cat proxysql/users/neutron.yaml mysql_users:   - username: "neutron"     password: "aYNsc0NKcGKDvhyYEQiiibGp18r6YhJRl9722sSv"  # 与neutron-server组件配置的数据库密码一致     transaction_persistent: 1     active: 1  $ cat proxysql/users/cinder.yaml mysql_users:   - username: "cinder"     password: "12Nwlivw0rV7QmJPoOXKdwI5gC66B3c7vKuB6poH"  # 与cinder组件配置的数据库密码一致     transaction_persistent: 1     active: 1 ...... ...... ...... 

  • 只有被配置的用户才有权限通过 proxysql 访问后端数据库，否则会被拒绝。例如:

  Database error: (1045, "ProxySQL Error: Access denied for user 'root') 

• 用户配置

  $ ll -lh proxysql/users/ -rw-rw----. 1 root root 719 Jul 23 10:09 cinder.yaml -rw-rw----. 1 root root 719 Jul 23 10:09 glance.yaml -rw-rw----. 1 root root 720 Jul 23 10:09 gnocchi.yaml -rw-rw----. 1 root root 717 Jul 23 10:10 heat.yaml -rw-rw----. 1 root root 720 Jul 23 10:10 horizon.yaml -rw-rw----. 1 root root 721 Jul 23 10:10 keystone.yaml -rw-rw----. 1 root root 719 Jul 23 10:10 manila.yaml -rw-rw----. 1 root root 750 Jul 23 10:10 mariadb.yaml -rw-rw----. 1 root root 720 Jul 23 10:11 neutron.yaml -rw-rw----. 1 root root 717 Jul 23 10:11 nova-cell.yaml -rw-rw----. 1 root root 843 Jul 23 10:11 nova.yaml -rw-rw----. 1 root root 722 Jul 23 10:11 placement.yaml 

在 proxysql 生效的用户配置：

MySQL [(none)]> SELECT * FROM mysql_users; +--------------+------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------+ | username     | password                                 | active | use_ssl | default_hostgroup | default_schema | schema_locked | transaction_persistent | fast_forward | backend | frontend | max_connections | attributes | comment | +--------------+------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------+ | cinder       | 12Nwlivw0rV7QmJPoOXKdwI5gC66B3c7vKuB6poH | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | glance       | XNYstM1eU6drqnDU0tfTK5kObT6njjrISzrD6vM4 | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | gnocchi      | 4dbCpgaw25qfyPbNgPP1n3eNx3U0k0XjvcpFFQ46 | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | heat         | G0C78zEMgL2aDJZAqghAilNyN8UCibJa2xfzH8Cx | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | horizon      | nVggeoWMlN94QKFpEqWKY6dHG11pE9uWTwUwBLY0 | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | keystone     | wtwjQKmVrsRiUpOub17idW6vWiuWGPR5ghu0QYWI | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | manila       | Cqb01BzrtDftsz9tScztXGN56ksTleBdSZr6ckBn | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | root_shard_0 | Msc7WhEeCLQSAstDV4IbfV8644TffapkRrcRbJ9V | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | neutron      | aYNsc0NKcGKDvhyYEQiiibGp18r6YhJRl9722sSv | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | placement    | kkboyBeKF3mMUCtB8Iv5rwdsvg3ANAAq8Ah165Ep | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | nova         | db8uFbJtZ8QUODB1l6bV2JWUHpELHi2RPLfUuuTe | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | | nova_api     | o8zpPMvglwzFkCEsZ8Q7qVXr1WSVCoTqHxg1uvzo | 1      | 0       | 0                 |                | 0             | 1                      | 0            | 1       | 1        | 10000           |            |         | +--------------+------------------------------------------+--------+---------+-------------------+----------------+---------------+------------------------+--------------+---------+----------+-----------------+------------+---------+ 12 rows in set (0.001 sec) 

• proxysql.yaml 主体配置
  • hostgroup 0 定义为写组；hostgroup 1 定义为备写组；hostgroup 2 定义为读组；hostgroup 3 定义为离线组
  • writer_is_also_reader: 0，表示写节点不支持读

$ cat proxysql/proxysql.yaml datadir: "/var/lib/proxysql" errorlog: "/var/log/kolla/proxysql/proxysql.log"  admin_variables:    admin_credentials: "kolla-admin:Do65z48HMsERpGWCec1Iv3Wpwjq4jfNR35l0HTAr"    mysql_ifaces: "18.242.143.61:6032;18.242.143.69:6032;/var/lib/kolla/proxysql/admin.sock"    stats_credentials: "kolla-stats:KtLqNzv1Rrsahz0XIdxxbSOjzLdH62soihO5T86i"  mysql_variables:    threads: 5    max_connections: 40000    interfaces: "18.242.143.69:3306"     # proxysql 监听地址    monitor_username: "monitor"    monitor_password: "ITlG70FfcqE14RFjj14kv0PfoW5UfDguiuysncfK"  mysql_servers:   - address: "18.242.143.62"     port :  3306     hostgroup :  0              # write group     max_connections: 10000     max_replication_lag: 0     weight :  100   - address: "18.242.143.60"     port :  3306     hostgroup :  1              # backup write group     max_connections: 10000     max_replication_lag: 0     weight :  100   - address: "18.242.143.61"     port :  3306     hostgroup :  1              # backup write group     max_connections: 10000     max_replication_lag: 0     weight :  100   - address: "18.242.143.60"     port :  3306     hostgroup :  2              # read group     max_connections: 10000     max_replication_lag: 0     weight :  100   - address: "18.242.143.61"     port :  3306     hostgroup :  2              # read group     max_connections: 10000     max_replication_lag: 0     weight :  100  mysql_galera_hostgroups:   - writer_hostgroup: 0         # write group ID     backup_writer_hostgroup: 1  # baciup write group ID     reader_hostgroup: 2         # read group ID     offline_hostgroup: 3        # offline group ID     max_connections: 10000     max_writers: 1     writer_is_also_reader: 0    # read group     comment: "Galera cluster for shard 0" 

• proxysql 规则配置

$ cat proxysql/rules.bak/neutron.yaml mysql_query_rules:   - schemaname: "neutron"     destination_hostgroup: 0     apply: 1     active: 1  $ cat proxysql/rules.bak/cinder.yaml mysql_query_rules:   - schemaname: "cinder"     destination_hostgroup: 0     apply: 1     active: 1  ...... ...... ...... 

proxysql-galera默认配置的工作机制

kolla 默认配置会将所有组件的 读/写 sql 流量都分发到 hostgroup: 0 的写组单节点。这一点与 haproxy 没有本质区别，都是将流量分发到了 galera 集群主节点。

• proxysql 节点运行信息
  • 只有 mariadb 节点 18.242.143.62 在写组，且状态是：ONLINE
  • 状态 SHUNNED 表示不适合接收请求。

MySQL [(none)]> SELECT * FROM mysql_servers; +--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | hostgroup_id | hostname      | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | +--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | 0            | 18.242.143.62 | 3306 | 0         | ONLINE | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 1            | 18.242.143.60 | 3306 | 0         | ONLINE | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 1            | 18.242.143.61 | 3306 | 0         | ONLINE | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 2            | 18.242.143.60 | 3306 | 0         | ONLINE | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 2            | 18.242.143.61 | 3306 | 0         | ONLINE | 100    | 0           | 10000           | 0                   | 0       | 0              |         | +--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ 5 rows in set (0.002 sec)  MySQL [(none)]> MySQL [(none)]> SELECT * FROM runtime_mysql_servers; +--------------+---------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | hostgroup_id | hostname      | port | gtid_port | status  | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | +--------------+---------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ | 0            | 18.242.143.60 | 3306 | 0         | SHUNNED | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 0            | 18.242.143.61 | 3306 | 0         | SHUNNED | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 0            | 18.242.143.62 | 3306 | 0         | ONLINE  | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 1            | 18.242.143.60 | 3306 | 0         | ONLINE  | 100    | 0           | 10000           | 0                   | 0       | 0              |         | | 1            | 18.242.143.61 | 3306 | 0         | ONLINE  | 100    | 0           | 10000           | 0                   | 0       | 0              |         | +--------------+---------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+ 5 rows in set (0.003 sec) 

• proxysql 的运行规则
  • match_digest(摘要匹配，忽略参数值)，match_pattern(完整文本匹配，包括参数) 都为 NULL
  • destination_hostgroup 都为 0，表示将所有schemaname的请求，都代理至写组，即 online 的 18.242.143.62 节点

MySQL [(none)]> SELECT rule_id, schemaname, match_digest, match_pattern, destination_hostgroup, apply, active, comment FROM runtime_mysql_query_rules; +---------+------------+--------------+---------------+-----------------------+-------+--------+---------+ | rule_id | schemaname | match_digest | match_pattern | destination_hostgroup | apply | active | comment | +---------+------------+--------------+---------------+-----------------------+-------+--------+---------+ | 1       | gnocchi    | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 2       | heat       | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 3       | nova       | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 4       | nova_cell0 | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 5       | nova_api   | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 6       | horizon    | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 7       | keystone   | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 8       | neutron    | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 9       | cinder     | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 10      | glance     | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 11      | manila     | NULL         | NULL          | 0                     | 1     | 1      | NULL    | | 12      | placement  | NULL         | NULL          | 0                     | 1     | 1      | NULL    | +---------+------------+--------------+---------------+-----------------------+-------+--------+---------+ 12 rows in set (0.002 sec) 

• sql路由确认

所有sql语句（无论是读还是写）都被路由到了 hostgroup = 0 的节点：18.242.143.62

MySQL [(none)]> SELECT hostgroup, username, count_star, SUBSTRING(digest_text, 1, 180) AS digest_text_short FROM stats_mysql_query_digest; +-----------+-----------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | hostgroup | username  | count_star | digest_text_short                                                                                                                                                                    | +-----------+-----------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 0         | manila    | 8          | UPDATE reservations SET updated_at=reservations.updated_at,deleted_at=?,deleted=reservations.id WHERE reservations.deleted = ? AND reservations.expire < ?                           | | 0         | placement | 16         | SELECT rp.id,rp.uuid,rp.name,rp.generation,rp.updated_at,rp.created_at,root_rp.uuid AS root_provider_uuid,parent_rp.uuid AS parent_provider_uuid FROM resource_providers AS rp INNER | | 0         | placement | 16         | SELECT me.id,me.uuid,parent.id AS parent_id,parent.uuid AS parent_uuid,root.id AS root_id,root.uuid AS root_uuid FROM resource_providers AS me INNER JOIN resource_providers AS root | | 0         | nova      | 16         | DELETE FROM console_auth_tokens WHERE console_auth_tokens.expires <= ?                                                                                                               | | 0         | placement | 16         | SELECT placement_aggregates.id,placement_aggregates.uuid FROM placement_aggregates INNER JOIN resource_provider_aggregates ON placement_aggregates.id = resource_provider_aggregates |                                                                                                                                                                           |                                                                                                                                                                          | | 0         | neutron   | 796        | SELECT agents.id AS agents_id,agents.agent_type AS agents_agent_type,agents.`binary` AS agents_binary,agents.topic AS agents_topic,agents.host AS agents_host,agents.availability_zo | | 0         | neutron   | 130        | SELECT agents.id AS agents_id,agents.agent_type AS agents_agent_type,agents.`binary` AS agents_binary,agents.topic AS agents_topic,agents.host AS agents_host,agents.availability_zo | | 0         | neutron   | 130        | SELECT networkdhcpagentbindings.network_id AS networkdhcpagentbindings_network_id,networkdhcpagentbindings.dhcp_agent_id AS networkdhcpagentbindings_dhcp_agent_id,networkdhcpagentb |                                                                                                                                                       | | 0         | neutron   | 54         | SELECT subnetroutes.destination AS subnetroutes_destination,subnetroutes.nexthop AS subnetroutes_nexthop,subnetroutes.subnet_id AS subnetroutes_subnet_id,subnets_1.id AS subnets_1_ | | 0         | neutron   | 54         | SELECT dnsnameservers.address AS dnsnameservers_address,dnsnameservers.subnet_id AS dnsnameservers_subnet_id,dnsnameservers.`order` AS dnsnameservers_order,subnets_1.id AS subnets_ | | 0         | neutron   | 54         | SELECT tags.standard_attr_id AS tags_standard_attr_id,tags.tag AS tags_tag,standardattributes_1.id AS standardattributes_1_id FROM (SELECT networks.id AS networks_id FROM networks  | | 0         | neutron   | 54         | SELECT standardattributes.id AS standardattributes_id,standardattributes.resource_type AS standardattributes_resource_type,standardattributes.description AS standardattributes_desc | | 0         | neutron   | 54         | SELECT subnet_service_types.subnet_id AS subnet_service_types_subnet_id,subnet_service_types.service_type AS subnet_service_types_service_type,subnets_1.id AS subnets_1_id FROM (SE | | 0         | neutron   | 54         | SELECT subnets.project_id AS subnets_project_id,subnets.id AS subnets_id,subnets.name AS subnets_name,subnets.network_id AS subnets_network_id,subnets.segment_id AS subnets_segment | | 0         | neutron   | 54         | SELECT segmenthostmappings.segment_id AS segmenthostmappings_segment_id,segmenthostmappings.host AS segmenthostmappings_host,networksegments_1.id AS networksegments_1_id FROM (SELE | | 0         | neutron   | 54         | SELECT networksegments.id AS networksegments_id,networksegments.network_id AS networksegments_network_id,networksegments.network_type AS networksegments_network_type,networksegment | | 0         | neutron   | 54         | SELECT agents.id AS agents_id,agents.agent_type AS agents_agent_type,agents.`binary` AS agents_binary,agents.topic AS agents_topic,agents.host AS agents_host,agents.availability_zo | | 0         | neutron   | 27         | SELECT standardattributes.id AS standardattributes_id,standardattributes.resource_type AS standardattributes_resource_type,standardattributes.description AS standardattributes_desc | | 0         | neutron   | 27         | SELECT extradhcpopts.id AS extradhcpopts_id,extradhcpopts.port_id AS extradhcpopts_port_id,extradhcpopts.opt_name AS extradhcpopts_opt_name,extradhcpopts.opt_value AS extradhcpopts | | 0         | neutron   | 27         | SELECT ml2_port_binding_levels.port_id AS ml2_port_binding_levels_port_id,ml2_port_binding_levels.host AS ml2_port_binding_levels_host,ml2_port_binding_levels.level AS ml2_port_bin | | 0         | keystone  | 9          | SELECT federated_user.id AS federated_user_id,federated_user.user_id AS federated_user_user_id,federated_user.idp_id AS federated_user_idp_id,federated_user.protocol_id AS federate | | 0         | neutron   | 27         | SELECT subnetroutes.destination AS subnetroutes_destination,subnetroutes.nexthop AS subnetroutes_nexthop,subnetroutes.subnet_id AS subnetroutes_subnet_id,anon_1.subnets_id AS anon_ | | 0         | keystone  | 24         | SELECT project_option.project_id AS project_option_project_id,project_option.option_id AS project_option_option_id,project_option.option_value AS project_option_option_value,anon_1 | | 0         | keystone  | 24         | SELECT project_tag.project_id AS project_tag_project_id,project_tag.name AS project_tag_name,anon_1.project_id AS anon_1_project_id FROM (SELECT project.id AS project_id FROM proje | | 0         | keystone  | 37         | SELECT revocation_event.id AS revocation_event_id,revocation_event.domain_id AS revocation_event_domain_id,revocation_event.project_id AS revocation_event_project_id,revocation_eve | | 0         | keystone  | 175        | SELECT ?  | ...... ...... ...... 

proxysql-galera请求路径

openstack 的 proxysql-galera 环境，各应用对 galera 集群的访问演变为如下图所示：