K8S+nginx+MYSQL+TOMCAT高可用架构企业自建网站

技术分享 1个月前 (05-14) 0 999+

以下是基于 多Master高可用Kubernetes集群 的企业级部署详细步骤，涵盖 Nginx Ingress + MySQL高可用集群 + Tomcat负载均衡 的完整流程：

一、前置条件准备

1. 节点规划

Master节点：3台（高可用控制平面，需奇数台）
Worker节点：至少2台
操作系统：CentOS 7/8 或 Ubuntu 20.04+
网络要求：所有节点间网络互通，禁用防火墙/SELinux

2. 配置主机名及解析

# 所有节点执行 sudo hostnamectl set-hostname master1  # 按实际修改为master1, master2, master3, worker1等 sudo vi /etc/hosts # 添加以下内容（替换实际IP）： 192.168.1.101 master1 192.168.1.102 master2 192.168.1.103 master3 192.168.1.201 worker1 192.168.1.202 worker2

3. 安装依赖工具

# 所有节点执行 sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common  # Ubuntu # 或 sudo yum install -y yum-utils device-mapper-persistent-data lvm2  # CentOS

二、部署高可用Kubernetes集群

1. 安装Docker

# 所有节点执行 curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun sudo systemctl enable docker && sudo systemctl start docker

2. 安装kubeadm/kubelet/kubectl

# 所有节点执行（以Ubuntu为例） sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet=1.28.0-00 kubeadm=1.28.0-00 kubectl=1.28.0-00 sudo apt-mark hold kubelet kubeadm kubectl

3. 初始化第一个Master节点

# 在master1节点执行 sudo kubeadm init    --control-plane-endpoint "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"    --upload-certs    --image-repository registry.aliyuncs.com/google_containers    --kubernetes-version v1.28.0    --service-cidr=10.96.0.0/12    --pod-network-cidr=192.168.0.0/16    --apiserver-advertise-address=192.168.1.101  # 输出中会包含加入其他Master和Worker的命令，保存备用 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

4. 加入其他Master节点

# 在master2和master3执行（使用上一步生成的命令，形如）： sudo kubeadm join LOAD_BALANCER_DNS:LOAD_BALANCER_PORT    --token xxxx    --discovery-token-ca-cert-hash sha256:xxxx    --control-plane    --certificate-key xxxx

5. 加入Worker节点

# 在所有Worker节点执行（使用kubeadm init输出的命令）： sudo kubeadm join LOAD_BALANCER_DNS:LOAD_BALANCER_PORT --token xxxx --discovery-token-ca-cert-hash sha256:xxxx

6. 安装网络插件（Calico）

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

三、配置存储（NFS示例）

1. 部署NFS Server（可选）

# 在存储节点执行（例如192.168.1.250） sudo apt-get install -y nfs-kernel-server  # Ubuntu sudo mkdir -p /data/nfs sudo chmod 777 /data/nfs sudo vi /etc/exports # 添加： /data/nfs *(rw,sync,no_root_squash) sudo exportfs -a sudo systemctl restart nfs-server

2. 部署NFS StorageClass

# 使用Helm安装NFS Provisioner helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ helm install nfs-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner    --set nfs.server=192.168.1.250    --set nfs.path=/data/nfs    --set storageClass.name=nfs-sc  # 验证StorageClass kubectl get storageclass

四、部署高可用MySQL集群

1. 创建Secret存储密码

kubectl create secret generic mysql-secret    --from-literal=root_password=yourpassword    --from-literal=replication_password=replpassword

2. 部署MySQL StatefulSet

# mysql-ha.yaml apiVersion: apps/v1 kind: StatefulSet metadata:   name: mysql spec:   serviceName: mysql   replicas: 3   selector:     matchLabels:       app: mysql   template:     metadata:       labels:         app: mysql     spec:       containers:       - name: mysql         image: mysql:8.0         env:         - name: MYSQL_ROOT_PASSWORD           valueFrom:             secretKeyRef:               name: mysql-secret               key: root_password         - name: MYSQL_REPLICATION_PASSWORD           valueFrom:             secretKeyRef:               name: mysql-secret               key: replication_password         args:         - "--server-id=$(expr $RANDOM % 100 + 1)"         - "--gtid-mode=ON"         - "--enforce-gtid-consistency=ON"         - "--log-bin=mysql-bin"         - "--binlog-format=ROW"         - "--relay-log=mysql-relay"         - "--innodb_flush_log_at_trx_commit=1"         - "--sync_binlog=1"         ports:         - containerPort: 3306         volumeMounts:         - name: mysql-data           mountPath: /var/lib/mysql   volumeClaimTemplates:   - metadata:       name: mysql-data     spec:       accessModes: [ "ReadWriteOnce" ]       storageClassName: "nfs-sc"       resources:         requests:           storage: 20Gi

3. 部署MySQL服务

# mysql-service.yaml apiVersion: v1 kind: Service metadata:   name: mysql spec:   ports:   - port: 3306   clusterIP: None   selector:     app: mysql

kubectl apply -f mysql-ha.yaml kubectl apply -f mysql-service.yaml

五、部署Tomcat应用

1. 创建Tomcat Deployment

# tomcat-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata:   name: tomcat-app spec:   replicas: 3   selector:     matchLabels:       app: tomcat   strategy:     rollingUpdate:       maxSurge: 1       maxUnavailable: 0   template:     metadata:       labels:         app: tomcat     spec:       containers:       - name: tomcat         image: tomcat:9.0-jdk17         ports:         - containerPort: 8080         env:         - name: DATABASE_URL           value: "jdbc:mysql://mysql.default.svc.cluster.local:3306/appdb?useSSL=false"         resources:           requests:             cpu: "100m"             memory: "512Mi"           limits:             cpu: "500m"             memory: "1Gi"         livenessProbe:           httpGet:             path: /             port: 8080           initialDelaySeconds: 30           periodSeconds: 10         readinessProbe:           httpGet:             path: /             port: 8080           initialDelaySeconds: 20           periodSeconds: 5

2. 创建Service

# tomcat-service.yaml apiVersion: v1 kind: Service metadata:   name: tomcat-service spec:   selector:     app: tomcat   ports:     - protocol: TCP       port: 80       targetPort: 8080

kubectl apply -f tomcat-deployment.yaml kubectl apply -f tomcat-service.yaml

六、部署Nginx Ingress Controller

1. 使用Helm安装

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm install ingress-nginx ingress-nginx/ingress-nginx    --set controller.replicaCount=3    --set controller.service.type=LoadBalancer    --set controller.service.externalTrafficPolicy=Local    --set controller.nodeSelector."kubernetes.io/os"=linux    --set controller.admissionWebhooks.patch.nodeSelector."kubernetes.io/os"=linux    --set defaultBackend.nodeSelector."kubernetes.io/os"=linux

2. 配置Ingress路由规则

# ingress-rule.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata:   name: web-ingress   annotations:     nginx.ingress.kubernetes.io/rewrite-target: / spec:   ingressClassName: nginx   rules:   - host: example.com     http:       paths:       - path: /         pathType: Prefix         backend:           service:             name: tomcat-service             port:               number: 80

kubectl apply -f ingress-rule.yaml

七、验证与维护

1. 查看集群状态

kubectl get nodes -o wide kubectl get pods -A -o wide kubectl get svc,pv,pvc

2. 测试数据库连接

kubectl exec -it mysql-0 -- mysql -uroot -p$(kubectl get secret mysql-secret -o jsonpath='{.data.root_password}' | base64 --decode) -e "CREATE DATABASE appdb;"

3. 访问测试

# 获取Ingress外部IP kubectl get svc ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'  # 测试访问（替换实际IP） curl -H "Host: example.com" http://<INGRESS_IP>

4. 配置HPA自动扩缩

# hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata:   name: tomcat-hpa spec:   scaleTargetRef:     apiVersion: apps/v1     kind: Deployment     name: tomcat-app   minReplicas: 2   maxReplicas: 10   metrics:   - type: Resource     resource:       name: cpu       target:         type: Utilization         averageUtilization: 80

八、架构示意图

用户访问 -> 云厂商LB/Nginx Ingress (外部流量)         ↓ K8S Ingress Controller (3副本)        ↓ Tomcat Pods (HPA自动扩缩)         ↓ MySQL Cluster (3节点StatefulSet)         ↓ NFS/Ceph Persistent Volumes