目录
- 一.系统环境
- 二.前言
- 三.Kubernetes 审计简介
- 四.审计策略简介
- 五.启用审计
- 六.审计策略
- 6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志
- 6.2 只记录audit命名空间里的日志
- 6.3 只记录audit命名空间的pods操作日志
- 6.4 只记录audit命名空间的pods,services,deployments操作日志
- 6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse
- 6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录
- 6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了
- 6.8 在 Metadata 级别为所有请求生成日志
- 七.总结
一.系统环境
本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。
| 服务器版本 | docker软件版本 | Kubernetes(k8s)集群版本 | CPU架构 |
|---|---|---|---|
| Ubuntu 18.04.5 LTS | Docker version 20.10.14 | v1.22.2 | x86_64 |
Kubernetes集群架构：k8scludes1作为master节点，k8scludes2，k8scludes3作为worker节点。
| 服务器 | 操作系统版本 | CPU架构 | 进程 | 功能描述 |
|---|---|---|---|---|
| k8scludes1/192.168.110.128 | Ubuntu 18.04.5 LTS | x86_64 | docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico | k8s master节点 |
| k8scludes2/192.168.110.129 | Ubuntu 18.04.5 LTS | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
| k8scludes3/192.168.110.130 | Ubuntu 18.04.5 LTS | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
二.前言
在当今的云计算时代，容器编排技术已成为企业部署和管理应用程序的关键组成部分。Kubernetes 作为最流行的容器编排平台，提供了强大的功能来部署、管理和扩展容器化应用程序。然而，随着应用程序的日益复杂和部署环境的扩大，确保 Kubernetes 集群的安全性变得越来越重要。在这篇博客中，我们将深入了解 Kubernetes 审计的概念，并探讨如何在 Kubernetes 环境中实施审计策略。
使用Kubernetes 审计（Auditing）的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。
三.Kubernetes 审计简介
Kubernetes 审计（Auditing） 功能提供了与安全相关的、按时间顺序排列的记录集， 记录每个用户、使用 Kubernetes API 的应用以及控制面自身引发的活动。
审计功能使得集群管理员能够回答以下问题：
- 发生了什么？
- 什么时候发生的？
- 谁触发的？
- 活动发生在哪个（些）对象上？
- 在哪观察到的？
- 它从哪触发的？
- 活动的后续处理行为是什么？
审计记录最初产生于 kube-apiserver 内部。每个请求在不同执行阶段都会生成审计事件；这些审计事件会根据特定策略被预处理并写入后端。 策略确定要记录的内容和用来存储记录的后端，当前的后端支持日志文件和 webhook。
每个请求都可被记录其相关的阶段（stage），阶段（stage）可以理解为什么时候记录。。已定义的阶段有：
RequestReceived- 此阶段对应审计处理器接收到请求后， 并且在委托给其余处理器之前生成的事件。ResponseStarted- 在响应消息的头部发送后，响应消息体发送前生成的事件。 只有长时间运行的请求（例如 watch）才会生成这个阶段。ResponseComplete- 当响应消息体完成并且没有更多数据需要传输的时候。Panic- 当 panic 发生时生成。
审计日志记录功能会增加 API server 的内存消耗，因为需要为每个请求存储审计所需的某些上下文。 内存消耗取决于审计日志记录的配置。
Kubernetes 审计是一种监控和记录 Kubernetes 集群中资源操作的方法，用于确保集群的安全性和符合性。通过审计，管理员可以跟踪对集群资源的访问和修改，以便在发生安全事件时进行调查和响应。Kubernetes 提供了审计日志记录的框架，允许管理员自定义审计策略，以确定哪些资源操作应该被记录。
四.审计策略简介
审计策略定义了关于应记录哪些事件以及应包含哪些数据的规则。 审计策略对象结构定义在 audit.k8s.io API 组。 处理事件时，将按顺序与规则列表进行比较。第一个匹配规则设置事件的审计级别（Audit Level），审计级别（Audit Level）可以理解为记录什么？ 已定义的审计级别有：
- None - 符合这条规则的日志将不会记录。
- Metadata - 记录请求的元数据（请求的用户、时间戳、资源、动词等等）， 但是不记录请求或者响应的消息体。
- Request - 记录事件的元数据和请求的消息体，但是不记录响应的消息体。 这不适用于非资源类型的请求。
- RequestResponse - 记录事件的元数据，请求和响应的消息体。这不适用于非资源类型的请求。
你可以使用 --audit-policy-file 标志将包含策略的文件传递给 kube-apiserver。 如果不设置该标志，则不记录事件。 注意 rules 字段必须在审计策略文件中提供。没有（0）规则的策略将被视为非法配置。
审计策略定义了哪些资源操作应该被审计以及审计记录的格式。在 Kubernetes 中，审计策略通过 Admission Controllers 实现，可以通过 Webhook 的方式进行集成。审计策略可以根据资源的类型、操作的类型和用户身份等信息进行过滤，以满足不同场景下的审计需求。
五.启用审计
5.1 引入审计
本次使用etcd2机器作为kubernetes集群的客户端，kctom是kubeconfig文件，etcd2机器使用kctom连接kubernetes集群，现在用户tom没有权限，需要先授权。kubeconfig文件在博客《Kubernetes(k8s)访问控制：身份认证》里已经详细介绍过了，这里不再赘述。
[root@etcd2 ~]# ls kc* -lh -rw------- 1 root root 5.5K 5月 5 17:46 kctom [root@etcd2 ~]# kubectl get node --kubeconfig=kctom Error from server (Forbidden): nodes is forbidden: User "tom" cannot list resource "nodes" in API group "" at the cluster scope 去kubernetes集群给tom用户授予cluster-admin权限，也可以授予部分权限。关于授权的详细信息，请查看博客《Kubernetes(k8s)访问控制：权限管理之RBAC授权/鉴权》。
root@k8scludes1:~# kubectl create clusterrolebinding crbindtom --clusterrole=cluster-admin --user=tom clusterrolebinding.rbac.authorization.k8s.io/crbindtom created 授予cluster-admin权限之后，客户端就具有权限了。
[root@etcd2 ~]# kubectl get node --kubeconfig=kctom NAME STATUS ROLES AGE VERSION k8scludes1 Ready control-plane,master 66d v1.22.2 k8scludes2 Ready <none> 65d v1.22.2 k8scludes3 Ready <none> 65d v1.22.2 编辑pod配置文件，表示使用nginx镜像创建pod。
root@k8scludes1:~# vim pod.yaml root@k8scludes1:~# cat pod.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: podtest name: podtest spec: #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。 terminationGracePeriodSeconds: 0 containers: - image: hub.c.163.com/library/nginx:latest #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像 imagePullPolicy: IfNotPresent name: podtest resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} 在admissioncontr命名空间创建一个pod。
root@k8scludes1:~# kubectl apply -f pod.yaml pod/podtest created 在客户端etcd2机器可以看到这个pod。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n admissioncontr NAME READY STATUS RESTARTS AGE podtest 1/1 Running 0 34s 在客户端删除pod。
[root@etcd2 ~]# kubectl delete pod podtest --kubeconfig=kctom -n admissioncontr pod "podtest" deleted kubectl get ev （查看事件），可以看到pod被删除掉了，但是看不到pod被谁删除掉了！
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide LAST SEEN TYPE REASON OBJECT SUBOBJECT SOURCE MESSAGE FIRST SEEN COUNT NAME 3m37s Normal Pulled pod/podtest spec.containers{podtest} kubelet, k8scludes3 Container image "hub.c.163.com/library/nginx:latest" already present on machine 3m37s 1 podtest.16fab43e14342a3f 3m37s Normal Created pod/podtest spec.containers{podtest} kubelet, k8scludes3 Created container podtest 3m37s 1 podtest.16fab43e1db484f4 3m36s Normal Started pod/podtest spec.containers{podtest} kubelet, k8scludes3 Started container podtest 3m36s 1 podtest.16fab43e3315ada7 2m47s Normal Killing pod/podtest spec.containers{podtest} kubelet, k8scludes3 Stopping container podtest 2m47s 1 podtest.16fab449c468f9af 我们的k8s集群会被各种用户连接使用，我们想知道连接k8s集群的各个用户做了什么操作？应该怎么办？通过审计解决！
5.2 启用审计
创建audit目录存放yaml文件。
root@k8scludes1:~# mkdir audit root@k8scludes1:~# cd audit/ root@k8scludes1:~/audit# pwd /root/audit 创建命名空间audit。
root@k8scludes1:~/audit# kubectl create ns audit namespace/audit created 切换到命名空间audit。
root@k8scludes1:~/audit# kubens audit Context "kubernetes-admin@kubernetes" modified. Active namespace is "audit". root@k8scludes1:~/audit# kubectl get pod No resources found in audit namespace. kubernetes默认并没有启用审计，需要启用审计功能。指定审计策略文件的路径：--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml root@k8scludes1:~# grep audit-policy-file /etc/kubernetes/manifests/kube-apiserver.yaml - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml 创建审计策略文件，文件内容可以先为空。
root@k8scludes1:~# mkdir /etc/kubernetes/audit/ root@k8scludes1:~# cd /etc/kubernetes/audit/ root@k8scludes1:/etc/kubernetes/audit# touch audit-policy.yaml root@k8scludes1:/etc/kubernetes/audit# ls /etc/kubernetes/audit/audit-policy.yaml /etc/kubernetes/audit/audit-policy.yaml 指定审计日志存放的路径：--audit-log-path=/var/log/kubernetes/audit/audit.log 。审计日志格式为JSONlines 格式 。
root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml root@k8scludes1:/etc/kubernetes/audit# grep audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml - --audit-log-path=/var/log/kubernetes/audit/audit.log 创建审计日志文件。
root@k8scludes1:/etc/kubernetes/audit# mkdir -p /var/log/kubernetes/audit/ root@k8scludes1:/etc/kubernetes/audit# touch /var/log/kubernetes/audit/audit.log root@k8scludes1:/etc/kubernetes/audit# ls /var/log/kubernetes/audit/audit.log /var/log/kubernetes/audit/audit.log 审计日志参数如下：
- --audit-log-path 指定用来写入审计事件的日志文件路径。不指定此标志会禁用日志后端。- 意味着标准化
- --audit-log-maxage 定义保留旧审计日志文件的最大天数
- --audit-log-maxbackup 定义要保留的审计日志文件的最大数量
- --audit-log-maxsize 定义审计日志文件轮转之前的最大大小（兆字节）
配置审计日志的参数。
root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml root@k8scludes1:/etc/kubernetes/audit# grep audit /etc/kubernetes/manifests/kube-apiserver.yaml - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml - --audit-log-path=/var/log/kubernetes/audit/audit.log - --audit-log-maxage=365 - --audit-log-maxbackup=730 - --audit-log-maxsize=250 如果你的集群控制面以 Pod 的形式运行 kube-apiserver，需要通过 hostPath 数据卷来访问策略文件和日志文件所在的目录，这样审计记录才会持久保存下来。
现在/etc/kubernetes/audit/audit-policy.yaml和/var/log/kubernetes/audit/audit.log在宿主机里是存在的，但是kube-apiserver是以pod的方式运行的，pod里并不存在这两个文件，需要通过hostPath 数据卷进行映射。
/etc/kubernetes/manifests/kube-apiserver.yaml所有的修改如下截图：
注意：/var/log/kubernetes/audit/的readOnly为false，否则不能写入日志！
自此，成功启用审计了，下面开始配置审计策略。
六.审计策略
6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志
先不用重启kubelet，因为审计策略文件还没写好。
现在编写审计策略文件，该审计策略表示什么都不记录。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: None resources: - group: "" 重启kubelet使配置生效。
root@k8scludes1:~# systemctl restart kubelet root@k8scludes1:~# kubectl get node NAME STATUS ROLES AGE VERSION k8scludes1 Ready control-plane,master 66d v1.22.2 k8scludes2 Ready <none> 66d v1.22.2 k8scludes3 Ready <none> 66d v1.22.2 在客户端etcd2机器执行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide。
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide No resources found in admissioncontr namespace. 查看审计日志，没有任何日志记录。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log 审计级别修改为Metadata，表示记录请求的元数据（请求的用户、时间戳、资源、动词等等）， 但是不记录请求或者响应的消息体。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: Metadata resources: - group: "" 重启kubelet使配置生效。
root@k8scludes1:~# systemctl daemon-reload ; systemctl restart kubelet 在客户端etcd2机器执行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide 。
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide No resources found in admissioncontr namespace. 查看审计日志，发现还是没有日志产生，看来重启kubelet还不行，现在重启机器。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log 现在直接重启k8scludes1机器，k8s worker节点不用重启。
root@k8scludes1:~# reboot 重启k8scludes1机器之后，现在/var/log/kubernetes/audit/audit.log里有很多数据，先删除数据，然后客户端执行命令，查看审计日志。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l 3317 清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 客户端执行kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide命令。
[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide No resources found in admissioncontr namespace. 现在不停的产生日志。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l 32 这是最新的日志。
root@k8scludes1:~# tail -1 /var/log/kubernetes/audit/audit.log {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"72736d33-508f-4279-89a5-d95aea7b98c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader","verb":"update","user":{"username":"system:serviceaccount:ingress-nginx:ingress-nginx","uid":"70a36f2c-225f-450d-849e-2432db224f40","groups":["system:serviceaccounts","system:serviceaccounts:ingress-nginx","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["ingress-nginx-controller-684bbc4b45-jmpxk"],"authentication.kubernetes.io/pod-uid":["308fdf30-854b-49f7-b041-4e5d770c0667"]}},"sourceIPs":["192.168.110.130"],"userAgent":"nginx-ingress-controller/v1.0.0 (linux/amd64) ingress-nginx/041eb167c7bfccb1d1653f194924b0c5fd885e10","objectRef":{"resource":"configmaps","namespace":"ingress-nginx","name":"ingress-controller-leader","uid":"db625974-8e33-4bd6-a2e2-1816301af942","apiVersion":"v1","resourceVersion":"2739119"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:03:36.751232Z","stageTimestamp":"2022-06-22T13:03:36.759442Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding "ingress-nginx/ingress-nginx" of Role "ingress-nginx" to ServiceAccount "ingress-nginx/ingress-nginx"","mutation.webhook.admission.k8s.io/round_0_index_1":"{"configuration":"gatekeeper-mutating-webhook-configuration","webhook":"mutation.gatekeeper.sh","mutated":false}"}} 客户端不执行命令就产生了太多审计日志了，需要修改审计策略。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l 1174 6.2 只记录audit命名空间里的日志
修改审计策略，现在配置只记录某个命名空间里的审计日志，namespaces: ["audit"]表示只记录audit命名空间里的日志。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: Metadata resources: - group: "" namespaces: ["audit"] 重启kubelet使其生效，但是审计策略没有生效，重启机器。
root@k8scludes1:~# systemctl restart kubelet 重启k8s master节点。
root@k8scludes1:~# reboot root@k8scludes1:~# kubectl get node NAME STATUS ROLES AGE VERSION k8scludes1 Ready control-plane,master 66d v1.22.2 k8scludes2 Ready <none> 66d v1.22.2 k8scludes3 Ready <none> 66d v1.22.2 重启k8s master节点之后，清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 客户端执行kubectl get pod --kubeconfig=kctom -n default -o wide。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n default -o wide No resources found in default namespace. 客户端执行 kubectl get pod --kubeconfig=kctom -n audit -o wide。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide No resources found in audit namespace. 查看审计日志，只记录了audit命名空间的操作，default命名空间的操作没有记录。
root@k8scludes1:~# tail -2 /var/log/kubernetes/audit/audit.log {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.732751Z"} {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.741665Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "crbindtom" of ClusterRole "cluster-admin" to User "tom""}} 6.3 只记录audit命名空间的pods操作日志
修改审计策略，该审计策略表示只记录audit命名空间的pods操作。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: Metadata resources: - group: "" resources: ["pods"] namespaces: ["audit"] 重启机器使审计策略生效。
root@k8scludes1:~# reboot 重启之后，清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 在客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide No resources found in audit namespace. 在客户端执行 kubectl get svc --kubeconfig=kctom -n audit -o wide 。
[root@etcd2 ~]# kubectl get svc --kubeconfig=kctom -n audit -o wide No resources found in audit namespace. 查看日志，发现只记录了audit命名空间的pod操作，svc操作没有记录。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.351036Z"} {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.434147Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "clusterrolebinding1" of ClusterRole "clusterole1" to User "tom""}} 6.4 只记录audit命名空间的pods,services,deployments操作日志
编辑审计策略文件，表示只记录audit命名空间的pods,services,deployments操作，因为deployments的apiVersion的父级为apps，所以需要group: "apps" 。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: Metadata resources: - group: "" resources: ["pods","services"] - group: "apps" resources: ["deployments"] namespaces: ["audit"] 6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse
编辑审计策略文件，表示只记录audit命名空间的pods操作,审计级别为RequestResponse，记录事件的元数据，请求和响应的消息体。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: RequestResponse resources: - group: "" resources: ["pods"] namespaces: ["audit"] 重启机器使审计策略生效。
root@k8scludes1:~# reboot 创建一个pod。
root@k8scludes1:~# kubectl apply -f pod.yaml pod/podtest created root@k8scludes1:~# kubectl get pod NAME READY STATUS RESTARTS AGE podtest 1/1 Running 0 5s 清空审计日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide 。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES podtest 1/1 Running 0 48s 10.244.1.97 k8scludes3 <none> <none> 查看审计日志，记录的内容很丰富，元数据，请求和响应的消息体都显示出来了。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"d6836f9b-4918-49f2-80dc-ad02e246949b","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.782241Z"} ...... 23T00:42:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{"type":"ContainersReady"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{"type":"Initialized"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{"type":"PodScheduled"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{"type":"Ready"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{"ip":"10.244.1.97"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]}}}]},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.785883Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "clusterrolebinding1" of ClusterRole "clusterole1" to User "tom""}} 6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录
编辑审计策略文件，设置审计策略：只记录audit命名空间下的tom用户的pods操作，其他用户的操作不记录。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: - level: Metadata users: ["tom"] resources: - group: "" resources: ["pods"] namespaces: ["audit"] 重启机器使审计策略生效。
root@k8scludes1:~# reboot 清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 使用管理员用户查看pod。
root@k8scludes1:~# kubectl get pod -n audit NAME READY STATUS RESTARTS AGE podtest 1/1 Running 0 29m 使用tom用户在客户端查看pod 。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES podtest 1/1 Running 0 29m 10.244.1.97 k8scludes3 <none> <none> 查看日志，可以发现只记录了tom用户的pod查询操作。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.103036Z"} {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.107872Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "crbindtom" of ClusterRole "cluster-admin" to User "tom""}} 6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了
编辑审计策略文件，有两条策略，一条是：apiVersion为group: ""的操作不记录日志，另外一条是：只记录audit命名空间下tom用户的pod操作。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "ResponseStarted" rules: #apiVersion为group: ""的操作不记录日志 - level: None resources: - group: "" #只记录audit命名空间下tom用户的pod操作 - level: Metadata users: ["tom"] resources: - group: "" resources: ["pods"] namespaces: ["audit"] 重启机器使配置生效。
root@k8scludes1:~# reboot 清空日志。
root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 使用管理员用户查询audit命名空间的pod。
root@k8scludes1:~# kubectl get pod -n audit NAME READY STATUS RESTARTS AGE podtest 1/1 Running 0 37m 使用tom用户在客户端机器etcd2上查询audit命名空间的pod。
[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n audit -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES podtest 1/1 Running 0 37m 10.244.1.97 k8scludes3 <none> <none> 查看审计日志，可以发现在客户端使用tom用户查询pod和使用管理员用户查看pod都没有生成审计日志，rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了。
root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log 6.8 在 Metadata 级别为所有请求生成日志
编辑审计策略文件，在 Metadata 级别为所有请求生成日志。
root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml apiVersion: audit.k8s.io/v1beta1 kind: Policy rules: - level: Metadata 七.总结
在 Kubernetes 1.22.2 环境中实施审计策略可以帮助管理员监控和记录集群中的资源操作，确保集群的安全性和符合性。通过启用审计 Admission Controller 和配置相应的审计策略，我们可以灵活地控制审计记录的格式和范围。
![洛谷 P11345 [KTSC 2023 R2] 基地简化 题解](http://www.itfaba.com/wp-content/themes/kemi/timthumb.php?src=http://www.itfaba.com/wp-content/themes/kemi/img/random/1.jpg&w=218&h=124&zc=1)
