Prometheus 全方位监控你的K8s

Prometheus 全方位监控你的K8s

Kubernetes 监控实现思路

Pod

kubelet的节点使用cAdvisor提供的metrics接口获取该节点所有Pod和容器相关的性能指标数据。 指标接口：             # 部署Prometheus

prometheus-configmap.yaml                # Prometheus配置文件，主要配置Kubernetes服务发现

prometheus-rules.yaml                        # Prometheus告警规则

grafana.yaml                                      # 可视化展示

alertmanager-configmap.yaml         # 配置文件，配置发件人和收件人

alertmanager-deployment.yaml      # 部署Alertmanager告警组件

在master节点，这里可以通过kube-state-metrics获取集群当中有哪些资源

Kubelet内置了cadvisor通过kubelet暴露出来的

Node exporter以deamonset方式去每个节点部署一个，采集该节点上宿主机上面所有的数据

上面数据采集到了就可以可视化展示，然后编写告警规则去告警

部署普罗米修斯和Grafana展示kubelet内置Cadvisor指标

prometheus-deployment.yaml               # 部署Prometheus

prometheus-configmap.yaml                # Prometheus配置文件，主要配置Kubernetes服务发现

prometheus-rules.yaml                        # Prometheus告警规则

prometheus-configmap.yaml

apiVersion: v1kind: ConfigMapmetadata: name: prometheus-config namespace: ops data: prometheus.yml: | rule_files: - /etc/config/rules/*.rules scrape_configs: - job_name: prometheus static_configs: - targets: - localhost:9090 - job_name: kubernetes-apiservers kubernetes_sd_configs: - role: endpoints relabel_configs: - action: keep regex: default;kubernetes; source_labels: - __meta_kubernetes_namespace - __meta_kubernetes_service_name - __meta_kubernetes_endpoint_port_name scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token - job_name: kubernetes-nodes-kubelet kubernetes_sd_configs: - role: node # 发现集群中的节点 relabel_configs: # 将标签(.*)作为新标签名，原有值不变 - action: labelmap regex: __meta_kubernetes_node_label_(.+) scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token - job_name: kubernetes-nodes-cadvisor kubernetes_sd_configs: - role: node relabel_configs: # 将标签(.*)作为新标签名，原有值不变 - action: labelmap regex: __meta_kubernetes_node_label_(.+) # 实际访问指标接口 - target_label: __metrics_path__ replacement: /metrics/cadvisor scheme: tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token - job_name: kubernetes-service-endpoints kubernetes_sd_configs: - role: endpoints # 从Service列表中的Endpoint发现Pod为目标 relabel_configs: # Service没配置注解prometheus.io/scrape的不采集 - action: keep regex: true source_labels: - __meta_kubernetes_service_annotation_prometheus_io_scrape # 重命名采集目标协议 - action: replace regex: ( source_labels: - __meta_kubernetes_service_annotation_prometheus_io_scheme target_label: __scheme__ # 重命名采集目标指标URL路径 - action: replace regex: (.+) source_labels: - __meta_kubernetes_service_annotation_prometheus_io_path target_label: __metrics_path__ # 重命名采集目标地址 - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 source_labels: - __address__ - __meta_kubernetes_service_annotation_prometheus_io_port target_label: __address__ # 将K8s标签(.*)作为新标签名，原有值不变 - action: labelmap regex: __meta_kubernetes_service_label_(.+) # 生成命名空间标签 - action: replace source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace # 生成Service名称标签 - action: replace source_labels: - __meta_kubernetes_service_name target_label: kubernetes_name - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod # 发现所有Pod为目标 # 重命名采集目标协议 relabel_configs: - action: keep regex: true source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_scrape # 重命名采集目标指标URL路径 - action: replace regex: (.+) source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_path target_label: __metrics_path__ # 重命名采集目标地址 - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 source_labels: - __address__ - __meta_kubernetes_pod_annotation_prometheus_io_port target_label: __address__ # 将K8s标签(.*)作为新标签名，原有值不变 - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # 生成命名空间标签 - action: replace source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace # 生成Service名称标签 - action: replace source_labels: - __meta_kubernetes_pod_name target_label: kubernetes_pod_name alerting: alertmanagers: - static_configs: - targets: ["alertmanager:80"]#这一块是普罗米修斯有权限访问apiserver tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token#默认访问的路径为metrics，这里需要改为 relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __metrics_path__ replacement: /metrics/cadvisor 因为暴露的访问恩地址为这个

kubernetes-nodes-kubelet∶ 获取kubelet暴露的指标，访问地址从Service列表中的Endpoint发现Pod为自标

kubernetes-pods∶ 发现所有Pod为目标

给Pod重新标记标签∶

配置采集pod的默认信息，例如协议，URL，端口给Pod添加一些标签，方便后面对数据的多维度查询

prometheus.io/scrape: 'true'

prometheus-deployment.yaml

apiVersion: apps/v1kind: Deploymentmetadata: name: prometheus namespace: ops labels: k8s-app: prometheusspec: replicas: 1 selector: matchLabels: k8s-app: prometheus template: metadata: labels: k8s-app: prometheus spec: serviceAccountName: prometheus initContainers: - name: "init-chown-data" image: "busybox:latest" imagePullPolicy: "IfNotPresent" command: ["chown", "-R", "65534:65534", "/data"] volumeMounts: - name: prometheus-data mountPath: /data subPath: "" containers: - name: prometheus-server-configmap-reload image: "jimmidyson/configmap-reload:v0.1" imagePullPolicy: "IfNotPresent" args: - --volume-dir=/etc/config - --webhook-url= volumeMounts: - name: config-volume mountPath: /etc/config readOnly: true resources: limits: cpu: 10m memory: 10Mi requests: cpu: 10m memory: 10Mi - name: prometheus-server image: "prom/prometheus:v2.20.0" imagePullPolicy: "IfNotPresent" args: - --config.file=/etc/config/prometheus.yml - --storage.tsdb.path=/data - --web.console.libraries=/etc/prometheus/console_libraries - --web.console.templates=/etc/prometheus/consoles - --web.enable-lifecycle ports: - containerPort: 9090 readinessProbe: path: /-/ready port: 9090 initialDelaySeconds: 30 timeoutSeconds: 30 livenessProbe: path: /-/healthy port: 9090 initialDelaySeconds: 30 timeoutSeconds: 30 resources: limits: cpu: 500m memory: 1500Mi requests: cpu: 200m memory: 1000Mi volumeMounts: - name: config-volume mountPath: /etc/config - name: prometheus-data mountPath: /data subPath: "" - name: prometheus-rules mountPath: /etc/config/rules volumes: - name: config-volume configMap: name: prometheus-config - name: prometheus-rules configMap: name: prometheus-rules - name: prometheus-data persistentVolumeClaim: claimName: prometheus---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: prometheus namespace: opsspec: storageClassName: "managed-nfs-storage" accessModes: - ReadWriteMany resources: requests: storage: 10Gi---apiVersion: v1kind: Servicemetadata: name: prometheus namespace: opsspec: type: NodePort ports: - name: port: 9090 protocol: TCP targetPort: 9090 nodePort: 30090 selector: k8s-app: prometheus---apiVersion: v1kind: ServiceAccountmetadata: name: prometheus namespace: ops---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: prometheusrules: - apiGroups: - "" resources: - nodes - nodes/metrics - services - endpoints - pods verbs: - get - list - watch - apiGroups: - "" resources: - configmaps verbs: - get - nonResourceURLs: - "/metrics" verbs: - get---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: prometheusroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheussubjects:- kind: ServiceAccount name: prometheus namespace: ops

prometheus-rules.yaml

apiVersion: v1kind: ConfigMapmetadata: name: prometheus-rules namespace: opsdata: general.rules: | groups: - name: general.rules rules: - alert: InstanceDown expr: up == 0 for: 1m labels: severity: error annotations: summary: "Instance {{ $labels.instance }} 停止工作" description: "{{ $labels.instance }} job {{ $labels.job }} 已经停止5分钟以上." node.rules: | groups: - name: node.rules rules: - alert: NodeFilesystemUsage expr: | 100 - (node_filesystem_free{fstype=~"ext4|xfs"} / node_filesystem_size{fstype=~"ext4|xfs"} * 100) > 80 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高" description: "{{ $labels.instance }}: {{ $labels.mountpoint }} 分区使用大于80% (当前值: {{ $value }})" - alert: NodeMemoryUsage expr: | 100 - (node_memory_MemFree+node_memory_Cached+node_memory_Buffers) / node_memory_MemTotal * 100 > 80 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} 内存使用率过高" description: "{{ $labels.instance }}内存使用大于80% (当前值: {{ $value }})" - alert: NodeCPUUsage expr: | 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100) > 60 for: 1m labels: severity: warning annotations: summary: "Instance {{ $labels.instance }} CPU使用率过高" description: "{{ $labels.instance }}CPU使用大于60% (当前值: {{ $value }})" - alert: KubeNodeNotReady expr: | kube_node_status_condition{condition="Ready",status="true"} == 0 for: 1m labels: severity: error annotations: message: '{{ $labels.node }} 已经有10多分钟没有准备好了.' pod.rules: | groups: - name: pod.rules rules: - alert: PodCPUUsage expr: | sum(rate(container_cpu_usage_seconds_total{image!=""}[1m]) * 100) by (pod_name, namespace) > 80 for: 5m labels: severity: warning annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod_name }} CPU使用大于80% (当前值: {{ $value }})" - alert: PodMemoryUsage expr: | sum(container_memory_rss{image!=""}) by(pod_name, namespace) / sum(container_spec_memory_limit_bytes{image!=""}) by(pod_name, namespace) * 100 != +inf > 80 for: 5m labels: severity: warning annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod_name }} 内存使用大于80% (当前值: {{ $value }})" - alert: PodNetworkReceive expr: | sum(rate(container_network_receive_bytes_total{image!="",name=~"^k8s_.*"}[5m]) /1000) by (pod_name,namespace) > 30000 for: 5m labels: severity: warning annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod_name }} 入口流量大于30MB/s (当前值: {{ $value }}K/s)" - alert: PodNetworkTransmit expr: | sum(rate(container_network_transmit_bytes_total{image!="",name=~"^k8s_.*"}[5m]) /1000) by (pod_name,namespace) > 30000 for: 5m labels: severity: warning annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod_name }} 出口流量大于30MB/s (当前值: {{ $value }}/K/s)" - alert: PodRestart expr: | sum(changes(kube_pod_container_status_restarts_total[1m])) by (pod,namespace) > 0 for: 1m labels: severity: warning annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod }} Pod重启 (当前值: {{ $value }})" - alert: PodFailed expr: | sum(kube_pod_status_phase{phase="Failed"}) by (pod,namespace) > 0 for: 5s labels: severity: error annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod }} Pod状态Failed (当前值: {{ $value }})" - alert: PodPending expr: | sum(kube_pod_status_phase{phase="Pending"}) by (pod,namespace) > 0 for: 1m labels: severity: error annotations: summary: "命名空间: {{ $labels.namespace }} | Pod名称: {{ $labels.pod }} Pod状态Pending (当前值: {{ $value }})"

grafana.yaml

apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: opsspec: replicas: 1 selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: containers: - name: grafana image: grafana/grafana:7.1.0 ports: - containerPort: 3000 protocol: TCP resources: limits: cpu: 100m memory: 256Mi requests: cpu: 100m memory: 256Mi volumeMounts: - name: grafana-data mountPath: /var/lib/grafana subPath: grafana securityContext: fsGroup: 472 runAsUser: 472 volumes: - name: grafana-data persistentVolumeClaim: claimName: grafana ---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: grafana namespace: opsspec: storageClassName: "managed-nfs-storage" accessModes: - ReadWriteMany resources: requests: storage: 5Gi---apiVersion: v1kind: Servicemetadata: name: grafana namespace: opsspec: type: NodePort ports: - port : 80 targetPort: 3000 nodePort: 30030 selector: app: grafana

上面两张图片可以看到vadvisor数据已经被采集到了，那么使用granfana可以展示了

部署NodePort监控k8s主机资源

nodeport以deamonset的方式部署在每个节点上面

apiVersion: apps/v1 kind: DaemonSetmetadata: name: node-exporter namespace: ops labels: k8s-app: node-exporter spec: selector: matchLabels: k8s-app: node-exporter version: v0.15.2 template: metadata: labels: k8s-app: node-exporter version: v0.15.2 spec: containers: - name: prometheus-node-exporter image: "prom/node-exporter:v0.15.2" imagePullPolicy: "IfNotPresent" args: - --path.procfs=/host/proc - --path.sysfs=/host/sys ports: - name: metrics containerPort: 9100 hostPort: 9100 volumeMounts: - name: proc mountPath: /host/proc readOnly: true - name: sys mountPath: /host/sys readOnly: true resources: limits: cpu: 10m memory: 50Mi requests: cpu: 10m memory: 50Mi hostNetwork: true hostPID: true hostIPC: true volumes: - name: proc hostPath: path: /proc - name: sys hostPath: path: /sys - name: rootfs hostPath: path: / - name: dev hostPath: path: /dev---apiVersion: v1kind: Servicemetadata: name: node-exporter namespace: ops annotations: prometheus.io/scrape: "true"spec: clusterIP: None ports: - name: metrics port: 9100 protocol: TCP targetPort: 9100 selector: k8s-app: node-exporter

Nodeexporter会将宿主机上面的目录挂载到容器当中，能让其读取到宿主机上面的资源

hostNetwork: true hostPID: true hostIPC: true volumes: - name: proc hostPath: path: /proc - name: sys hostPath: path: /sys - name: rootfs hostPath: path: / - name: dev hostPath: path: /dev

可以看到这些目录都统统挂载在容器当中，从这些目录当中获取宿主机的资源指标共享宿主机的网络和ipc,以demonset方式去部署

部署完成之后会有以node开头的指标

granfana去展示

部署Kube-stat-metrics监控Pod资源使用率

Kube-stat-metrics是deployment部署的

apiVersion: apps/v1kind: Deploymentmetadata: name: kube-state-metrics namespace: ops labels: k8s-app: kube-state-metricsspec: selector: matchLabels: k8s-app: kube-state-metrics version: v1.3.0 replicas: 1 template: metadata: labels: k8s-app: kube-state-metrics version: v1.3.0 spec: serviceAccountName: kube-state-metrics containers: - name: kube-state-metrics image: lizhenliang/kube-state-metrics:v1.8.0 ports: - name: containerPort: 8080 - name: telemetry containerPort: 8081 readinessProbe: path: /healthz port: 8080 initialDelaySeconds: 5 timeoutSeconds: 5 - name: addon-resizer image: lizhenliang/addon-resizer:1.8.6 resources: limits: cpu: 100m memory: 30Mi requests: cpu: 100m memory: 30Mi env: - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: config-volume mountPath: /etc/config command: - /pod_nanny - --config-dir=/etc/config - --container=kube-state-metrics - --cpu=100m - --extra-cpu=1m - --memory=100Mi - --extra-memory=2Mi - --threshold=5 - --deployment=kube-state-metrics volumes: - name: config-volume configMap: name: kube-state-metrics-config---apiVersion: v1kind: ConfigMapmetadata: name: kube-state-metrics-config namespace: opsdata: NannyConfiguration: |- apiVersion: nannyconfig/v1alpha1 kind: NannyConfiguration---apiVersion: v1kind: Servicemetadata: name: kube-state-metrics namespace: ops annotations: prometheus.io/scrape: 'true'spec: ports: - name: port: 8080 targetPort: protocol: TCP - name: telemetry port: 8081 targetPort: telemetry protocol: TCP selector: k8s-app: kube-state-metrics---apiVersion: v1kind: ServiceAccountmetadata: name: kube-state-metrics namespace: ops---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kube-state-metricsrules:- apiGroups: [""] resources: - configmaps - secrets - nodes - pods - services - resourcequotas - replicationcontrollers - limitranges - persistentvolumeclaims - persistentvolumes - namespaces - endpoints verbs: ["list", "watch"]- apiGroups: ["apps"] resources: - statefulsets - daemonsets - deployments - replicasets verbs: ["list", "watch"]- apiGroups: ["batch"] resources: - cronjobs - jobs verbs: ["list", "watch"]- apiGroups: ["autoscaling"] resources: - horizontalpodautoscalers verbs: ["list", "watch"]- apiGroups: ["networking.k8s.io", "extensions"] resources: - ingresses verbs: ["list", "watch"]- apiGroups: ["storage.k8s.io"] resources: - storageclasses verbs: ["list", "watch"]- apiGroups: ["certificates.k8s.io"] resources: - certificatesigningrequests verbs: ["list", "watch"]- apiGroups: ["policy"] resources: - poddisruptionbudgets verbs: ["list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata: name: kube-state-metrics-resizer namespace: opsrules:- apiGroups: [""] resources: - pods verbs: ["get"]- apiGroups: ["extensions","apps"] resources: - deployments resourceNames: ["kube-state-metrics"] verbs: ["get", "update"]---apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBindingmetadata: name: kube-state-metricsroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metricssubjects:- kind: ServiceAccount name: kube-state-metrics namespace: ops---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: name: kube-state-metrics namespace: opsroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kube-state-metrics-resizersubjects:- kind: ServiceAccount name: kube-state-metrics namespace: ops

部署Alter manager实现告警

alertmanager-configmap.yaml

apiVersion: v1kind: ConfigMapmetadata: name: alertmanager-config namespace: opsdata: alertmanager.yml: | global: resolve_timeout: 5m smtp_smarthost: 'smtp.163.com:25' smtp_from: 'baojingtongzhi@163.com' smtp_auth_username: 'baojingtongzhi@163.com' smtp_auth_password: 'NCKBJTSASSXMRQBM' receivers: - name: default-receiver email_configs: - to: "lulei@163.com" route: group_interval: 1m group_wait: 10s receiver: default-receiver repeat_interval: 1m

alertmanager-deployment.yaml

apiVersion: apps/v1kind: Deploymentmetadata: name: alertmanager namespace: opsspec: replicas: 1 selector: matchLabels: k8s-app: alertmanager version: v0.14.0 template: metadata: labels: k8s-app: alertmanager version: v0.14.0 spec: containers: - name: prometheus-alertmanager image: "prom/alertmanager:v0.14.0" imagePullPolicy: "IfNotPresent" args: - --config.file=/etc/config/alertmanager.yml - --storage.path=/data - --web.external-url=/ ports: - containerPort: 9093 readinessProbe: path: /#/status port: 9093 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - name: config-volume mountPath: /etc/config - name: storage-volume mountPath: "/data" subPath: "" resources: limits: cpu: 10m memory: 50Mi requests: cpu: 10m memory: 50Mi - name: prometheus-alertmanager-configmap-reload image: "jimmidyson/configmap-reload:v0.1" imagePullPolicy: "IfNotPresent" args: - --volume-dir=/etc/config - --webhook-url= volumeMounts: - name: config-volume mountPath: /etc/config readOnly: true resources: limits: cpu: 10m memory: 10Mi requests: cpu: 10m memory: 10Mi volumes: - name: config-volume configMap: name: alertmanager-config - name: storage-volume persistentVolumeClaim: claimName: alertmanager---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: alertmanager namespace: opsspec: storageClassName: managed-nfs-storage accessModes: - ReadWriteOnce resources: requests: storage: "2Gi"---apiVersion: v1kind: Servicemetadata: name: alertmanager namespace: ops labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Alertmanager"spec: type: "NodePort" ports: - name: port: 80 protocol: TCP targetPort: 9093 nodePort: 30093 selector: k8s-app: alertmanager

具体的告警规则参考上面普罗米修斯里面定义的告警规则

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。