kubernetes node 相关指标梳理

这一篇我们梳理node相关的指标，话不多说，先上指标。

1.kubelet自身指标梳理

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
gc的时间统计（summary指标）

# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
goroutine 数量

# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
os的线程数量


# HELP kubelet_cgroup_manager_duration_seconds [ALPHA] Duration in seconds for cgroup manager operations. Broken down by method.
# TYPE kubelet_cgroup_manager_duration_seconds histogram
操作cgroup的时长分布，按照操作类型统计

# HELP kubelet_containers_per_pod_count [ALPHA] The number of containers per pod.
# TYPE kubelet_containers_per_pod_count histogram
pod中container数量的统计(spec.containers的数量)


# HELP kubelet_docker_operations_duration_seconds [ALPHA] Latency in seconds of Docker operations. Broken down by operation type.
# TYPE kubelet_docker_operations_duration_seconds histogram
操作docker的时长分布，按照操作类型统计

# HELP kubelet_docker_operations_errors_total [ALPHA] Cumulative number of Docker operation errors by operation type.
# TYPE kubelet_docker_operations_errors_total counter
操作docker的错误累计次数，按照操作类型统计

# HELP kubelet_docker_operations_timeout_total [ALPHA] Cumulative number of Docker operation timeout by operation type.
# TYPE kubelet_docker_operations_timeout_total counter
操作docker的超时统计，按照操作类型统计

# HELP kubelet_docker_operations_total [ALPHA] Cumulative number of Docker operations by operation type.
# TYPE kubelet_docker_operations_total counter
操作docker的累计次数，按照操作类型统计

# HELP kubelet_eviction_stats_age_seconds [ALPHA] Time between when stats are collected, and when pod is evicted based on those stats by eviction signal
# TYPE kubelet_eviction_stats_age_seconds histogram
驱逐操作的时间分布，按照驱逐信号(原因)分类统计

# HELP kubelet_evictions [ALPHA] Cumulative number of pod evictions by eviction signal
# TYPE kubelet_evictions counter
驱逐次数统计，按照驱逐信号（原因）统计

# HELP kubelet_http_inflight_requests [ALPHA] Number of the inflight http requests
# TYPE kubelet_http_inflight_requests gauge
请求kubelet的inflight请求数,按照method path server_type统计
注意与每秒的request数区别开

# HELP kubelet_http_requests_duration_seconds [ALPHA] Duration in seconds to serve http requests
# TYPE kubelet_http_requests_duration_seconds histogram
请求kubelet的请求时间统计,按照method path server_type统计

# HELP kubelet_http_requests_total [ALPHA] Number of the http requests received since the server started
# TYPE kubelet_http_requests_total counter
请求kubelet的请求数统计,按照method path server_type统计

# HELP kubelet_managed_ephemeral_containers [ALPHA] Current number of ephemeral containers in pods managed by this kubelet. Ephemeral containers will be ignored if disabled by the EphemeralContainers feature gate, and this number will be 0.
# TYPE kubelet_managed_ephemeral_containers gauge
当前kubelet管理的临时容器数量

# HELP kubelet_network_plugin_operations_duration_seconds [ALPHA] Latency in seconds of network plugin operations. Broken down by operation type.
# TYPE kubelet_network_plugin_operations_duration_seconds histogram
网络插件的操作耗时分布 ，按照操作类型(operation_type)统计
如果 --feature-gates=EphemeralContainers=false，否则一直为0

# HELP kubelet_network_plugin_operations_errors_total [ALPHA] Cumulative number of network plugin operation errors by operation type.
# TYPE kubelet_network_plugin_operations_errors_total counter
网络插件累计操作错误数统计，按照操作类型(operation_type)统计

# HELP kubelet_network_plugin_operations_total [ALPHA] Cumulative number of network plugin operations by operation type.
# TYPE kubelet_network_plugin_operations_total counter
网络插件累计操作数统计，按照操作类型(operation_type)统计

# HELP kubelet_node_name [ALPHA] The node's name. The count is always 1.
# TYPE kubelet_node_name gauge
node name

# HELP kubelet_pleg_discard_events [ALPHA] The number of discard events in PLEG.
# TYPE kubelet_pleg_discard_events counter
PLEG(pod lifecycle event generator) 丢弃的event数统计

# HELP kubelet_pleg_last_seen_seconds [ALPHA] Timestamp in seconds when PLEG was last seen active.
# TYPE kubelet_pleg_last_seen_seconds gauge
PLEG上次活跃的时间戳

# HELP kubelet_pleg_relist_duration_seconds [ALPHA] Duration in seconds for relisting pods in PLEG.
# TYPE kubelet_pleg_relist_duration_seconds histogram
PLEG relist pod时间分布

# HELP kubelet_pleg_relist_interval_seconds [ALPHA] Interval in seconds between relisting in PLEG.
# TYPE kubelet_pleg_relist_interval_seconds histogram
PLEG relist 间隔时间分布

# HELP kubelet_pod_start_duration_seconds [ALPHA] Duration in seconds for a single pod to go from pending to running.
# TYPE kubelet_pod_start_duration_seconds histogram
pod启动时间(从pending到running)分布
kubelet watch到pod时到pod中contianer都running后
（watch各种source channel的pod变更）

# HELP kubelet_pod_worker_duration_seconds [ALPHA] Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync
# TYPE kubelet_pod_worker_duration_seconds histogram
pod状态变化的时间分布， 按照操作类型（create update sync）统计
worker就是kubelet中处理一个pod的逻辑工作单位

# HELP kubelet_pod_worker_start_duration_seconds [ALPHA] Duration in seconds from seeing a pod to starting a worker.
# TYPE kubelet_pod_worker_start_duration_seconds histogram
kubelet watch到pod到worker启动的时间分布

# HELP kubelet_run_podsandbox_duration_seconds [ALPHA] Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handler.
# TYPE kubelet_run_podsandbox_duration_seconds histogram
启动sandbox的时间分布

# HELP kubelet_run_podsandbox_errors_total [ALPHA] Cumulative number of the run_podsandbox operation errors by RuntimeClass.Handler.
# TYPE kubelet_run_podsandbox_errors_total counter
启动sanbox出现error的总数

# HELP kubelet_running_containers [ALPHA] Number of containers currently running
# TYPE kubelet_running_containers gauge
当前containers运行状态的统计
按照container状态统计，created running exited

# HELP kubelet_running_pods [ALPHA] Number of pods that have a running pod sandbox
# TYPE kubelet_running_pods gauge
当前处于running状态pod数量

# HELP kubelet_runtime_operations_duration_seconds [ALPHA] Duration in seconds of runtime operations. Broken down by operation type.
# TYPE kubelet_runtime_operations_duration_seconds histogram
容器运行时的操作耗时
(container在create list exec remove stop等的耗时)

# HELP kubelet_runtime_operations_errors_total [ALPHA] Cumulative number of runtime operation errors by operation type.
# TYPE kubelet_runtime_operations_errors_total counter
容器运行时的操作错误数统计(按操作类型统计)

# HELP kubelet_runtime_operations_total [ALPHA] Cumulative number of runtime operations by operation type.
# TYPE kubelet_runtime_operations_total counter
容器运行时的操作总数统计(按操作类型统计)

# HELP kubelet_started_containers_errors_total [ALPHA] Cumulative number of errors when starting containers
# TYPE kubelet_started_containers_errors_total counter
kubelet启动容器错误总数统计(按code和container_type统计)
code包括ErrImagePull ErrImageInspect ErrImagePull ErrRegistryUnavailable ErrInvalidImageName等
container_type一般为"container" "podsandbox"

# HELP kubelet_started_containers_total [ALPHA] Cumulative number of containers started
# TYPE kubelet_started_containers_total counter
kubelet启动容器总数

# HELP kubelet_started_pods_errors_total [ALPHA] Cumulative number of errors when starting pods
# TYPE kubelet_started_pods_errors_total counter
kubelet启动pod遇到的错误总数(只有创建sandbox遇到错误才会统计）

# HELP kubelet_started_pods_total [ALPHA] Cumulative number of pods started
# TYPE kubelet_started_pods_total counter
kubelet启动的pod总数

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
统计cpu使用率

# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
允许进程打开的最大fd数

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
当前打开的fd数量

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
进程驻留内存大小

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
进程启动时间

# HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
# TYPE rest_client_request_duration_seconds histogram
请求apiserver的耗时统计(按照url和请求类型统计verb)

# HELP rest_client_requests_total [ALPHA] Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
请求apiserver的总次数(按照返回码code和请求类型method统计)

# HELP storage_operation_duration_seconds [ALPHA] Storage operation duration
# TYPE storage_operation_duration_seconds histogram
存储操作耗时(按照存储plugin(configmap emptydir hostpath 等 )和operation_name分类统计)
# HELP volume_manager_total_volumes [ALPHA] Number of volumes in Volume Manager
# TYPE volume_manager_total_volumes gauge
本机挂载的volume数量统计(按照plugin_name和state统计
plugin_name包括"host-path" "empty-dir" "configmap" "projected")
state(desired_state_of_world期状态/actual_state_of_world实际状态）

2.kube-proxy指标

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
gc时间

# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
goroutine数量

# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
系统线程数量

# HELP kubeproxy_network_programming_duration_seconds [ALPHA] In Cluster Network Programming Latency in seconds
# TYPE kubeproxy_network_programming_duration_seconds histogram
service或者pod发生变化到kube-proxy规则同步完成时间
指标含义较复杂，参照https://github.com/kubernetes/community/blob/master/sig-scalability/slos/network_programming_latency.md

# HELP kubeproxy_sync_proxy_rules_duration_seconds [ALPHA] SyncProxyRules latency in seconds
# TYPE kubeproxy_sync_proxy_rules_duration_seconds histogram
规则的同步耗时

# HELP kubeproxy_sync_proxy_rules_endpoint_changes_pending [ALPHA] Pending proxy rules Endpoint changes
# TYPE kubeproxy_sync_proxy_rules_endpoint_changes_pending gauge
ep发生变化后规则同步pending的次数

# HELP kubeproxy_sync_proxy_rules_endpoint_changes_total [ALPHA] Cumulative proxy rules Endpoint changes
# TYPE kubeproxy_sync_proxy_rules_endpoint_changes_total counter
ep变化发生规则同步的总次数

# HELP kubeproxy_sync_proxy_rules_iptables_restore_failures_total [ALPHA] Cumulative proxy iptables restore failures
# TYPE kubeproxy_sync_proxy_rules_iptables_restore_failures_total counter
本机上iptable restore失败的总数数

# HELP kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds [ALPHA] The last time a sync of proxy rules was queued
# TYPE kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds gauge
最近一次规则同步的请求时间戳
如果比下一个指标kubeproxy_sync_proxy_rules_last_timestamp_seconds 大很多，
那说明同步hung住了

# HELP kubeproxy_sync_proxy_rules_last_timestamp_seconds [ALPHA] The last time proxy rules were successfully synced
# TYPE kubeproxy_sync_proxy_rules_last_timestamp_seconds gauge
最近一次规则同步的完成时间戳

# HELP kubeproxy_sync_proxy_rules_service_changes_pending [ALPHA] Pending proxy rules Service changes
# TYPE kubeproxy_sync_proxy_rules_service_changes_pending gauge
service变化引起的规则同步pending数量

# HELP kubeproxy_sync_proxy_rules_service_changes_total [ALPHA] Cumulative proxy rules Service changes
# TYPE kubeproxy_sync_proxy_rules_service_changes_total counter
service变化引起的规则同步总数

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
利用这个指标统计cpu使用率

# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
进程可以打开的最大fd数

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
进程当前打开的fd数
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
统计内存使用大小

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
进程启动时间戳

# HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
# TYPE rest_client_request_duration_seconds histogram
请求apiserver的耗时 (按照url和verb统计）

# HELP rest_client_requests_total [ALPHA] Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
请求apiserver的总数(按照code method host统计)

3.kube-state-metrics指标梳理

# HELP kube_configmap_info Information about configmap.
# TYPE kube_configmap_info gauge
统计configmap数量(label包含configmap的ns+name)

# HELP kube_configmap_created Unix creation timestamp
# TYPE kube_configmap_created gauge
统计configmap创建时间戳

# HELP kube_configmap_metadata_resource_version Resource version representing a specific version of the configmap.
# TYPE kube_configmap_metadata_resource_version gauge
configmap label: ns+name value: resource version

# HELP kube_daemonset_created Unix creation timestamp
# TYPE kube_daemonset_created gauge
统计deamonset 创建时间戳

# HELP kube_daemonset_status_current_number_scheduled The number of nodes running at least one daemon pod and are supposed to.
# TYPE kube_daemonset_status_current_number_scheduled gauge
daemonset 成功调度的个数
value应该与下面指标kube_daemonset_status_desired_number_scheduled相等

# HELP kube_daemonset_status_desired_number_scheduled The number of nodes that should be running the daemon pod.
# TYPE kube_daemonset_status_desired_number_scheduled gauge
daemonset期望调度的个数
value理论上应该等于可调度的node数

# HELP kube_daemonset_status_number_available The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and available
# TYPE kube_daemonset_status_number_available gauge
可以运行deamonset pod的node数

# HELP kube_daemonset_status_number_misscheduled The number of nodes running a daemon pod but are not supposed to.
# TYPE kube_daemonset_status_number_misscheduled gauge
daemonset 没有被调度的的数量

# HELP kube_daemonset_status_number_ready The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready.
# TYPE kube_daemonset_status_number_ready gauge
daemonset处于ready状态的pod数量

# HELP kube_daemonset_status_number_unavailable The number of nodes that should be running the daemon pod and have none of the daemon pod running and available
# TYPE kube_daemonset_status_number_unavailable gauge
daemonset处于不可用状态的pod数量

# HELP kube_daemonset_status_observed_generation The most recent generation observed by the daemon set controller.
# TYPE kube_daemonset_status_observed_generation gauge
daemonset controller看到的最新一代generation
统计daemonset创建后更新过多少次

# HELP kube_daemonset_status_updated_number_scheduled The total number of nodes that are running updated daemon pod
# TYPE kube_daemonset_status_updated_number_scheduled gauge
daemonset最近一次更新成功的数量

# HELP kube_daemonset_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_daemonset_labels gauge
daemonset的ns+name 作为指标label

# HELP kube_deployment_status_replicas The number of replicas per deployment.
# TYPE kube_deployment_status_replicas gauge
统计deployment的副本数
label 是ns+name，value是副本数

# HELP kube_deployment_status_replicas_ready The number of ready replicas per deployment.
# TYPE kube_deployment_status_replicas_ready gauge
deployment处于ready状态的副本数
label 是ns+name，value是副本数

# HELP kube_deployment_status_replicas_available The number of available replicas per deployment.
# TYPE kube_deployment_status_replicas_available gauge
deployment可用的副本数
label 是ns+name，value是副本数

# HELP kube_deployment_status_replicas_unavailable The number of unavailable replicas per deployment.
# TYPE kube_deployment_status_replicas_unavailable gauge
deployment处于不可用状态的副本数
label 是ns+name，value是副本数

# HELP kube_deployment_status_replicas_updated The number of updated replicas per deployment.
# TYPE kube_deployment_status_replicas_updated gauge
deployment更新的副本数
统计单个deployment中有多少处于更新过了
label 是ns+name，value是副本数

# HELP kube_deployment_status_observed_generation The generation observed by the deployment controller.
# TYPE kube_deployment_status_observed_generation gauge
deployment最近一次更新的generation
统计deployment创建后更新过多少次

# HELP kube_deployment_status_condition The current status conditions of a deployment.
# TYPE kube_deployment_status_condition gauge
统计deployment处于各condition的pod数量
condition取值Processing 、 Available
status 取值 True False Unknown

# HELP kube_deployment_spec_replicas Number of desired pods for a deployment.
# TYPE kube_deployment_spec_replicas gauge
deployment期望的副本数

# HELP kube_deployment_spec_paused Whether the deployment is paused and will not be processed by the deployment controller.
# TYPE kube_deployment_spec_paused gauge
deployment 是否处于pause状态

# HELP kube_deployment_spec_strategy_rollingupdate_max_unavailable Maximum number of unavailable replicas during a rolling update of a deployment.
# TYPE kube_deployment_spec_strategy_rollingupdate_max_unavailable gauge
deployment最大不可用副本数(默认为副本数的25%)

# HELP kube_deployment_spec_strategy_rollingupdate_max_surge Maximum number of replicas that can be scheduled above the desired number of replicas during a rolling update of a deployment.
# TYPE kube_deployment_spec_strategy_rollingupdate_max_surge gauge
deployment跟新最大副本数(默认为副本数的25%)

# HELP kube_deployment_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_deployment_labels gauge
deployment的label
label为ns+name

# HELP kube_endpoint_created Unix creation timestamp
# TYPE kube_endpoint_created gauge
endpoint的创建时间戳

# HELP kube_endpoint_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_endpoint_labels gauge
endpoint 信息
label ns+name

# HELP kube_endpoint_address_available Number of addresses available in endpoint.
# TYPE kube_endpoint_address_available gauge
endpoint可用数量统计

# HELP kube_endpoint_address_not_ready Number of addresses not ready in endpoint
# TYPE kube_endpoint_address_not_ready gauge
endpoint不可用数量统计

# HELP kube_ingress_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_ingress_labels gauge
ingress 信息
label ns+name
# HELP kube_ingress_created Unix creation timestamp
# TYPE kube_ingress_created gauge
ingress 创建时间戳

# HELP kube_ingress_metadata_resource_version Resource version representing a specific version of ingress.
# TYPE kube_ingress_metadata_resource_version gauge
ingress的resouce version

# HELP kube_ingress_path Ingress host, paths and backend service information.
# TYPE kube_ingress_path gauge
ingress 路径
label为 ns ingress host service_name service_port

# HELP kube_ingress_tls Ingress TLS host and secret information.
# TYPE kube_ingress_tls gauge
ingress是否启用TLS

# HELP kube_namespace_created Unix creation timestamp
# TYPE kube_namespace_created gauge
ns创建时间戳

# HELP kube_namespace_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_namespace_labels gauge
ns的信息
label为namespace

# HELP kube_namespace_status_phase kubernetes namespace status phase.
# TYPE kube_namespace_status_phase gauge
处于不同phase的ns统计

# HELP kube_node_created Unix creation timestamp
# TYPE kube_node_created gauge
node创建的时间戳

# HELP kube_node_info Information about a cluster node.
# TYPE kube_node_info gauge
node信息
label为node kernerl_version os_image container_rumtime_version kubelet_version
kubeproxy_version pod_cidr internal_ip

# HELP kube_node_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_node_labels gauge
node 信息
label为node(node=nodename)

# HELP kube_node_role The role of a cluster node.
# TYPE kube_node_role gauge
node角色,是否为controller-plane 是否为master
label： role="controll-plane"

# HELP kube_node_spec_taint The taint of a cluster node.
# TYPE kube_node_spec_taint gauge
node是否有taint信息

# HELP kube_node_spec_unschedulable Whether a node can schedule new pods.
# TYPE kube_node_spec_unschedulable gauge
node是否处于不可调度状态

# HELP kube_node_status_allocatable The allocatable for different resources of a node that are available for scheduling.
# TYPE kube_node_status_allocatable gauge
node处于可分配的资源统计
资源主要包含：cpu(单位毫核) memory(单位byte) pods(单位整数)
ephemeral_storage(单位G) hugepages_1Gi(单位byte) hugepages_2Mi(单位byte)

# HELP kube_node_status_capacity The capacity for different resources of a node.
# TYPE kube_node_status_capacity gauge
node资源的上限

# HELP kube_node_status_condition The condition of a cluster node.
# TYPE kube_node_status_condition gauge
node的资源压力状态统计
label 为node condition status
condition枚举
 DiskPressure
 MemoryPressure
 NetworkUnavailable
 PIDPressure
 Ready
 status: true false unknown


# HELP kube_persistentvolumeclaim_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_persistentvolumeclaim_labels gauge
pvc 信息
label为 namespace + persistentvolumeclaim

# HELP kube_persistentvolumeclaim_info Information about persistent volume claim.
# TYPE kube_persistentvolumeclaim_info gauge
pvc信息
label为 namespace persistentvolumeclaim storageclass volumename

# HELP kube_persistentvolumeclaim_status_phase The phase the persistent volume claim is currently in.
# TYPE kube_persistentvolumeclaim_status_phase gauge
pvc所处的阶段统计
label： namespace persistentvolumeclaim phase
phase 取值 Bound Lost Pending
# HELP kube_persistentvolumeclaim_resource_requests_storage_bytes The capacity of storage requested by the persistent volume claim.
# TYPE kube_persistentvolumeclaim_resource_requests_storage_bytes gauge
pvc声明的大小，单位byte
label：namespace persistentvolumeclaim

# HELP kube_persistentvolumeclaim_access_mode The access mode(s) specified by the persistent volume claim.
# TYPE kube_persistentvolumeclaim_access_mode gauge
pvc的访问模式
label: namespace persistentvolumeclaim access_mode
# HELP kube_persistentvolume_claim_ref Information about the Persistent Volume Claim Reference.
# TYPE kube_persistentvolume_claim_ref gauge
pvc的引用计数(被多少对象挂载)

# HELP kube_persistentvolume_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_persistentvolume_labels gauge
pv信息
label: persistentvolume

# HELP kube_persistentvolume_status_phase The phase indicates if a volume is available, bound to a claim, or released by a claim.
# TYPE kube_persistentvolume_status_phase gauge
pv所处的阶段统计
label persistentvolume phase
phase枚举值: vailable Bound Failed Pending Released
# HELP kube_persistentvolume_info Information about persistentvolume.
# TYPE kube_persistentvolume_info gauge
pv 信息
label: persistentvolume storageclass gce_persistent_disk_name
ebs_volume_id azure_disk_name fc_wwids fc_lun fc_target_wwns
iscsi_target_portal iscsi_iqn iscsi_lun iscsi_initiator_name
nfs_server nfs_path csi_driver csi_volume_handle

# HELP kube_persistentvolume_capacity_bytes Persistentvolume capacity in bytes.
# TYPE kube_persistentvolume_capacity_bytes gauge
pv容量统计
label persistentvolume, 单位byte

# HELP kube_pod_completion_time Completion time in unix timestamp for a pod.
# TYPE kube_pod_completion_time gauge
处于Error或者Unknown状态的Pod数

# HELP kube_pod_container_info Information about a container in a pod.
# TYPE kube_pod_container_info gauge
container信息
label:namespace pod uid container
image_spec image
image_id container_id

# HELP kube_pod_container_resource_limits The number of requested limit resource by a container.
# TYPE kube_pod_container_resource_limits gauge
容器资源limit
label: namespace pod uid container node resource unit
value是资源数限制

# HELP kube_pod_container_resource_requests The number of requested request resource by a container.
# TYPE kube_pod_container_resource_requests gauge
容器资源request
label:namespace pod uid container node resource unit
value是资源数限制

# HELP kube_pod_container_state_started Start time in unix timestamp for a pod container.
# TYPE kube_pod_container_state_started gauge
容器的启动时间戳
label namespace pod uid container

# HELP kube_pod_container_status_last_terminated_reason Describes the last reason the container was in terminated state.
# TYPE kube_pod_container_status_last_terminated_reason gauge
容器的terminated原因

# HELP kube_pod_container_status_ready Describes whether the containers readiness check succeeded.
# TYPE kube_pod_container_status_ready gauge
readiness探针是否探测成功

# HELP kube_pod_container_status_restarts_total The number of container restarts per container.
# TYPE kube_pod_container_status_restarts_total counter
容器的重启次数

# HELP kube_pod_container_status_running Describes whether the container is currently in running state.
# TYPE kube_pod_container_status_running gauge
容器是否处于running状态

# HELP kube_pod_container_status_terminated Describes whether the container is currently in terminated state.
# TYPE kube_pod_container_status_terminated gauge
容器是否处于terminated状态

# HELP kube_pod_container_status_terminated_reason Describes the reason the container is currently in terminated state.
# TYPE kube_pod_container_status_terminated_reason gauge
容器teminated状态的原因

# HELP kube_pod_container_status_waiting Describes whether the container is currently in waiting state.
# TYPE kube_pod_container_status_waiting gauge
容器是否处于waiting状态

# HELP kube_pod_container_status_waiting_reason Describes the reason the container is currently in waiting state.
# TYPE kube_pod_container_status_waiting_reason gauge
容器处于waiting状态的原因

# HELP kube_pod_created Unix creation timestamp
# TYPE kube_pod_created gauge
pod创建的时间戳

# HELP kube_pod_info Information about pod.
# TYPE kube_pod_info gauge
pod 信息
label:namespace pod uid host_ip pod_ip node
created_by_kind created_by_name priority_class host_network

# HELP kube_pod_init_container_info Information about an init container in a pod.
# TYPE kube_pod_init_container_info gauge
init容器的信息
label:namespace pod uid container image_spec image image_id container_id

# HELP kube_pod_init_container_status_ready Describes whether the init containers readiness check succeeded.
# TYPE kube_pod_init_container_status_ready gauge
init容器是否ready(rediness探针是否通过)

# HELP kube_pod_init_container_status_restarts_total The number of restarts for the init container.
# TYPE kube_pod_init_container_status_restarts_total counter
init容器重启的次数

# HELP kube_pod_init_container_status_running Describes whether the init container is currently in running state.
# TYPE kube_pod_init_container_status_running gauge
init容器是否处于running状态

# HELP kube_pod_init_container_status_terminated Describes whether the init container is currently in terminated state.
# TYPE kube_pod_init_container_status_terminated gauge
init容器是否处于terminated状态

# HELP kube_pod_init_container_status_terminated_reason Describes the reason the init container is currently in terminated state.
# TYPE kube_pod_init_container_status_terminated_reason gauge
init容器处于terminated状态的原因(Completed越多越好)

# HELP kube_pod_init_container_status_waiting Describes whether the init container is currently in waiting state.
# TYPE kube_pod_init_container_status_waiting gauge
init容器是否处于waiting状态

# HELP kube_pod_init_container_status_waiting_reason Describes the reason the init container is currently in waiting state.
# TYPE kube_pod_init_container_status_waiting_reason gauge
init容器处于waiting状态的原因

# HELP kube_pod_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_pod_labels gauge
pod信息
label:namespace pod uid

# HELP kube_pod_owner Information about the Pod's owner.
# TYPE kube_pod_owner gauge
pod的owner信息 (是否为单pod owner类型 )
namespace pod uid owner_kind owner_name owner_is_controller

# HELP kube_pod_restart_policy Describes the restart policy in use by this pod.
# TYPE kube_pod_restart_policy gauge
pod的重启策略
label:namespace pod uid type
type: Never Always OnFailure
# HELP kube_pod_spec_volumes_persistentvolumeclaims_info Information about persistentvolumeclaim volumes in a pod.
# TYPE kube_pod_spec_volumes_persistentvolumeclaims_info gauge
pod内pvc的信息
label:namespace pod uid volume persistentvolumeclaim

# HELP kube_pod_spec_volumes_persistentvolumeclaims_readonly Describes whether a persistentvolumeclaim is mounted read only.
# TYPE kube_pod_spec_volumes_persistentvolumeclaims_readonly gauge
pod内的pvc是否只读

# HELP kube_pod_start_time Start time in unix timestamp for a pod.
# TYPE kube_pod_start_time gauge
Pod启动的时间戳(最后一个container处于running状态的时间，区分创建时间戳)

# HELP kube_pod_status_phase The pods current phase.
# TYPE kube_pod_status_phase gauge
pod所处阶段
label:namespace pod uid phase
phase: Failed Pending Running Succeeded Unknown

# HELP kube_pod_status_ready Describes whether the pod is ready to serve requests.
# TYPE kube_pod_status_ready gauge
pod是否处于ready状态

# HELP kube_pod_status_reason The pod status reasons
# TYPE kube_pod_status_reason gauge
pod当前状态的原因

# HELP kube_pod_status_scheduled Describes the status of the scheduling process for the pod.
# TYPE kube_pod_status_scheduled gauge
pod是否处于调度完成状态

# HELP kube_pod_status_scheduled_time Unix timestamp when pod moved into scheduled status
# TYPE kube_pod_status_scheduled_time gauge
Pod调度完成的时间戳

# HELP kube_replicaset_created Unix creation timestamp
# TYPE kube_replicaset_created gauge
rs创建的时间戳

# HELP kube_replicaset_status_replicas The number of replicas per ReplicaSet.
# TYPE kube_replicaset_status_replicas gauge
rs的副本数

# HELP kube_replicaset_status_ready_replicas The number of ready replicas per ReplicaSet.
# TYPE kube_replicaset_status_ready_replicas gauge
处于ready状态的副本数

# HELP kube_replicaset_status_observed_generation The generation observed by the ReplicaSet controller.
# TYPE kube_replicaset_status_observed_generation gauge
rs的更新次数

# HELP kube_replicaset_spec_replicas Number of desired pods for a ReplicaSet.
# TYPE kube_replicaset_spec_replicas gauge
rs期望副本数

# HELP kube_replicaset_owner Information about the ReplicaSet's owner.
# TYPE kube_replicaset_owner gauge
rs的owner
label: namespace replicaset owner_kind owner_name owner_is_controller

# HELP kube_replicaset_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_replicaset_labels gauge
rs信息
label: namespace replicaset

# HELP kube_secret_info Information about secret.
# TYPE kube_secret_info gauge
secret info
label: namespace secret=
# HELP kube_secret_type Type about secret.
# TYPE kube_secret_type gauge
secret type
label:namespace secret type
type: helm.sh/release.v1
kubernetes.io/service-account-token
kubernetes.io/tls
Opaque

# HELP kube_secret_created Unix creation timestamp
# TYPE kube_secret_created gauge
secret创建时间戳

# HELP kube_service_info Information about service.
# TYPE kube_service_info gauge
service 信息
label: namespace service uid cluster_ip external_name load_balancer_ip

# HELP kube_service_created Unix creation timestamp
# TYPE kube_service_created gauge
service创建时间戳

# HELP kube_service_spec_type Type about service.
# TYPE kube_service_spec_type gauge
service类型
label: namespace service uid type
type: NodePort ClusterIP...

# HELP kube_service_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_service_labels gauge
service信息
label: namespace service uid

4.cadvisor(node上负载)指标梳理

# HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals.
# TYPE container_cpu_cfs_periods_total counter
cfs时间片总数
完全公平调度的时间片总数(分配到cpu的时间片数)
# HELP container_cpu_cfs_throttled_periods_total Number of throttled period intervals.
# TYPE container_cpu_cfs_throttled_periods_total counter
容器被throttle的时间片总数

# HELP container_cpu_cfs_throttled_seconds_total Total time duration the container has been throttled.
# TYPE container_cpu_cfs_throttled_seconds_total counter
容器被throttle的时间

# HELP container_file_descriptors Number of open file descriptors for the container.
# TYPE container_file_descriptors gauge
容器打开的fd数

# HELP container_memory_usage_bytes Current memory usage in bytes, including all memory regardless of when it was accessed
# TYPE container_memory_usage_bytes gauge
容器内存使用量，单位byte

# HELP container_network_receive_bytes_total Cumulative count of bytes received
# TYPE container_network_receive_bytes_total counter
容器入方向的流量

# HELP container_network_transmit_bytes_total Cumulative count of bytes transmitted
# TYPE container_network_transmit_bytes_total counter
容器出方向的流量

# HELP container_spec_cpu_period CPU period of the container.
# TYPE container_spec_cpu_period gauge
容器的cpu调度单位时间

# HELP container_spec_cpu_quota CPU quota of the container.
# TYPE container_spec_cpu_quota gauge
容器的cpu规格 ，除以单位调度时间可以计算核数


# HELP container_spec_memory_limit_bytes Memory limit for the container.
# TYPE container_spec_memory_limit_bytes gauge
容器的内存规格，单位byte

# HELP container_threads Number of threads running inside the container
# TYPE container_threads gauge
容器当前的线程数
# HELP container_threads_max Maximum number of threads allowed inside the container, infinity if value is zero
# TYPE container_threads_max gauge
允许容器启动的最大线程数

上面container_cpu_cfs_throttled_periods_total与container_cpu_cfs_throttled_seconds_total的区别，举个例子。 + 假设容器的limit的是0.4核，那么意味着100ms内可以运行的时间为40ms； + 不加资源限制的情况下，进程需要运行持续200ms；

到这里，Kubernetes相关组件的指标我们就梳理完成了， 下一步我们要针对不同的组件指标进行采集。采集方式主要分为两种:本地采集或者集中式采集。

本地采集： 适用于采集目标数比较多或者不适合开放外部采集的场景 + 比如针对pod内的业务指标，就是sidecar方式部署categraf到pod内; + 针对集群规模比较大的采集，比如大集群的node指标采集，我们可以用daemonset的方式，采集基础指标的同时，通过本地请求也采集了kubelet、kube-proxy的指标； + 针对默认监听在物理机127.0.0.1地址上的服务，我们也可以采用这种本地部署的方式；

集中式采集 适用于采集目标数比较少的场景，再配合服务发现，这类指标采集就会变得相对简单了。 + 比如控制面组件的指标采集，当然kube-controller-manager， kube-scheduler要修改--bind-address参数，原本监听在127.0.0.1的要改成监听0.0.0.0 + kube-state-metrics本身一般是deployment部署，暴露端口又是http方式，所以也适用集中式采集。

答疑过程中发现有很多关于上篇文章中采集配置的咨询，下一篇我们会重点分享采集的yaml配置与实践。

首发在夜莺监控公众号，版权所有，转载请注明作者和出处