验证prometheus监控targets的metrics

在Prometheus监控Targets页面，显示的都是监控标的列表。我们如何验证访问此/metrics的结果呢？

monitoring/kube-apiserver

这种https://172.16.64.233:6443/metrics URL，是需要带证书才能访问的：

curl -k --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
https://172.16.64.233:6443/metrics
...
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.2317e-05
go_gc_duration_seconds{quantile="0.25"} 3.1153e-05
go_gc_duration_seconds{quantile="0.5"} 4.6201e-05

monitoring/kube-state-metrics

在页面上显示的Endpoint是https://10.244.169.167:9443/metrics

然后查看kube-state-metrics-serviceMonitor.yaml

发现有个配置bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

进入到pod prometheus-k8s-0, 通过/var/run/secrets/kubernetes.io/serviceaccount/token文件拿到token。

然后执行下面的命令就可以拿到指标数据了。

更多详情请看此文 Prometheus monitoring/kube-state-metrics监控指标

curl -k https://10.244.169.167:9443/metrics -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb21ldGhldXMtazhzLXRva2VuLXNtNmdkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMtazhzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMTk2OTkyOTItNDY2Yi00NWQ4LWJmYmYtYzkyZjIwNjczOWY3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1vbml0b3Jpbmc6cHJvbWV0aGV1cy1rOHMifQ.iggZ4ZxmD0y04OQfDlo4P6zRgzn0ryVhcdhlgncpnBY5BJ39Xz0a2AA51ePa78R2njFDjPcecgDJRcqPv76X3o-C-G7EZvN_Ru8zSdB51YxqlLNoIW5hy6Jr27aw74lMslg1MYX_31kkRTqD9DxVn6lq6Uqf4Djebj_E-2maiwl863GCeNRfS1X6KM8idsVknLlpdVINbM8U_l1Yuw-auNzelAk1NQlBdbJqsm1CZKIg_YBsT-KbiyTsbjX2v0uL1D6-Q5Xs9NZMLEAa7dfwz_EOYMDnIGbv-eyhD-924H4_pGOIoQ0dCBP01cxFm7pLJPGouwLaEwPs5BRS0B6u-w"

monitoring/kubelet

https://172.16.64.233:10250/metrics

monitoring/node-exporter

https://172.16.64.233:9100/metrics