Finology 大数据金融

通过大数据以量化金融

0%

发表于 更新于 分类于 ， 阅读次数： 评论数：

通过kube-prometheus安装的k8s监控系统。

在targets页面，我们来分析一下monitoring/node-exporter/的这个Targets。

node-exporter-serviceMonitor.yaml内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https
    relabelings:
    - action: replace
      regex: (.*)
      replacment: $1
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      targetLabel: instance
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: node-exporter

monitoring/node-exporter/0

Endpoint: https://172.16.64.233:9100/metrics

查看Configuration页面，发现是以role endpoints做为自动服务发现的

1
2
3
4
5
6
7
8
9
- job_name: monitoring/node-exporter/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - role: endpoints
...

获取token

1
kubectl get secret prometheus-k8s-token-sm6gd -n monitoring -o template={{.data.token}} | base64 -d

通过template获取secret里面的值，要通过base64解密一下的。

指标大部分以node_开始
少许几个以promhttp_metric_开始的指标。

指标node-exporter-node0.txt 点击查看

发表于 更新于 分类于 ， 阅读次数： 评论数：

通过kube-prometheus安装的k8s监控系统。

在targets页面，我们来分析一下monitoring/kubelet/的两个Targets。

prometheus-serviceMonitorKubelet.yaml内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kubelet
  name: kubelet
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    interval: 30s
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    interval: 30s
    metricRelabelings:
    - action: drop
      regex: container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)
      sourceLabels:
      - __name__
    path: /metrics/cadvisor
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

monitoring/kubelet/0

Endpoint: https://172.16.64.232:10250/metrics

指标以kubelet_开始
apiserver_
http_request_
http_request_
rest_client_request_
storage_operation_
volume_manager_

指标kubelet-0-0.txt 点击查看

monitoring/kubelet/1

Endpoint: https://172.16.64.232:10250/metrics/cadvisor

使用的自动发现role为endpoints

1
2
3
4
5
6
7
8
9
- job_name: monitoring/kubelet/1
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  kubernetes_sd_configs:
  - role: endpoints

包含一个cadvisor_version_info指标可以查看cadvisor的版本信息。
其他指标以container_开始

指标kubelet-1-0.txt 点击查看

发表于 更新于 分类于 ， 阅读次数： 评论数：

通过kube-prometheus安装的k8s监控系统。

在targets页面，我们来分析一下monitoring/kube-state-metrics的两个Targets。

kube-state-metrics-serviceMonitor.yaml内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  honorLabels: true
  interval: 30s
  port: https-main
  scheme: https
  scrapeTimeout: 30s
  tlsConfig:
    insecureSkipVerify: true
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  interval: 30s
  port: https-self
  scheme: https
  tlsConfig:
    insecureSkipVerify: true
jobLabel: k8s-app
selector:
  matchLabels:
    k8s-app: kube-state-metrics

Endpoint

这里面有两个Targets，是因为在kube-state-metrics-serviceMonitor.yaml文件中的spec下面配置了两个endpoints。

并且这两个都是用bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token来获取权限的。

进入prometheus-k8s-0容器查看这个文件的内容，或者通过secret直接查看，我们选择后者。

1
2
3
4
5
kubectl get secret -n monitoring
NAME                              TYPE                                  DATA   AGE
prometheus-k8s                    Opaque                                1      3h12m
prometheus-k8s-token-sm6gd        kubernetes.io/service-account-token   3      58d
...
1
kubectl describe secret prometheus-k8s-token-sm6gd -n monitoring

取到token

1
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb21ldGhldXMtazhzLXRva2VuLXNtNmdkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMtazhzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMTk2OTkyOTItNDY2Yi00NWQ4LWJmYmYtYzkyZjIwNjczOWY3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1vbml0b3Jpbmc6cHJvbWV0aGV1cy1rOHMifQ.iggZ4ZxmD0y04OQfDlo4P6zRgzn0ryVhcdhlgncpnBY5BJ39Xz0a2AA51ePa78R2njFDjPcecgDJRcqPv76X3o-C-G7EZvN_Ru8zSdB51YxqlLNoIW5hy6Jr27aw74lMslg1MYX_31kkRTqD9DxVn6lq6Uqf4Djebj_E-2maiwl863GCeNRfS1X6KM8idsVknLlpdVINbM8U_l1Yuw-auNzelAk1NQlBdbJqsm1CZKIg_YBsT-KbiyTsbjX2v0uL1D6-Q5Xs9NZMLEAa7dfwz_EOYMDnIGbv-eyhD-924H4_pGOIoQ0dCBP01cxFm7pLJPGouwLaEwPs5BRS0B6u-w

monitoring/kube-state-metrics/0

Endpoint: https://10.244.36.69:8443/metrics

执行如下命令即可获取所有指标了。

1
curl -k https://10.244.36.69:8443/metrics -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb21ldGhldXMtazhzLXRva2VuLXNtNmdkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMtazhzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMTk2OTkyOTItNDY2Yi00NWQ4LWJmYmYtYzkyZjIwNjczOWY3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1vbml0b3Jpbmc6cHJvbWV0aGV1cy1rOHMifQ.iggZ4ZxmD0y04OQfDlo4P6zRgzn0ryVhcdhlgncpnBY5BJ39Xz0a2AA51ePa78R2njFDjPcecgDJRcqPv76X3o-C-G7EZvN_Ru8zSdB51YxqlLNoIW5hy6Jr27aw74lMslg1MYX_31kkRTqD9DxVn6lq6Uqf4Djebj_E-2maiwl863GCeNRfS1X6KM8idsVknLlpdVINbM8U_l1Yuw-auNzelAk1NQlBdbJqsm1CZKIg_YBsT-KbiyTsbjX2v0uL1D6-Q5Xs9NZMLEAa7dfwz_EOYMDnIGbv-eyhD-924H4_pGOIoQ0dCBP01cxFm7pLJPGouwLaEwPs5BRS0B6u-w"

指标以kube_开始。
kube_configmap_
kube_cronjob_
kube_daemonset_
kube_deployment_
kube_endpoint_
kube_namespace_
kube_node_
kube_persistentvolumeclaim_
kube_persistentvolume_
kube_pod_
kube_pod_container_
kube_secret_
kube_statefulset_
kube_storageclass_

我将指标存入kube-state-metrics-0.txt 点击查看，供下载查看。

monitoring/kube-state-metrics/1

Endpoint: https://10.244.36.69:9443/metrics

指标为go系统级别指标。

同理可得其指标。kube-state-metrics-1.txt 点击查看