我安装了一个minikube kubernetes集群,通过prometheus进行监控。kubernetes版本为v1.13.4,直接在vm主机上运行(--vm-驱动=无)。我在Prometheus配置文件中添加了一个特定的作业,以便抓取cAdvisor or容器指标。问题是Prometheus无法从cAdvisorendpoint抓取指标。
我在prometheus. yml中包含了以下配置
- job_name: 'kubernetes-cadvisor'
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
但在prometheus目标webUI(超文本传输协议://my_ip: 30900/目标)我看到kubernetes-cAdvisor在DOWN状态与以下错误消息
http://kubernetes.default.svc:443/api/v1/nodes/minikube/proxy/metrics/cadvisor: context deadline exceeded
kubernetes. default.svc:443应该是DNS默认集群,可从pod中访问,但正如预期的那样,我无法在prometheus pod中ping它。
幸运的是,我注意到我可以成功地从url中获取所有cAdvisor容器指标:
http://my_dashboard_ip_and_port/api/v1/nodes/minikube/proxy/metrics/cadvisor
prometheus吊舱内的日志是:
kubectl logs prometheus-deployment-6f64ff68f9-8c9xm
level=info ts=2019-03-29T14:33:18.939973334Z caller=main.go:285 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-03-29T14:33:18.940326462Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.8.1, branch=HEAD, revision=4d60eb36dcbed725fcac5b27018574118f12fffb)"
level=info ts=2019-03-29T14:33:18.94039376Z caller=main.go:322 build_context="(go=go1.11.6, user=root@bfdd6a22a683, date=20190328-18:04:08)"
level=info ts=2019-03-29T14:33:18.940455316Z caller=main.go:323 host_details="(Linux 4.15.0 #1 SMP Tue Mar 26 02:53:14 UTC 2019 x86_64 prometheus-deployment-6f64ff68f9-8c9xm (none))"
level=info ts=2019-03-29T14:33:18.94050961Z caller=main.go:324 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-03-29T14:33:18.940570849Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-03-29T14:33:18.941555805Z caller=main.go:640 msg="Starting TSDB ..."
level=info ts=2019-03-29T14:33:18.941946171Z caller=web.go:418 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-03-29T14:33:18.946861683Z caller=main.go:655 msg="TSDB started"
level=info ts=2019-03-29T14:33:18.947193152Z caller=main.go:724 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-03-29T14:33:18.948922627Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-03-29T14:33:18.950164896Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-03-29T14:33:18.951281382Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-03-29T14:33:18.952276845Z caller=main.go:751 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-03-29T14:33:18.952303937Z caller=main.go:609 msg="Server is ready to receive web requests."
但我不知道如何正确配置Prometheus yml文件,以便也能够通过Prometheus获取暴露的指标。
提前非常感谢。
我想你可能不得不使用HTTPS模式来抓取对HTTP的请求,这似乎保留在你的配置中:
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
为了跳过API服务器证书验证,您可以将insecure_skip_verify: true
参数添加到现有的tls_config
:
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true