我正在实现Prometheus来监控我的Kubernetes系统的健康状况,其中我有多个集群和名称空间。
我的目标是只监视一个名为default
的特殊命名空间,只监视我自己的pod,不包括prometheus pods和监视细节。
我尝试在kubernetes_sd_configs
中指定命名空间,如下所示:
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- 'default'
我仍然得到我不需要的指标。
以下是我的configMap.yml:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: default
data:
prometheus.rules: |-
groups:
- name: devopscube demo alert
rules:
- alert: High Pod Memory
expr: sum(container_memory_usage_bytes) > 1
for: 1m
labels:
severity: slack
annotations:
summary: High Memory Usage
prometheus.yml: |-
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/prometheus.rules
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- "alertmanager.monitoring.svc:9093"
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- 'default'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
namespaces:
names:
- 'default'
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- 'default'
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
namespaces:
names:
- 'default'
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- 'default'
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
我不想以下面这个细节为例来加以监视:
✔container_memory_rss{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",id="/system.slice/kubelet.service",instance="minikube",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="minikube",kubernetes_io_os="linux"}
✔container_memory_rss{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",id="/system.slice/docker.service",instance="minikube",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="minikube",kubernetes_io_os="linux"}
✔container_memory_rss{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",id="/kubepods/podda7b74d8-b611-4dff-885c-70ea40091b7d",instance="minikube",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="minikube",kubernetes_io_os="linux",namespace="kube-system",pod="default-http-backend-59f7ff8999-ktqnl",pod_name="default-http-backend-59f7ff8999-ktqnl"}
如果您只想防止某些指标被摄取(即防止保存在 Prometheus 数据库中),您可以使用指标重新标记来删除它们:
- job_name: kubernetes-cadvisor
metric_relabel_configs:
- source_labels: [__name__]
regex: container_memory_rss
action: drop
请注意,在< code>kubernetes-cadvisor作业中,您使用< code >节点服务发现角色。这将发现Kubernetes节点,它们是非命名空间资源,因此在这种情况下,您对< code>default的命名空间限制可能没有任何影响。
嘿,刚在文档中找到这个。
# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
names:
[ - <string> ]
就在 https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ingress 下
如果您想从特定的应用程序或服务中抓取度量,那么只将prometheus抓取注释应用于您感兴趣的应用程序服务。
样品
apiVersion: apps/v1beta2 # for versions before 1.8.0 use extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: weave
labels:
app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9102'
spec:
containers:
- name: fluentd-elasticsearch
image: gcr.io/google-containers/fluentd-elasticsearch:1.20
pod上的注释允许您控制是否需要抓取指标
prometheus.io/scrape:默认配置将抓取所有 Pod,如果设置为 false,则此注释将从抓取过程中排除该 Pod。
prometheus.io/path:如果指标路径不是 /metrics,请使用此注释对其进行定义。
prometheus.io/port:抓取指定端口上的pod,而不是pod声明的端口(如果没有声明端口,则默认为无端口目标)。