k8s 基于 prometheus的custom metrics逻辑

olivee 5年前 ⋅ 1182 阅读

逻辑流程图

k8s-custom-metrics.jpg

说明

1. 定义HPA

这里定义了一个测试的HPA(sample-app),这个HAP关联的Deployment为sample-app,动态扩容的范围为1~10,扩容的条件是根据Pods的metrics(http_requests)。

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: http_requests
      targetAverageValue: 500m

2. ContollerManager调用查询metric,根据查询的metric值扩容

对于前面定义的HPA(sample-app),ContollerManager会调用APIServer接口/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests?labelSelector=app%3Dsample-app查询metric

3. APIService:v1beta1.custom.metrics.k8s.io转发到monitoring/prometheus-adapter服务

4. monitoring/prometheus-adapter服务转发到POD(prometheus-adapter)

5. POD(prometheus-adapter)根据ConfigMAP(adapter-config)配置从prometheus查询统计结果,并返回数据

POD(prometheus-adapter)的接口/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests?labelSelector=app%3Dsample-app会根据以下配置查询prometheus的数据(查询sample-app的http_requests_totalmetric数据)

    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:
      - isNot: .*_seconds_total
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)

5.1 每个POD(sample-app)的metrics数据

如prometheus通pull获取到的POD(sample-app)的http_requests_total数据:

http_requests_total{endpoint="http",instance="10.244.146.157:8080",job="sample-app",namespace="default",pod="sample-app-579bc6774c-86m9v",service="sample-app"} 11444
http_requests_total{endpoint="http",instance="10.244.146.165:8080",job="sample-app",namespace="default",pod="sample-app-579bc6774c-jvkw5",service="sample-app"} 11440

6. prometheus返回数据