kubernetes引入HPA进行扩缩容，HPA可以根据指标(metrics)，自动对workload进行弹性的扩缩容。
HPA资源在autoscaling这个group下面，HPA分为v1和v2版本：
- HPA v1仅支持基于CPU指标的伸缩；
- HPA v2beta1引入了基于Memory和自定义指标的伸缩；
- HPA v2beta2引入了自定义策略(behavior)的伸缩；
- HPA v2将v2beta2变成GA功能；
# kubectl api-versions|grep auto
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
autoscaling/v2一.HPA的指标类型
Resouce: 提供Pod级别的CPU/Memory使用量；
- 通过metrics-server提供指标，调用metrics.k8s.io；
ContainerResource: 提供Container级别的CPU/Memory使用量；
- 通过metrics-server提供指标，调用metrics.k8s.io；
Object: 提供k8s集群内任意资源的相关指标；
- 通过prometheus-adapter提供指标，调用custom.metrics.k8s.io；
Pods: 提供k8s集群内任意资源的相关指标；
- 通过prometheus-adapter提供指标，调用custom.metrics.k8s.io；
- External: 提供k8s集群外的指标数据；
二.HPA自定义指标的伸缩
1. 部署prometheus-adapter
自定义指标由prometheus-adapter查询prometheus得到，所以需要部署prometheus和prometheus-adatper组件。
这里使用helm chart的方式部署：
- prometheus: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
- prometheus-adatper: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter
部署完成以后：
# kubectl get pod -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 4d20h
kube-state-metrics-796659d8b6-vr5w8 3/3 Running 0 4d20h
node-exporter-rn9nj 2/2 Running 0 4d20h
prometheus-adapter-5f4df978c8-4fxjq 1/1 Running 0 46s
prometheus-k8s-0 3/3 Running 1 4d20h
prometheus-operator-7bc7bd7d7-jp754 2/2 Running 0 4d20h2. 创建deploy和service
deploy: 初始为1个副本
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app
labels:
app: sample-app
spec:
replicas: 1
selector:
matchLabels:
app: sample-app
template:
metadata:
labels:
app: sample-app
spec:
containers:
- image: luxas/autoscale-demo:v0.1.2
name: metrics-provider
ports:
- name: http
containerPort: 8080service:
apiVersion: v1
kind: Service
metadata:
labels:
app: sample-app
name: sample-app
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app: sample-app
type: ClusterIP该sample-app仅暴露了一个指标：http_requests_total:
# curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics
# HELP http_requests_total The amount of requests served by the server in total
# TYPE http_requests_total counter
http_requests_total 61793. 创建serviceMonitor
创建serviceMonitor，以便让prometheus发现sample-app，拉到它的metrics:
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
name: sample-app
labels:
app: sample-app
spec:
selector:
matchLabels:
app: sample-app
endpoints:
- port: http4. 创建prometheusRule自定义指标
通过prometheusRule创建自定义指标：
- 创建了自定义指标：sample_app_http_requests_rate = rate(http_requests_total[5m])
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: k8s
role: alert-rules
name: sample-app-rules
namespace: monitoring
spec:
groups:
- name: sample-app.expr.rules
rules:
- expr: |
rate(http_requests_total[5m])
record: sample_app_http_requests_rate
labels:
namespace: default
service: sample-app可以通过kubectl查询自定义指标：
# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_http_requests_rate"|python -m json.tool
{
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"items": [
{
"describedObject": {
"apiVersion": "/v1",
"kind": "Pod",
"name": "sample-app-7cfb596f98-jmswq",
"namespace": "default"
},
"metricName": "sample_app_http_requests_rate",
"selector": null,
"value": "33m"
}
],
"kind": "MetricValueList",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/sample_app_http_requests_rate"
}
}5. 创建HPA
使用 sample_app_http_requests_rate 指标，创建HPA：
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: sample-app
spec:
scaleTargetRef:
# point the HPA at the sample application
# you created above
apiVersion: apps/v1
kind: Deployment
name: sample-app
# autoscale between 1 and 10 replicas
minReplicas: 1
maxReplicas: 10
metrics:
# use a "Pods" metric, which takes the average of the
# given metric across all pods controlled by the autoscaling target
- type: Pods
pods:
# use the metric that you used above: pods/sample_app_http_requests_rate
metricName: sample_app_http_requests_rate
# target 500 milli-requests per second,
# which is 1 request every two seconds
targetAverageValue: 500m6. 加大流量观察HPA
通过curl命令，加大流量：
# while true; do curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics; sleep 0.1; done
等待一会后，观察hpa对象，可以看到deploy已扩容为5个副本：
# kubectl get hpa sample-app
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
sample-app Deployment/sample-app 697m/500m 1 10 5 45h再次通过kubectl 查询自定义指标：
# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/sample_app_http_requests_rate?selector=app%3Dsample-app | python -m json.tool
{
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"items": [
{
"describedObject": {
"apiVersion": "/v1",
"kind": "Pod",
"name": "sample-app-7cfb596f98-9vkrb",
"namespace": "default"
},
"metricName": "sample_app_http_requests_rate",
"selector": null,
"value": "263m"
},
{
"describedObject": {
"apiVersion": "/v1",
"kind": "Pod",
"name": "sample-app-7cfb596f98-f4dqr",
"namespace": "default"
},
"metricName": "sample_app_http_requests_rate",
"selector": null,
"value": "553m"
},
{
"describedObject": {
"apiVersion": "/v1",
"kind": "Pod",
"name": "sample-app-7cfb596f98-jmswq",
"namespace": "default"
},
"metricName": "sample_app_http_requests_rate",
"selector": null,
"value": "1740m"
},
{
"describedObject": {
"apiVersion": "/v1",
"kind": "Pod",
"name": "sample-app-7cfb596f98-qg24g",
"namespace": "default"
},
"metricName": "sample_app_http_requests_rate",
"selector": null,
"value": "234m"
}
],
"kind": "MetricValueList",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/sample_app_http_requests_rate"
}
}参考：
1.github-doc:https://github.com/arielb135/HPA-with-prometheus-and-RabbitMQ/blob/master/README.md
2.HPA指标类型：https://mp.weixin.qq.com/s/ul2G1WFPmHo5QR0xI143jg
