记:logstash优化

版本7.3.0

优化之前情况:

三台logstash接收udp协议数据,发送到kafka,5台logstash接收kafka数据发送到es。
esIP写了三个,一个es协调节点cpu及内存基本无消耗。日志通过udp数据发送过来,已经有丢失的情况,5台logstash消费kafka,有严重的消息堆积情况,kafka吞吐量达到平均十万每秒,之前也有过,但是基本就是七八万,偶尔达到十万以上,所以突然日志量增加,峰值达到14万,一天多了12亿条日志
服务器资源紧张,所以想通过优化参数改进。共用4个topic,只有一个特别大,其他量很小。

机器配置8c16g

logstash.yml修改配置:

1.pipeline.workers
通过zabbix查看cpu使用率特别低,而且配置是默认的,后来调大了一倍,cpu也增长了一倍
2.pipeline.batch.size
默认125,改为2500,同时config/jvm.options调整为 -Xms12g -Xmx12g,
3.将logstash加入了es的x-pack监控
以下是logstash.yml的配置:

# ------------  Node identity ------------
#
# Use a descriptive name for the node:
#
node.name: indexer-1
#
# If omitted the node name will default to the machine's host name
#
# ------------ Pipeline Settings --------------
#
# The ID of the pipeline.
#
pipeline.id: indexer
#
# Set the number of workers that will, in parallel, execute the filters+outputs
# stage of the pipeline.
#
# This defaults to the number of the host's CPU cores.
#
pipeline.workers: 16
#
# How many events to retrieve from inputs before sending to filters+workers
#
pipeline.batch.size: 2500
#
# ------------ Pipeline Configuration Settings --------------
#
# Where to fetch the pipeline configuration for the main pipeline
#
path.config: /koala/logstash-7.3.0/config/logstash-indexer.conf
#
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "110.119.120.114"
#
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: your_password
xpack.monitoring.elasticsearch.hosts: ["http://ip1:9200", "http://ip2:9200"]
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

针对kafka消费者的优化:

1.增加output中的es的节点IP
2.input配置中消费kafka的线程增加到15 consumer_threads => 15
刚开始改到10发现还是稍微有点慢,资源使用率也较低,所以尝试调大,具体如何计算不太清楚,只是一点点试
3.topic区分消费,不同机器消费不同的topic

评论

暂无评论