ELK集群模式部署

Wesley13
• 阅读 448

架构拓扑图为:

ELK集群模式部署

准备工作:

下载资源包:

Elasticsearch: wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz  # 这一步用docker启动,可以不用下载。

Kibana: wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86\_64.tar.gz

Logstash:wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz

Filebeat:wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86\_64.tar.gz

安装:

根据拓扑图把对应的服务安装在对应的服务器。

tar xvf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/

tar xvf logstash-6.2.4.tar.gz -C /usr/local/

tar -xvf filebeat-6.2.4-linux-x86_64.tar.gz -C /usr/local/

部署启动:

Elasticsearch-master(192.168.11.1):

需自行提前安装好docker 和 docker-compose。 

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.1:9200:9200"
      - "192.168.11.1:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

Elasticsearch-node2.data(192.168.11.2):

需自行提前安装好docker 和 docker-compose。 

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.2"      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.2:9200:9200"
      - "192.168.11.2:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

Elasticsearch-node3.data(192.168.11.3):

需自行提前安装好docker 和 docker-compose。 

vim docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3
    container_name: elasticsearch
    environment:
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "network.publish_host=192.168.11.3"      - "discovery.zen.ping.unicast.hosts=192.168.11.1"
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /data/elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "192.168.11.3:9200:9200"
      - "192.168.11.3:9300:9300"

mkdir -p /data/elasticsearch_data && chmod 775 /data/elasticsearch_data && docker-compose up -d 

查看集群节点:curl -XGET 'http://192.168.11.1:9200/\_cat/nodes?pretty'

 kibana(192.168.11.4):

cd /usr/local/kibana-6.2.3-linux-x86_64

cp ./config/kibana.yml ./config/kibana.yml.bak

echo '' > ./config/kibana.yml

vim ./config/kibana.yml

server.port: 5601
server.host: "192.168.11.4"
elasticsearch.url: "http://192.168.11.1:9200"

启动:./bin/kibana

扩展包(可装可不装):

kibana 报警扩展: 

安装 ./bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.3-2/sentinl-v6.2.3.zip

重启 kibana

logstash(192.168.11.5)

cd /usr/local/logstash-6.2.4

vim nginx.conf

input {
beats {
port => 5066
host => "192.168.11.5"
codec => "json"
}
}
filter {

mutate {

gsub => ["message", "\\x", "\\\x"]

}

json {

source => "message"

}

}
output {
    elasticsearch {
        action => "index"
        hosts => "192.168.11.1:9200"
        index => "nginx-json-%{+YYYY.MM.dd}"
   }
}

mkdir -p /data/logstash-data/nginx

启动:./bin/logstash -f ./nginx.conf --path.data=/data/logstash-data/nginx &

扩展

grok调试地址:http://grokdebug.herokuapp.com/

如需调试请修改output成以下,方便debug

output {
        stdout {
        codec    => rubydebug
    }
}

filebeat(192.168.11.6)

cd /usr/local/filebeat-6.2.4-linux-x86_64

vim nginx.yml

filebeat.prospectors:
- input_type: log
  document_type: jsonlog
  paths:
    - /var/log/nginx/access.log
output:
  logstash:
    hosts: ["192.168.11.5:5066"]

启动:./filebeat -c ./nginx.yml &

nginx 日志的输出格式为:

log_format json '{"@timestamp":"$time_iso8601",'
               '"remote_addr":"$remote_addr",'
               '"request":"$request",'
               '"status":$status,'
               '"body_bytes":$body_bytes_sent,'
               '"user_agent":"$http_user_agent",'
               '"resp_time":"$upstream_response_time",'
               '"req_time":$request_time,'
               '"host":"$host",'
               '"@version":"1",'
               '"http_x_forwarded_for":"$http_x_forwarded_for",'
               '"upstream_addr":"$upstream_addr",'
               '"req_body":"$request_body"}';

参考地址:

https://www.elastic.co/guide/index.html 

http://docs.docker.com/compose/

https://www.elastic.co/guide/en/logstash/current/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docker.html

https://www.elastic.co/guide/en/kibana/current/install.html

点赞
收藏
评论区
推荐文章
Jacquelyn38 Jacquelyn38
1年前
2020年前端实用代码段,为你的工作保驾护航
有空的时候,自己总结了几个代码段,在开发中也经常使用,谢谢。1、使用解构获取json数据let jsonData  id: 1,status: "OK",data: 'a', 'b';let  id, status, data: number   jsonData;console.log(id, status, number )
blmius blmius
1年前
MySQL:[Err] 1292 - Incorrect datetime value: ‘0000-00-00 00:00:00‘ for column ‘CREATE_TIME‘ at row 1
文章目录问题用navicat导入数据时,报错:原因这是因为当前的MySQL不支持datetime为0的情况。解决修改sql\mode:sql\mode:SQLMode定义了MySQL应支持的SQL语法、数据校验等,这样可以更容易地在不同的环境中使用MySQL。全局s
Easter79 Easter79
1年前
swap空间的增减方法
(1)增大swap空间去激活swap交换区:swapoff v /dev/vg00/lvswap扩展交换lv:lvextend L 10G /dev/vg00/lvswap重新生成swap交换区:mkswap /dev/vg00/lvswap激活新生成的交换区:swapon v /dev/vg00/lvswap
Stella981 Stella981
1年前
KVM调整cpu和内存
一.修改kvm虚拟机的配置1、virsheditcentos7找到“memory”和“vcpu”标签,将<namecentos7</name<uuid2220a6d1a36a4fbb8523e078b3dfe795</uuid
Stella981 Stella981
1年前
ASMSupport教程4.9 生成三元运算符
<p这节我们介绍如何用ASMSupport生成三元运算符(...?...:...)运算符。我们预计生成如下代码:</p<divid"scid:9D7513F9C04C4721824A2B34F0212519:935e30cc33214e0093ba9834f3a4e044"class"wlWriterEditableS
Wesley13 Wesley13
1年前
MySQL查询按照指定规则排序
1.按照指定(单个)字段排序selectfromtable_nameorderiddesc;2.按照指定(多个)字段排序selectfromtable_nameorderiddesc,statusdesc;3.按照指定字段和规则排序selec
Wesley13 Wesley13
1年前
Java日期时间API系列36
  十二时辰,古代劳动人民把一昼夜划分成十二个时段,每一个时段叫一个时辰。二十四小时和十二时辰对照表:时辰时间24时制子时深夜11:00凌晨01:0023:0001:00丑时上午01:00上午03:0001:0003:00寅时上午03:00上午0
Stella981 Stella981
1年前
Android蓝牙连接汽车OBD设备
//设备连接public class BluetoothConnect implements Runnable {    private static final UUID CONNECT_UUID  UUID.fromString("0000110100001000800000805F9B34FB");
Stella981 Stella981
1年前
Angular material mat
IconIconNamematiconcode_add\_comment_addcommenticon<maticonadd\_comment</maticon_attach\_file_attachfileicon<maticonattach\_file</maticon_attach\
Wesley13 Wesley13
1年前
MySQL部分从库上面因为大量的临时表tmp_table造成慢查询
背景描述Time:20190124T00:08:14.70572408:00User@Host:@Id:Schema:sentrymetaLast_errno:0Killed:0Query_time:0.315758Lock_
helloworld_34035044 helloworld_34035044
7个月前
皕杰报表之UUID
​在我们用皕杰报表工具设计填报报表时,如何在新增行里自动增加id呢?能新增整数排序id吗?目前可以在新增行里自动增加id,但只能用uuid函数增加UUID编码,不能新增整数排序id。uuid函数说明:获取一个UUID,可以在填报表中用来创建数据ID语法:uuid()或uuid(sep)参数说明:sep布尔值,生成的uuid中是否包含分隔符'',缺省为