一、软件及对应端口

软件 端口
Elasticsearch 9200(数据连接)、9300(集群通信)
Cerebro 9000
Kibana 5601
Logstash 6514(tcp/udp)、6515(tcp)

二、文件及目录说明

文件及目录 说明
/opt/es/config/elasticsearch.yml es的主配置文件
/opt/es/config/jvm.options es的jvm配置文件
/opt/es es安装目录
/opt/es_logs/ es日志目录
/opt/es_data/ es数据目录
/opt/kibana/config/kibana.yml kibana主配置文件
/opt/cerebro/conf/application.conf cerebro主配置文件
/opt/logstash-conf/ logstash配置文件目录

三、优化系统

本机如果运行过,可以忽略

cd ~
rm -f bigops-init.tar.gz
wget http://dl.bigops.com/bigops-init.tar.gz
tar zxvf bigops-init.tar.gz
cd bigops-init
sh init.sh

四、安装ElasticSearch单机

1)安装ElasticSearch

cd ~
rm -f elk.tar.gz
wget http://dl.bigops.com/elk.tar.gz
tar zxvf elk.tar.gz
cd elk
sh node.sh

2)安装Kibana、Cerebro、Logstash

sh kibana_cerebro_logstash.sh

3)修改ElasticSearch和Logstash的JVM配置

编辑/opt/es/config/jvm.options和/opt/logstash/config/jvm.options,根据主机可用内存大小合理分配

-Xms8g
-Xmx8g

4)修改防火墙策略

确认防火墙策略,容许ELK相关端口容许被访问。如果使用了公有云,需要打开端口策略。

五、安装ElasticSearch集群

集群规划。例如:

节点名 IP node.master node.data
node1 192.168.0.2 true true
node2 192.168.0.3 true true
node3 192.168.0.4 true true

1)安装ElasticSearch

在node1、node2、node3上都运行

cd ~
rm -f elk.tar.gz
wget http://dl.bigops.com/elk.tar.gz
tar zxvf elk.tar.gz
cd elk
sh node.sh

2)安装Kibana、Cerebro、Logstash

在node1上运行

sh kibana_cerebro_logstash.sh

3)编辑的ElasticSearch配置文件/opt/es/config/elasticsearch.yml

node1节点

cluster.name: bigops
node.name: node1
node.master: true
node.data: true
network.host: 0.0.0.0
http.port: 9200
path.data: /opt/es_data
path.logs: /opt/es_logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

cluster.max_shards_per_node: 900000
indices.query.bool.max_clause_count: 10240
indices.fielddata.cache.size: 40%
indices.memory.index_buffer_size: 40%

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

transport.tcp.port: 9300
cluster.initial_master_nodes: ["192.168.0.2:9300"]
discovery.seed_hosts: ["192.168.0.2:9300","192.168.0.3:9300","192.168.0.4:9300"]

node2节点

cluster.name: bigops
node.name: node2
node.master: true
node.data: true
network.host: 0.0.0.0
http.port: 9200
path.data: /opt/es_data
path.logs: /opt/es_logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

cluster.max_shards_per_node: 900000
indices.query.bool.max_clause_count: 10240
indices.fielddata.cache.size: 40%
indices.memory.index_buffer_size: 40

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

transport.tcp.port: 9300
cluster.initial_master_nodes: ["192.168.0.2:9300"]
discovery.seed_hosts: ["192.168.0.2:9300","192.168.0.3:9300","192.168.0.4:9300"]

node3节点

cluster.name: bigops
node.name: node3
node.master: true
node.data: true
network.host: 0.0.0.0
http.port: 9200
path.data: /opt/es_data
path.logs: /opt/es_logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

cluster.max_shards_per_node: 900000
indices.query.bool.max_clause_count: 10240
indices.fielddata.cache.size: 40%
indices.memory.index_buffer_size: 40%

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

transport.tcp.port: 9300
cluster.initial_master_nodes: ["192.168.0.2:9300"]
discovery.seed_hosts: ["192.168.0.2:9300","192.168.0.3:9300","192.168.0.4:9300"]

4)修改ElasticSearch和Logstash的JVM配置

node1、node2、node3依次运行,编辑/opt/es/config/jvm.options,根据主机内存合理分配,例如:

-Xms8g
-Xmx8g

5)启动集群

node1、node2、node3依次运行,检查状态正常后运行下一台

systemctl restart es
systemctl status es

6)生成证书

node1运行

cd /opt/es
./bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""

把node1上的证书拷贝到node2和node3

scp /opt/es/config/elastic-certificates.p12 192.168.0.3:/opt/es/config/
scp /opt/es/config/elastic-certificates.p12 192.168.0.4:/opt/es/config/

在node1、node2、node3上依次运行

chown es.es /opt/es/config/elastic*

在node1、node2、node3上依次编辑/opt/es/config/elasticsearch.yml文件,末尾添加

xpack.ml.enabled: false
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /opt/es/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /opt/es/config/elastic-certificates.p12

在node1、node2、node3上依次重启ElasticSearch

systemctl restart es

7)设置集群密码,在node1上运行,不需要在node2和node3上运行

./bin/elasticsearch-setup-passwords interactive

防止输错,可以把密码复制12行后粘贴

8)在node2和node3主机执行

rm -rf /opt/es_data/*
systemctl restart es

9)检查集群状态

修改cerebro配置/opt/cerebro/conf/application.conf

    auth = {
      username = "elastic"
      password = "password"
    }

重启cerebro后浏览器查看集群状态

systemctl restart cerebro

10)增加master节点,编辑/opt/es/config/elasticsearch.yml

修改node1、node2、node3节点配置并重启es服务

cluster.initial_master_nodes: ["192.168.0.2:9300","192.168.0.3:9300","192.168.0.4:9300"]

六、升级ElasticSearch

1)确认不存在目录/opt/es_bak、/opt/kibana_bak、/opt/logstash_bak

2)升级Elasticsearch程序

systemctl stop es
mv /opt/es/ /opt/es_bak
tar zxvf elasticsearch-7.11.1-linux-x86_64.tar.gz
mv elasticsearch-7.11.1 /opt/es
cp -f /opt/es_bak/config/elasticsearch.yml /opt/es/config/
cp -f /opt/es_bak/config/jvm.options /opt/es/config/

3)如果有证书,拷贝证书

cp -f /opt/es_bak/config/elastic-certificates.p12 /opt/es/config/elastic-certificates.p12

4)顺序重启集群ElasticSearch,重启完一个后,在cerebro检查节点和集群的状态,确认状态恢复为绿色后再重启下一台

chown -R es:es /opt/es
systemctl restart es
systemctl status es

5)升级Kibana

systemct stop kibana
tar zxvf kibana-7.11.1-linux-x86_64.tar.gz
mv /opt/kibana /opt/kibana_bak
mv kibana-7.11.1-linux-x86_64 /opt/kibana
cp -f /opt/kibana_bak/config/kibana.yml /opt/kibana/config/
systemctl restart kibana
systemctl status kibana

6)升级Logstash

tar zxvf logstash-7.11.1-linux-x86_64.tar.gz
mv /opt/logstash /opt/logstash_bak
mv logstash-7.11.1 /opt/logstash
cp -f /opt/logstash_bak/config/startup.options /opt/logstash/config/startup.options
cp -f /opt/logstash_bak/config/jvm.options /opt/logstash/config/jvm.options
cp -f /opt/logstash_bak/bin/logstash.lib.sh /opt/logstash/bin/logstash.lib.sh
/opt/logstash/bin/system-install /opt/logstash/config/startup.options systemd
systemctl restart logstash
systemctl status logstash

results matching ""

    No results matching ""