一、CentOS 7.9 安装 elasticsearch-7.8.1
地址
- https://github.com/medcl/elasticsearch-analysis-ik
- https://www.elastic.co/guide/en/logstash/7.8/index.html
二、安装准备
JDK 11+：由于elasticsearch 运行需要JDK环境，如果 JDK 低于 1.8 会有一些问题。
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk1.8.0_333/jre] does not meet this requirement版本需要和elasticserach版本一致，否则会报错
三、安装elasticsearch
解压
tar -zxvf /opt/software/elastic/elasticsearch-7.8.1-linux-x86_64.tar.gz -C /opt/elastic/进入elasticsearch目录中
目录中自带了jdk
四、修改配置文件
1 修改配置：jvm.options
因为默认的jvm参数-Xms和Xmx都是1G。自己搭的虚拟机可能内存分配不多，会导致启动失败，根据自己的机器情况合理分配。
vim /opt/elastic/elasticsearch-7.8.1/config/jvm.options
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512m
################################################################2 修改配置：elasticsearch.yml
vim /opt/elastic/elasticsearch-7.8.1/config/elasticsearch.yml
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /opt/elastic/es/data # 数据存放位置
#
# Path to log files:
#
path.logs: /opt/elastic/es/logs # 日志存放位置
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.0. # 开放外网访问
#
# Set a custom port for HTTP:
#
http.port: 9200 # 设置端口
#
# For more information, consult the network module documentation.五、尝试启动
1 require 必须 Java 11
2 运行elasticsearch不能使用root用户启动
查看代码
[root@ecs-65685 bin]# /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk1.8.0_333/jre] does not meet this requirement
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk1.8.0_333/jre] does not meet this requirement
[2022-10-11T17:40:42,664][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ecs-65685] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.1.jar:7.8.1]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /opt/elastic/es/logs/elasticsearch.log
[root@ecs-65685 bin]# 2022-10-11 09:40:42,993994 UTC [6056] ERROR CLogger.cc@311 Cannot log to named pipe /tmp/elasticsearch-1369077120224648390/controller_log_5893 as it could not be opened for writing
2022-10-11 09:40:42,994085 UTC [6056] INFO Main.cc@104 Parent process died - ML controller exiting3 创建一个用户
groupadd es # 创建一个组
useradd es -g es # 用户属于es组
passwd es # 密码
# R（递归）分配目录下所有文件/目录的拥有者
chown -R es:es /opt/elastic/elasticsearch-7.8.1
# 授权存储数据权限
chmod -R 777 /opt/elastic/es
4 再次启动es
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch
[1] 说你的系统太low了，给爷增加点增加虚拟内存，至少262144
vim /etc/sysctl.conf
vm.max_map_count=655360
# 使设置的虚拟内存生效
[root@ecs-65685 config]# sysctl -p
vm.swappiness = 0
net.core.somaxconn = 1024
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_max_syn_backlog = 1024
vm.max_map_count = 655360
[2] 解决：在elasticsearch的config目录下，修改 elasticsearch.yml配置文件
# 取消注释，并保留一个节点
# skipping cluster bootstrapping as local node does not match bootstrap requirements: [node-1]
cluster.initial_master_nodes: ["node-1"]
5 再次启动es
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch -d6 访问
curl 192.168.0.98:9200
公网地址：114.116...:9200
注意：放行9200
五、Kibana安装
1 下载
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.8.1-linux-x86_64.tar.gz2 解压
tar -zxvf /opt/software/elastic/kibana-7.8.1-linux-x86_64.tar.gz -C /opt/elastic/3 配置 kibana.yml
vim /opt/elastic/kibana-7.8.1-linux-x86_64/config/kibana.yml
# 端口
server.port: 5601
# 开放外网访问
server.host: "0.0.0.0"
# es的服务器
elasticsearch.hosts: ["http://192.168.0...:9200"]
4 启动
查看代码
# 如果是要用root用户启动 就在后面加 --allow-root ，要么就切换用户执行
[root@ecs-65685 bin]# /opt/elastic/kibana-7.8.1-linux-x86_64/bin/kibana
Kibana should not be run as root. Use --allow-root to continue.
[root@ecs-65685 bin]# /opt/elastic/kibana-7.8.1-linux-x86_64/bin/kibana --allow-root
log [17:31:37.776] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: apm_oss
log [17:31:37.787] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: triggers_actions_ui
log [17:32:03.147] [info][plugins-service] Plugin "visTypeXy" is disabled.
log [17:32:03.148] [info][plugins-service] Plugin "endpoint" is disabled.
log [17:32:03.148] [info][plugins-service] Plugin "ingestManager" is disabled.
log [17:32:03.149] [info][plugins-service] Plugin "lists" is disabled.
log [17:32:05.837] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [17:32:05.904] [info][plugins-system] Setting up [94] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,devTools,translations,uiActions,statusPage,share,newsfeed,mapsLegacy,mapsLegacyLicensing,kibanaUtils,kibanaReact,inspector,embeddable,advancedUiActions,embeddableEnhanced,drilldowns,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,telemetryManagementSection,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,inputControlVis,savedObjects,navigation,lens,graph,maps,visualize,dashboard,dashboardEnhanced,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,triggers_actions_ui,infra,monitoring,logstash,uptime,ml,siem,apm]
log [17:32:05.914] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml
log [17:32:06.058] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml
log [17:32:06.059] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [17:32:06.109] [warning][actions][actions][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.
log [17:32:06.122] [warning][alerting][alerting][plugins][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.
log [17:32:06.149] [info][monitoring][monitoring][plugins] config sourced from: production cluster
log [17:32:06.150] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: undefined
log [17:32:06.188] [info][crossClusterReplication][plugins] Your basic license does not support crossClusterReplication. Please upgrade your license.
log [17:32:06.189] [info][plugins][watcher] Your basic license does not support watcher. Please upgrade your license.
log [17:32:06.196] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
log [17:32:06.393] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [17:32:06.412] [info][savedobjects-service] Starting saved objects migrations
log [17:32:06.447] [info][savedobjects-service] Creating index .kibana_task_manager_1.
log [17:32:06.450] [info][savedobjects-service] Creating index .kibana_1.
log [17:32:06.899] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
log [17:32:06.947] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.
log [17:32:06.999] [info][savedobjects-service] Finished in 552ms.
log [17:32:07.022] [info][savedobjects-service] Finished in 577ms.
log [17:32:07.025] [info][plugins-system] Starting [72] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,translations,share,discover,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,inputControlVis,lens,graph,visualize,dashboard,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,infra,monitoring,logstash,uptime,ml,siem,apm]
log [17:32:09.117] [info][status][plugin:kibana@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:09.121] [info][status][plugin:elasticsearch@7.8.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [17:32:09.121] [info][status][plugin:elasticsearch@7.8.1] Status changed from yellow to green - Ready
log [17:32:09.123] [info][status][plugin:xpack_main@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:09.133] [info][status][plugin:monitoring@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:09.136] [warning][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml
log [17:32:09.138] [warning][plugins][reporting] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Centos 7.9.2009 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'.
log [17:32:09.139] [warning][reporting] Enabling the Chromium sandbox provides an additional layer of protection.
log [17:32:10.356] [info][status][plugin:reporting@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.383] [info][status][plugin:spaces@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.385] [info][status][plugin:security@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.391] [info][status][plugin:dashboard_mode@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.393] [info][status][plugin:beats_management@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.438] [info][status][plugin:maps@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.446] [info][plugins][taskManager][taskManager] TaskManager is identified by the Kibana UUID: 66a6ab9a-0339-47c1-9fa8-ef059a59152e
log [17:32:10.449] [info][status][plugin:task_manager@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.451] [info][status][plugin:encryptedSavedObjects@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.454] [info][status][plugin:apm_oss@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.456] [info][status][plugin:console_legacy@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.459] [info][status][plugin:region_map@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.464] [info][status][plugin:ui_metric@7.8.1] Status changed from uninitialized to green - Ready
log [17:32:10.470] [info][listening] Server running at http://192.168.0...:5601
log [17:32:10.901] [info][server][Kibana][http] http server running at http://192.168.0...:56015 浏览器访问：IP:5601
六、elasticsearch-analysis-ik分词器插件安装
作用：如果直接使用Elasticsearch的分词器在处理中文内容的搜索时，ES会将中文词语分成一个一个的汉字。当用Kibana作图，按照term来分组的时候，也会将一个汉字单独分成一组。这对于我们的使用是及其不方便的，因此我们引入es之中文的分词器插件es-ik就能解决这个问题。
1 elasticsearch-analysis-ik分词器下载地址：https://github.com/medcl/elasticsearch-analysis-ik/releases
2 创建分词器目录
分词器安装在ES目录下即可，所有在ES目录下创建分词器目录
mkdir /opt/elastic/elasticsearch-7.8.1/plugins/ik3 解压安装包
unzip /opt/software/elastic/elasticsearch-analysis-ik-7.8.1.zip -d /opt/elastic/elasticsearch-7.8.1/plugins/ik/4 重新对ES目录授权
chown -R es:es /opt/elastic/elasticsearch-7.8.1
5 关闭ES和Kibana
可以先通过Kill命令杀掉ES和Kibana进程
ps -ef | grep elastic
kill -96 启动
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch -d
/opt/elastic/kibana-7.8.1-linux-x86_64/bin/kibana此时的ES日志是可以查看到是有加载分词器插件的
七、logstash安装
1 下载
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.8.1.tar.gz2 解压
tar -zxvf /opt/software/elastic/logstash-7.8.1.tar.gz -C /opt/elastic/3 配置 logstash.yml
vim /opt/elastic/logstash-7.8.1/config/logstash.yml
系统配置不高可以适当减少
4 启动
- -t 测试配置文件是否正确。
-f 指定logstash的配置文件。
-e 格式是后面跟字符串，这个字符串就被当做是logstash的配置，如果""，那么默认使用stdin作为输出。
logstash启动命令位于安装路径的bin目录中，直接启动会报错，需要按照如下方式提供参数。
命令行启动日志中看到 "Successfully started Logstash API endpoint {:port=>9600}" ，就说明启动成功。
上面命令行定义的是一个标准输入插件（即 stdin）和一个标准输出插件（即 stdout）。
意思就是从命令行提取输入，并从命令行直接将提取的数据输出，我们可以测试下，启动后如下图，界面在等待输入。
/opt/elastic/logstash-7.8.1/bin/logstash -e "input {stdin {}} output {stdout {}}"
5 连接 Elasticsearch
logstash可以通过命令行并使用 -e 参数传入配置字符串，指定了标准输入 stdin 插件和 stdout 插件。
但在实际应用中，通常使用配置文件指定插件，配置文件的语法形式与命令行相同，要使用的插件名称来指定，一般配置文件均放置到部署目录的config目录中，配置中找到名为 logstash-sample.conf 的文件，也可新建配置文件
启动配置文件 logstash-sample.conf
vim /opt/elastic/logstash-7.8.1/config/logstash-sample.conf
input {
file {
type => "systemlog-xyz" #标签
path => "/opt/elastic/logstash/logs/xyz" #采集点
start_position => "beginning" #开始收集点
stat_interval => "2" #扫描间隔时间，默认是1s，建议5s
}
}
output {
elasticsearch {
hosts => ["192.168.0.98:9200"] #指定信息发送到es
index => "app-xyz-log-%{+YYYY.MM.dd}" #定义格式
}
file {
path => "/opt/elastic/logstash/logs/xyz/xyz.log"
}
}启动命令
/opt/elastic/logstash-7.8.1/bin/logstash -f ../config/logstash-sample.conf -t
/opt/elastic/logstash-7.8.1/bin/logstash -f ../config/logstash-sample.conf八、Filebeat安装
报错
1 failed; error='Not enough space
虚拟机内存不足，修改虚拟机分配内存大小的参数是，-Xmx和-Xms
[es@ecs-65685 ~]$ /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
output:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 549755813888 bytes for committing reserved memory.
# An error report file with more information is saved as:
# logs/hs_err_pid12013.log
error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f38ee000000, 549755813888, 0) failed; error='Not enough space' (errno=12)
at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
[es@ecs-65685 ~]$ 2 Permission denied
es用户启动 elasticsearch时，es目录没有权限：chmod -R 777 /opt/elastic/es
[es@ecs-65685 ~]$ /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch
2022-10-11 23:54:04,818 main ERROR RollingFileManager (/opt/elastic/es/logs/elasticsearch_deprecation.log) java.io.FileNotFoundException: /opt/elastic/es/logs/elasticsearch_deprecation.log (Permission denied) java.io.FileNotFoundException: /opt/elastic/es/logs/elasticsearch_deprecation.log (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:291)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:234)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:155)
3 logstash直接启动会报错
[root@ecs-65685 ~]# /opt/elastic/logstash-7.8.1/bin/logstash
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/opt/elastic/logstash-7.8.1/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /opt/elastic/logstash-7.8.1/logs which is now configured via log4j2.properties
ERROR: Pipelines YAML file is empty. Location: /opt/elastic/logstash-7.8.1/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2022-10-12T14:31:11,184][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit标签：
留言评论