Filebeat入门案例

Filebeat入门案例

一.介绍

二. Filebeat

2.1介绍

2.2架构

2.3安装

使用rz工具将Filebeat压缩文件上传到Linux虚拟机解压：tar -zxvf filebeat-7.12.1-linux-x86_64.tar.gz -C /opt/

2.4入门案例

接下来我们使用filebeat读取一个普通的日志文件

# 进入filebeat文件夹下 cd /opt/filebeat-7.12.1-linux-x86_64/# 创建配置文件 vim mylogconfig.ymlfilebeat.inputs:- type: log enabled: true paths: - /opt/log/mylog.logoutput.console: pretty: true enable: true

基于配置文件启动filebeat

./filebeat -e -c mylogconfig.yml

参数说明： -e：标准输出，输出到控制台 -c：指定配置文件 4. 向文本文件追加数据，测试filebeat是否能为增量数据生成日志数据

# 打开另一个会话窗口，进入文本文件的目录下 cd /opp/# 向文本文件中追加内容，再次查看filebeat的控制台 echo '科比是我的偶像' >> mylog.log

2.5自定义字段

Filebeat读取日志文件后会生成json格式的日志，我们还可以为生成的日志添加一些自定义字段：

# 修改配置文件：filebeat.inputs:- type: log enabled: true paths: - /opt/log/mylog.log tags: ["mylog","test"]#添加自定义标签，便于后续处理 fields: from: mylog fields_under_root: true #true为添加到根节点，false为添加到子节点中output.console: pretty: true enable: true# 重启filebeat ./filebeat -e -c mylogconfig.yml# 向文本文件追加数据 echo '科比是我的偶像' >> mylog.log

2.6收集Nginx日志

安装Nginx

tar -zxvf nginx-1.21.1.tar.gz -C /usr/local/# 安装依赖包yum -y install gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel# 安装Nginx./configuremake & make install# 启动Nginx/usr/local/nginx/sbin/nginx./nginx

Nginx的日志文件在/usr/local/nginx/logs中，正常日志存在access.log中，异常日志存在error.log 中。

读取Nginx日志的配置文件

# 在filebeat中创建配置文件cd /usr/local/filebeat-7.12.1-linux-x86_64/vim nginxlogconfig.ymlfilebeat.inputs:- type: log enabled: true paths: - /usr/local/nginx/logs/*.log tags: ["nginx"]output.console: pretty: true enable: true# 启动filebeat，如果filebeat还在启动，关闭已启动的filebeat ./filebeat -e -c nginxlogconfig.yml

2.72.7 Filebeat模板

在收集Ngnix日志时，日志内容并没有处理，难以阅读其中的具体数据。Filebeat针对常见的服务提供了处理日志的模板。接下来我们讲解Filebeat中Module的使用。

配置Nginx读取模板：

# 查看Filebeat的日志处理模板 ./filebeat modules list [root@node0 filebeat]# ./filebeat modules listEnabled:nginxDisabled:activemqapacheauditdawsazurebarracudabluecoatcefcheckpointciscocorednscrowdstrikecyberarkcylanceelasticsearchenvoyproxyf5fortinetgcpgoogle_workspacegooglecloudgsuitehaproxyibmmqicingaiisimpervainfobloxiptablesjuniperkafkakibanalogstashmicrosoftmispmongodbmssqlmysqlmysqlenterprisenatsnetflownetscouto365oktaoracleosquerypanwpensandopostgresqlproofpointrabbitmqradwareredissantasnortsnyksonicwallsophossquidsuricatasystemthreatinteltomcattraefikzeekzoomzscaler# 启用模板 ./filebeat modules enable nginxcd modules.d/ vim nginx.yml# 配置日志处理模板- module: nginx # Access logs access: enabled: true var.paths: ["/usr/local/nginx/logs/access.log"] # Error logs error: enabled: true var.paths: ["/usr/local/nginx/logs/error.log"]

修改配置文件：

vim nginxlogconfig.ymlfilebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: trueoutput.console: pretty: true enable: true

启动filebeat，如果filebeat还在启动，关闭已启动的filebeat

./filebeat -e -c nginxlogconfig.yml

{ "@timestamp": "2021-12-03T04:57:55.131Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "7.12.1", "pipeline": "filebeat-7.12.1-nginx-access-pipeline" }, "event": { "dataset": "nginx.access", "module": "nginx", "timezone": "+08:00" }, "fileset": { "name": "access" }, "ecs": { "version": "1.8.0" }, "host": { "name": "node0" }, "log": { "offset": 1746, "file": { "path": "/usr/local/nginx/logs/access.log" } }, "message": "192.168.134.1 - - [03/Dec/2021:12:57:51 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\"", "input": { "type": "log" }, "service": { "type": "nginx" }, "agent": { "id": "1043e956-ee86-47e0-8edd-084cab906fd9", "name": "node0", "type": "filebeat", "version": "7.12.1", "hostname": "node0", "ephemeral_id": "dda18f29-2c37-49e9-b1b0-755f61cf47ea" }}

2.8 将数据输出到ES中

启动Elasticsearch启动Kibana，连接Elasticsearch修改Filebeat配置文件：

vim nginxlogconfig.ymlfilebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: trueoutput.elasticsearch: hosts: ["node0:9200"]

启动filebeat，如果filebeat还在启动，关闭已启动的filebeat

./filebeat -e -c nginxlogconfig.yml

进入Kibana查看数据

GET /filebeat-7.12.1/_search{ "query": { "match_all": {} }}

{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 2, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-7.12.1", "_type" : "_doc", "_id" : "K4XUfn0BLb2Jb20VAvp9", "_score" : 1.0, "_source" : { "agent" : { "hostname" : "node0", "name" : "node0", "id" : "1043e956-ee86-47e0-8edd-084cab906fd9", "ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af", "type" : "filebeat", "version" : "7.12.1" }, "nginx" : { "access" : { "remote_ip_list" : [ "192.168.134.1" ] } }, "log" : { "file" : { "path" : "/usr/local/nginx/logs/access.log" }, "offset" : 2319 }, "source" : { "address" : "192.168.134.1", "ip" : "192.168.134.1" }, "fileset" : { "name" : "access" }, "url" : { "original" : "/" }, "input" : { "type" : "log" }, "@timestamp" : "2021-12-03T05:46:00.000Z", "ecs" : { "version" : "1.8.0" }, "related" : { "ip" : [ "192.168.134.1" ] }, "service" : { "type" : "nginx" }, "host" : { "name" : "node0" }, ": { "request" : { "method" : "GET" }, "response" : { "status_code" : 304, "body" : { "bytes" : 0 } }, "version" : "1.1" }, "event" : { "ingested" : "2021-12-03T05:46:02.766382193Z", "timezone" : "+08:00", "created" : "2021-12-03T05:46:01.739Z", "kind" : "event", "module" : "nginx", "category" : [ "web" ], "type" : [ "access" ], "dataset" : "nginx.access", "outcome" : "success" }, "user_agent" : { "original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36", "os" : { "name" : "Windows", "version" : "10", "full" : "Windows 10" }, "name" : "Chrome", "device" : { "name" : "Other" }, "version" : "93.0.4577.82" } } }, { "_index" : "filebeat-7.12.1", "_type" : "_doc", "_id" : "LIXUfn0BLb2Jb20VBPp8", "_score" : 1.0, "_source" : { "agent" : { "hostname" : "node0", "name" : "node0", "id" : "1043e956-ee86-47e0-8edd-084cab906fd9", "type" : "filebeat", "ephemeral_id" : "934e20db-a77a-40ca-af7b-6c9556d7b1af", "version" : "7.12.1" }, "nginx" : { "access" : { "remote_ip_list" : [ "192.168.134.1" ] } }, "log" : { "file" : { "path" : "/usr/local/nginx/logs/access.log" }, "offset" : 2510 }, "source" : { "address" : "192.168.134.1", "ip" : "192.168.134.1" }, "fileset" : { "name" : "access" }, "url" : { "original" : "/" }, "input" : { "type" : "log" }, "@timestamp" : "2021-12-03T05:46:02.000Z", "ecs" : { "version" : "1.8.0" }, "related" : { "ip" : [ "192.168.134.1" ] }, "service" : { "type" : "nginx" }, "host" : { "name" : "node0" }, ": { "request" : { "method" : "GET" }, "response" : { "status_code" : 304, "body" : { "bytes" : 0 } }, "version" : "1.1" }, "event" : { "ingested" : "2021-12-03T05:46:03.760599179Z", "timezone" : "+08:00", "created" : "2021-12-03T05:46:02.757Z", "kind" : "event", "module" : "nginx", "category" : [ "web" ], "type" : [ "access" ], "dataset" : "nginx.access", "outcome" : "success" }, "user_agent" : { "original" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36", "os" : { "name" : "Windows", "version" : "10", "full" : "Windows 10" }, "name" : "Chrome", "device" : { "name" : "Other" }, "version" : "93.0.4577.82" } } } ] }}

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。