首页 文章

用于php monolog多行(stacktrace)日志的Logstash grok过滤器配置

提问于
浏览
0
[2018-02-12 09:15:43] development.WARNING: home page  
[2018-02-12 09:15:43] development.INFO: home page  
[2018-02-12 10:22:50] development.WARNING: home page  
[2018-02-12 10:22:50] development.INFO: home page  
[2018-02-12 10:22:50] development.ERROR: Call to undefined function vie() {"exception":"[object](Symfony\\Component\\Debug\\Exception\\FatalThrowableError(code: 0): Call to undefined function vie() at /var/www/html/routes/web.php:16 
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Route.php(198): Illuminate\\Routing\\Router->{closure}()
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Route.php(172): Illuminate\\Routing\\Route->runCallable()
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php(658): Illuminate\\Routing\\Route->run()
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): Illuminate\\Routing\\Router->Illuminate\\Routing\\{closure}(Object(Illuminate\\Http\\Request))
#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(41): Illuminate\\Routing\\Pipeline->Illuminate\\Routing\\{closure}(Object(Illuminate\\Http\\Request))
#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle(Object(Illuminate\\Http\\Request), Object(Closure))

.....
.....
.....

#45 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(151): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#46 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(116): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter(Object(Illuminate\\Http\\Request))
#47 /var/www/html/public/index.php(55): Illuminate\\Foundation\\Http\\Kernel->handle(Object(Illuminate\\Http\\Request))
#48{main}
"}

以上是我的Laravel monolog样本日志数据 . 我正在使用Logstash读取日志数据并将其发送到Elasticsearch . 下面是我的logstash.conf文件

input {
  file {
    path => '/var/www/html/php-app/application/storage/logs/laravel-*.log'
    start_position => 'beginning'
    ignore_older => 0
  }
}
filter {
  grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: %{DATA:message}at %{DATA:trace}" }
  }
}
output {
  elasticsearch {
    hosts => [ 'localhost:9200' ]
    index => "laravel-%{+YYYY-MM-dd}"
  }
  stdout {
    codec => rubydebug
  }
}

以上配置适用于单行日志消息 . 例如,在日志消息下面

[2018-02-12 09:15:43] development.WARNING: home page

生成输出为

"timestamp": "2018-02-12 10:57:25",
"@timestamp": "2018-02-12T10:57:26.614Z",
"severity": "INFO",
"path": "/var/www/html/php-app/application/storage/logs/laravel-2018-02-12.log",
"message": "[2018-02-12 10:57:25] development.INFO: home page  ",
"env": "development"

但对于Multiline消息(即带有堆栈跟踪的消息),它为每一行生成如下所示 .

"@timestamp" => 2018-02-12T10:56:47.785Z,
"path" => "/var/www/html/php-app/application/storage/logs/laravel-2018-02-12.log",
"message" => "#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(149): Illuminate\\\\Foundation\\\\Http\\\\Middleware\\\\ValidatePostSize->handle(Object(Illuminate\\\\Http\\\\Request), Object(Closure))",
"tags" => [
    [0] "_grokparsefailure"
],

我也试过了 multi-line 过滤器 . 多行错误日志仍然没有成功 . 我需要一个适合多行和单行错误消息的解决方案 .

请帮我找到适合单行和多行错误日志的正确grok配置 .

1 回答

  • 1

    最后!我得到了解决问题的方法 . 并发布logstash配置,这对将来的其他人有用 .

    input {
        file {
            path => '/var/www/html/php-app/application/storage/logs/laravel-*.log'
            start_position => 'beginning'
            ignore_older => 0
            codec => multiline { pattern => "\[[\d]{4}" negate => "true" what => "previous" }
        }
    }
    
    filter {
        grok {
            match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: %{DATA:message}" }
        }
    }
    
    output {
        elasticsearch {
            hosts => [ 'localhost:9200' ]
            index => "laravel-%{+YYYY-MM-dd}"
        }
        stdout {
            codec => rubydebug
        }
    }
    

    @baudsp感谢您帮助我解决这个问题 .

相关问题