我正在尝试将Modsecurity OSSEC日志添加到ELK . 使用带有以下prospector的filebeat将日志发送到ELK服务器:

-
  paths:      
    - /var/ossec/logs/alerts/alerts.log

  document_type: ossec-log
  multiline:
    pattern: '^\*\*'
    negate: true   
    match: before
    max_lines: 20
    timeout: 5s

在ELK服务器上,我在conf.d中创建了一个自定义logstash配置,其中包含以下内容:

filter {
  if [type] == "ossec-log" {
    grok {
      match => ["message", "(?m)\*\* Alert %{DATA:timestamp_seconds}:%{SPACE}%{WORD}?%{SPACE}\- %{DATA:ossec_group},\n%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{HOSTNAME}\-\>%{DATA:reporting_source}\nRule: %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> '%{DATA:signature}'\nSrc IP: %{IP:src_ip}\n\[modsecurity\] \[client %{IP:src_ip}\] \[domain %{DATA:FQDN}\] \[%{NONNEGINT:DenyCode}\] \[%{DATA:LogLocation}\]  \[file \"%{DATA:RuleConfig}\"\] \[line \"%{NONNEGINT:LineNumber}\"\] \[id \"%{NONNEGINT:RuleID}\"\] \[rev \"%{NONNEGINT:RevID}\"\] \[msg \"%{DATA:message}\"\] \[severity \"%{DATA:severity_text}\"\]%{GREEDYDATA}"]
    }
  }
}

它应该匹配的数据结构如下:

** Alert 1471539036.4425320: mail  - modsecurity,access_denied,
2016 Aug 18 18:50:36 myserver->/var/log/httpd/audit_log
Rule: 397989 (level 7) -> 'Atomicorp.com WAF Rules: MSIE 6.0 detected (Disable if you want to allow MSIE 6) '
Src IP: 127.127.127.127
[modsecurity] [client 127.127.127.127] [domain mydomain.org] [403] [/apache/20160818/20160818-1850/20160818-185034-YWhIuJT7g-sAAHaRtlsAAAAv]  [file "/etc/httpd/modsecurity.d/20_asl_useragents.conf"] [line "369"] [id "397989"] [rev "1"] [msg "Atomicorp.com WAF Rules: MSIE 6.0 detected (Disable if you want to allow MSIE 6)"] [severity "WARNING"] Access denied with code 403 (phase 2). Match of "rx (MS Web Services Client Protocol|WormlyBot|webauth@cmcm\\.com)" against "REQUEST_HEADERS:User-Agent" required.

这在grokdebugger中有效,但不知何故它没有在ELK中解析 . 日志项以ELK显示,具有正确的类型(ossec-log),但它们似乎未被处理 . 列出了日志条目,但整个行都转储在“message”中,标记名为“beats_input_codec_plain_applied,_grokparsefailure”,并且没有我在grok中定义的字段 .

我究竟做错了什么?

编辑190816:我通过将匹配括号切换为大括号并将逗号切换为=>(match => {“message”=>“grok pattern”)来修复它 . 后来我还将filebeat配置匹配指令从'before'切换到'after' .