用于日志的自定义Grok模式-Java 学习之路

所以这是我的日志示例：

23:28:32.226 WARN  [MsgParser:ListProc-Q0:I5]   Parsing error
Error mapping the fieldAdditional Information: 

    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:178)
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:96)
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:119)
    at com.authentic.mapper.parsing.MsgParser.processReadEnumeration(MsgParser.java:339)
    at com.authentic.mapper.parsing.MsgParser.parseIncomingMessageBody(MsgParser.java:295)
    at com.authentic.mapper.MapperMgr.parseMsg(MapperMgr.java:1033)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.parseMessage(AbstractConnectionHandler.java:4408)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.plainMessageReceivedEvent(AbstractConnectionHandler.java:2031)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.messageReceivedEvent(AbstractConnectionHandler.java:1911)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:801)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:282)
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:261)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.processEventQueue(AbstractConnectionHandler.java:4110)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.access$100(AbstractConnectionHandler.java:320)
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler$ConnectionHandlerRunner.execute(AbstractConnectionHandler.java:416)
    at com.authentic.architecture.actions.ListProcessor.suspend(ListProcessor.java:1130)
    at com.authentic.architecture.actions.ListProcessor.run(ListProcessor.java:775)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: "^123"
    at java.lang.NumberFormatException.forInputString(Unknown Source)
    at java.lang.Integer.parseInt(Unknown Source)
    at java.lang.Integer.parseInt(Unknown Source)
    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:170)
    ... 17 more

我必须将此日志解析为以下字段：timestamp，log-level，logger，msg，stacktrace .

我使用过多线过滤器：

multiline {
pattern => "%{TIME:timestamp}"
negate => true
what => “previous”
}

和我在grok过滤器中使用的模式：

match=>{"message"=>"%{TIME:timestamp} %{LOGLEVEL:loglevel} \s*\[%{DATA:logger}\]\s*%{GREEDYDATA:msg}\n*(?<stacktrace>(.|\r|\n)*)"}

我用http://grokconstructor.appspot.com/do/match检查了它 . 但是为stacktrace字段得到了这个匹配错误 .
enter image description here

请提出一些建议 . 提前致谢 .

1 回答

1
如果要匹配整个堆栈跟踪，则需要多行过滤器 . 这个多行过滤器应该适合你：
```
codec => multiline {
        pattern => "^%{TIME} "
        negate => true
        what => previous
    }
```
说明：每个不以时间戳开头的行（如23：28：32.226）将作为上一行的一部分进行重新定位 . 另见docs关于处理多线 .

现在你的模式 . 以下为我工作：
```
%{TIME:timestamp} %{LOGLEVEL:loglevel}  \[%{DATA:logger}\]   %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)
```
非常自我解释，我希望：使用 \[ 和 \] ， \n 转换为[和]的大括号以匹配换行符 . 还要注意条目之间的空格 .

对于最后一部分（stacktrace），另请参阅this question，了解如何匹配包括换行符在内的所有内容 .

完整配置可能如下所示：
```
input {
  file {
    path => "/var/log/yourlog.log"
    start_position => "beginning"
    codec => multiline {
        pattern => "^%{TIME} "
        negate => true
        what => previous
    }
  }
}
filter {
  grok {
    match => [ "message", "%{TIME:timestamp} %{LOGLEVEL:loglevel}  \[%{DATA:logger}\]   %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)" ]
  }
}
```
结果http://grokconstructor.appspot.com：
回复于 2024-05-16T14:03:52+08:00

用于日志的自定义Grok模式

1 回答

相关问题