有一些方法可以收集docker或k8s containter日志：
- 使用像gelf，fluent等流日志驱动程序，但是这不能使用docker logs命令来调试，所以它不是解决问题的好方法 . Send log to multiple log drivers Enhance local logging / remote logging
由于docker的日志驱动程序不支持多行，因此/var/lib/docker/containter//.json中的日志是逐行的log driver should support multiline .
- 使用日志收集器程序，如filebeat，fluend，collector-sidecar，logstash来解析/ var / lib / docker / containter //中的docker json文件 . 但我仍然找不到收集日志的好方法 .
经过几天的搜索，我认为流利是解决这个问题的最佳工具，并找到两种方法来解决问题 . 但我仍然解决我的问题 .
- 使用fluent-plugin-detect-exceptions gelf，但这仍然是两条消息 . Docker logs (Java stack trace that are long) NOT meaningful in Splunk Is available to concat stack trace and previous line with error message
原始日志：
{"log":"2018-04-19 14:19:57,915 INFO [FixedTimeScheduler] com.testjavatest.fastdemo.ws.WebSocketClientManager: send ws message -{TAB} for envirnem -{{\"res\":\"heartbeat\"}}\n","stream":"stdou
t","time":"2018-04-19T06:19:57.916259717Z"}
{"log":"2018-04-19 14:19:57,915 INFO [FixedTimeScheduler] com.testjavatest.fastdemo.ws.WebSocketClientManager: send ws message -{TAB} for envirnem -{{\"res\":\"heartbeat\"}}\n","stream
":"stdout","time":"2018-04-19T06:19:57.916265977Z"}
{"log":"2018-04-19 14:20:43,446 ERROR [FixedTimeScheduler] com.testjavatest.fastdemo.task.JobTask: Connect to cloud.testjavatest.com:5002 [cloud.testjavatest.com/10.111.2.77] failed: Connection timed o
ut (Connection timed out)\n","stream":"stdout","time":"2018-04-19T06:20:43.448436321Z"}
{"log":"org.apache.http.conn.HttpHostConnectException: Connect to cloud.testjavatest.com:5002 [cloud.testjavatest.com/10.111.2.77] failed: Connection timed out (Connection timed out)\n","stream":"s
tdout","time":"2018-04-19T06:20:43.448475801Z"}
{"log":"\u0009at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)\n","stream":"stdout","time":"2018-04-19T06:20:43.4484860
6Z"}
{"log":"\u0009at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)\n","stream":"stdout","time":"2018-04-19T06:20:43.448492586
Z"}
{"log":"\u0009at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)\n","stream":"stdout","time":"2018-04-19T06:20:43.448498085Z"}
{"log":"\u0009at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)\n","stream":"stdout","time":"2018-04-19T06:20:43.448503302Z"}
{"log":"\u0009at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)\n","stream":"stdout","time":"2018-04-19T06:20:43.4485085Z"}
{"log":"\u0009at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)\n","stream":"stdout","time":"2018-04-19T06:20:43.448527373Z"}
{"log":"\u0009at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\n","stream":"stdout","time":"2018-04-19T06:20:43.448532363Z"}
{"log":"\u0009at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)\n","stream":"stdout","time":"2018-04-19T06:20:43.44853704Z"}
{"log":"\u0009at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)\n","stream":"stdout","time":"2018-04-19T06:20:43.448541864Z"}
{"log":"\u0009at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)\n","stream":"stdout","time":"2018-04-19T06:20:43.448546452Z"}
{"log":"\u0009at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)\n","stream":"stdout","time":"2018-04-19T06:20:43.448551288Z"}
{"log":"\u0009at com.testjavatest.httpclient.MyHttpClient.send(MyHttpClient.java:308)\n","stream":"stdout","time":"2018-04-19T06:20:43.448555115Z"}
{"log":"\u0009at com.testjavatest.httpclient.MyHttpClient.send(MyHttpClient.java:264)\n","stream":"stdout","time":"2018-04-19T06:20:43.448559028Z"}
{"log":"\u0009at com.testjavatest.httpclient.MyHttpClient.recvBytes(MyHttpClient.java:470)\n","stream":"stdout","time":"2018-04-19T06:20:43.448563048Z"}
{"log":"\u0009at com.testjavatest.httpclient.MyHttpClient.recvString(MyHttpClient.java:474)\n","stream":"stdout","time":"2018-04-19T06:20:43.448567524Z"}
{"log":"\u0009at com.testjavatest.httpclient.MyHttpClient.recvJSON(MyHttpClient.java:483)\n","stream":"stdout","time":"2018-04-19T06:20:43.448571872Z"}
{"log":"\u0009at com.testjavatest.fastdemo.task.JobTask.run(JobTask.java:70)\n","stream":"stdout","time":"2018-04-19T06:20:43.448576282Z"}
{"log":"\u0009at com.testjavatest.fastdemo.task.timer.FixedTimeScheduler.run(FixedTimeScheduler.java:43)\n","stream":"stdout","time":"2018-04-19T06:20:43.44858108Z"}
{"log":"Caused by: java.net.ConnectException: Connection timed out (Connection timed out)\n","stream":"stdout","time":"2018-04-19T06:20:43.448585665Z"}
{"log":"\u0009at java.net.PlainSocketImpl.socketConnect(Native Method)\n","stream":"stdout","time":"2018-04-19T06:20:43.448590045Z"}
{"log":"\u0009at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)\n","stream":"stdout","time":"2018-04-19T06:20:43.448596151Z"}
{"log":"\u0009at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)\n","stream":"stdout","time":"2018-04-19T06:20:43.448600888Z"}
{"log":"\u0009at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)\n","stream":"stdout","time":"2018-04-19T06:20:43.448605225Z"}
{"log":"\u0009at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n","stream":"stdout","time":"2018-04-19T06:20:43.448609704Z"}
{"log":"\u0009at java.net.Socket.connect(Socket.java:589)\n","stream":"stdout","time":"2018-04-19T06:20:43.448614111Z"}
{"log":"\u0009at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)\n","stream":"stdout","time":"2018-04-19T06:20:43.448618537Z"}
{"log":"\u0009at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)\n","stream":"stdout","time":"2018-04-19T06:20:43.4486230
34Z"}
{"log":"\u0009... 17 more\n","stream":"stdout","time":"2018-04-19T06:20:43.448627326Z"}
{"log":"2018-04-19 14:20:57,934 INFO [FixedTimeScheduler] com.testjavatest.fastdemo.config.WsJobTask: ws heartbeat run\n","stream":"stdout","time":"2018-04-19T06:20:57.934646438Z"}
gelf中的期望输出：
一个java事件是一个不是多行的 .
我的流利配置是：
<source>
@id fluentd-containers.log
@type tail
path "/var/lib/docker/containers/52b26345771ff24b76840a57f26808562aea0e917d72746d48aac6f9870b901b/52b26345771ff24b76840a57f26808562aea0e917d72746d48aac6f9870b901b-json.log"
pos_file "/var/log/fluentd-containers.log.pos"
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag "raw.docker.*"
format json
read_from_head true
<parse>
time_format %Y-%m-%dT%H:%M:%S.%NZ
@type json
time_type string
</parse>
</source>
<match raw.docker.**>
@id raw.docker
@type detect_exceptions
remove_tag_prefix "raw"
message "log"
stream "stream"
languages java, python
multiline_flush_interval 5
max_bytes 0
max_lines 0
</match>
<match docker.**>
@type copy
<store>
@type "gelf"
protocol "udp"
host "192.168.2.4"
port 12206
flush_interval 5s
<buffer>
flush_mode interval
retry_type exponential_backoff
flush_interval 5s
</buffer>
</store>
<store>
@type "stdout"
</store>
</match>
但这不行，有人可以帮我一把吗？
- 使用fluent-plugin-grok-parser fluent-plugin-concat gelf，但这仍然无法正常工作 . 帮助消息：Handle multiline with empty line from kubernetes/docker with --log-driver=json-file和Multiple grok patterns with multiline not parsing the log the same question消息如上所示，期望输出如上所述 .
我的流利配置是：
<source>
@id fluentd-containers.log
@type tail
from_encoding UTF-8
encoding UTF-8
path /var/lib/docker/containers/52b26345771ff24b76840a57f26808562aea0e917d72746d48aac6f9870b901b/52b26345771ff24b76840a57f26808562aea0e917d72746d48aac6f9870b901b-json.log
pos_file /var/log/fluentd-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag raw.docker.*
format json
read_from_head true
</source>
<filter raw.docker.**>
@type grep
regexp1 stream stdout
@type concat
key log
multiline_start_regexp /^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d+/
continuous_line_regexp /^\s+/
separator ""
flush_interval 3s
</filter>
<filter raw.docker.**>
@type parser
key_name log
inject_key_prefix log.
<parse>
@type multiline_grok
grok_failure_key grokfailure
<grok>
pattern /%{TIMESTAMP_ISO8601:log_time}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}\[%{DATA:threadname}\]%{SPACE}%{DATA:classname}%{SPACE}:%{SPACE}%{GREEDYDATA:log_message}/
</grok>
</parse>
</filter>
<match raw.docker.**>
@type copy
<store>
@type gelf
protocol udp
host 10.111.2.4
port 12204
flush_interval 5s
</store>
<store>
@type stdout
</store>
<store>
@type file
path /var/log/test
</store>
</match>
但这不行，有人可以帮我一把吗？谢谢 .
PS：我想知道有没有更好的方法来收集docker日志？