我有一个Elastic Search,Log Stash,Kibana和Filebeat的设置,其版本如下所示 .

1)弹性搜索:6.2.4 2)LogStash:6.2.4 3)kibana:6.2.4

我的Logstash管道如下:

sudo vim /etc/logstash/conf.d/02-beats-input.conf

input {
    beats {
        port => "5044"
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    }
}

filter {
  #If log line contains tab character followed by 'at' then we will tag that 
  entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }

  #Grokking Spring Boot's default log format
  grok {
    match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} % 
      {TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[(?<thread>[A-Za-z0- 
      9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:\s+(? 
      <logmessage>.*)"}
    }

  #grok {
    #  match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} 
    # %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(? 
    # <logmessage>.*)"}
  #}

  grok {
    match => { "source" => "/var/log/containers/%{DATA:pod_name}_% 
    {DATA:namespace}_%{GREEDYDATA:container_name}-%{DATA:container_id}.log" 
  }
  remove_field => ["source"]
}

sudo vim /etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

弹性搜索,Kibana,Logstash在一个VM中运行,其中应用程序和Filebeat在不同的VM上运行 .

目前这两个虚拟机之间存在日期差异,我还需要修复 .

The latest log on kibana discover is as below with a specific time stamp.

message:
  {"log":"TACACS+: No port assigned for host, \"XX.XX.XX.XX\". Using default 
  port 49 instead.\n","stream":"stdout","time":"**2018-05- 
  17T00:58:09.401752809Z**"}
  @timestamp:
  May 16th 2018, 17:58:09.408

The latest log at the application as below with a specific time stamp.

{"log":"TACACS+: No port assigned for host, \"XX.XX.XX.XX\".  Using default 
port 49 instead.\n","stream":"stdout","time":"**2018-05- 
17T06:06:44.365607578Z**"}

如果你看到上面两个日志,很明显kibana会显示日志有一些延迟,特别是在上面的情况下,它有5个小时的延迟 . 我也看到延迟不断增加 . 我看到了kibana上的所有日志 . 问题是我看到的延迟 .

有人可以帮我理解这种行为吗?这是因为两个虚拟机之间的时间差异?两者都在PDT时区 . 日志大小应该足够小,我不希望任何限制开始 .

如果您需要任何其他详细信息,请与我们联系 .