首页 文章

Apache Kafka日志保留配置问题或如何配置Kafka保留策略？

提问于
浏览 次
1

我正在使用kafka_2.12-1.1.0 . 以下是我的保留金额 . 我正在使用Win OS .

log.retention.ms=120000

log.segment.bytes=2800

log.retention.check.interval.ms=30000

我看到日志被翻转并生成* 0000.log到* 0069.log然后* 00103.log等然后我希望它删除旧文件但是当保留逻辑启动时我得到以下错误进程无法访问文件，因为它被另一个进程使用，kafka服务器崩溃，并没有从那里恢复 . 尝试删除00000000000000000000.log时会发生这种情况？你能让我知道检查它们是否是活动段的命令，或者为什么它试图删除它并崩溃 .

PFB日志文件，如果你想看到它 .

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 39

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 39 in 39 ms. 

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 69 

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 69 in 52 ms.

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 91

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 91 in 18 ms. 

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Found deletable segments with base offsets [0,69] due to retention time 120000ms breach (kafka.log.Log)

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 103 (kafka.log.ProducerStateManager)

[Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Rolled new log segment at offset 103 in 25 ms. (kafka.log.Log)

 [Log partition=myLocalTopic-0, dir=D:\tmp\kafka-logs] Scheduling log segment [baseOffset 0, size 2746] for deletion. (kafka.log.Log)

ERROR Error while deleting segments for myLocalTopic-0 in dir D:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log -> D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
.............   at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:98)
.......................................................
 ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for myLocalTopic-0 in dir D:\tmp\kafka-logs
Caused by: java.nio.file.FileSystemException: D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log -> D:\tmp\kafka-logs\myLocalTopic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

Click here for file list image

apache-kafka

1 回答

  • 0

    Apache Kafka为我们提供了以下保留策略

    • 基于时间的保留

    • 基于大小的保留

    对于 time based retention ，请尝试以下命令，

    用于设置保留时间的命令

    ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.ms=1680000

    用于删除主题级别保留时间的命令

    ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.ms

    对于 size based retention ，请尝试以下命令：设置保留大小：

    ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.bytes=104857600

    去除，

    ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.bytes
    回复于

相关问题