首页 文章

Storm kafka 警告消息有多个任务比分区一些任务将空闲

提问于
浏览 次
0

我有风暴拓扑，使用 KafkaSpout 接收来自 Kafka 经纪人的消息。但我没有看到喷口处理任何消息。在工作日志中，我找到了以下代码段。风暴 version-0.9.3

2017-12-13T18:43:49.598+0000 s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections
2017-12-13T18:43:49.600+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle
2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] Task [1/2] no partitions assigned
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] Deleted partition managers: []
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] New partition managers: []
2017-12-13T18:43:49.600+0000 s.k.ZkCoordinator [INFO] Task [1/2] Finished refreshing
2017-12-13T18:43:50.111+0000 s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections
2017-12-13T18:43:50.113+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}}
2017-12-13T18:43:50.113+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle
2017-12-13T18:43:50.113+0000 s.k.KafkaUtils [WARN] Task [1/2] no partitions assigned
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] Deleted partition managers: []
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] New partition managers: []
2017-12-13T18:43:50.113+0000 s.k.ZkCoordinator [INFO] Task [1/2] Finished refreshing

我不确定为什么我会收到这样的警告信息。我尝试使用 kafka-topic cli 验证 kafka 主题，但我找到了 3 个分区。

./kafka-topics.sh  --zookeeper xxx.xxx.xxx.xxx:2181 --topic topic-name  --describe
Topic:topic-name PartitionCount:3  ReplicationFactor:1 Configs:
  Topic: topic-name  Partition: 0  Leader: 2 Replicas: 2 Isr: 2
  Topic: topic-name  Partition: 1  Leader: 2 Replicas: 2 Isr: 2
  Topic: topic-name  Partition: 2  Leader: 2 Replicas: 2 Isr: 2

我在这里做错了什么或者可能是没有收到消息的可能问题？

apache-kafka apache-zookeeper apache-storm

1 回答

  • 0

    注意这些线条

    2017-12-13T18:43:49.600+0000 s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{partitionMap={}} 2017-12-13T18:43:49.600+0000 s.k.KafkaUtils [WARN] there are more tasks than partitions (tasks: 2; partitions: 0), some tasks will be idle

    它无法从 Zookeeper 中读取分区列表。使用 storm-kafka KafkaSpout 时，需要将其配置为连接到 Kafka 群集的 Zookeeper。您应该通过https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/ZkHosts.java#L33在 SpoutConfig 中进行设置。 zookeeper 连接字符串语法与您在终端中传递给 kafka-topics.sh 的语法相同。

    回复于

相关问题