我有kafka和zookeeper在一些本地docker容器中运行 .
我有一个node.js代码库,它使用node-rdkafka作为消费者连接到kafka . 我们称之为代码库 "the consumer"
消费者使用此处显示的代码连接到kafka:https://github.com/Blizzard/node-rdkafka/blob/master/examples/consumer-flow.md
尝试连接到现有的Kafka实例时,我得到以下输出:
consumer ready.{"name":"rdkafka#consumer-1"}
但是接收和处理消息的代码永远不会触发,或者在开始触发之前需要几分钟(有时显然是5到10分钟) .
为了确保生成消息,我使用kafka附带的脚本 kafka-console-consumer.sh
来观察消息流 . 果然,PLENTY数据即将到来 .
似乎连接有时会失败,有时需要很长时间才能连接 .
这是这段时间内的kafka日志:
kafka_1 | [2017-04-27 20:55:37,963] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
kafka_1 | [2017-04-27 20:55:48,576] WARN Attempting to send response via channel for which there is no open connection, connection id 1 (kafka.network.Processor)
kafka_1 | [2017-04-27 20:55:53,185] INFO [GroupCoordinator 1001]: Preparing to restabilize group video-caption-consumer with old generation 24 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 20:56:18,577] INFO [GroupCoordinator 1001]: Stabilized group video-caption-consumer generation 25 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 20:56:18,626] INFO [GroupCoordinator 1001]: Assignment received from leader for group video-caption-consumer for generation 25 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 20:58:24,951] INFO [GroupCoordinator 1001]: Preparing to restabilize group video-caption-consumer with old generation 25 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 20:58:51,643] INFO [GroupCoordinator 1001]: Stabilized group video-caption-consumer generation 26 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 20:58:51,648] INFO [GroupCoordinator 1001]: Assignment received from leader for group video-caption-consumer for generation 26 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 21:01:14,231] INFO [GroupCoordinator 1001]: Preparing to restabilize group video-caption-consumer with old generation 26 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 21:01:16,414] INFO [GroupCoordinator 1001]: Stabilized group video-caption-consumer generation 27 (kafka.coordinator.GroupCoordinator)
kafka_1 | [2017-04-27 21:01:16,419] INFO [GroupCoordinator 1001]: Assignment received from leader for group video-caption-consumer for generation 27 (kafka.coordinator.GroupCoordinator)
它只是无限地重复这种模式 .
想法?
为了它的 Value ,我也使用了kakfa节点模块一段时间了;有类似的问题,虽然我不能确定是否同样的问题 .
1 回答
只要没有人有一个好的答案,我会分享我的解决方案......
最终我停止在一个docker容器中运行kafka,这似乎是原因 . 在The Quick Start Guide之后,我改为旧学校 .
这解决了我的问题,所以问题可能不是
node-rdkafka
的错 . 事实上,我们已经成功运行了node-rdkafka
已经有近一年的时间了,没有任何问题 .最后,我没有努力,令人满意的答案,只有怀疑和许多痛苦的回忆试图让 Kafka 在集装箱里工作 . 当我最终放弃了码头化的 Kafka 时,这是一种甜蜜的解脱 . 我的一些同事似乎只分享了我的问题 .