我使用'Postgresql'数据库设置环境,使用'Debezium'连接器与Kafka Connect和Kafka . Kafka有多个实例(3)正在运行,并且它配置了Zookeeper(3) . 整个管道中的连接正在工作,但是根据Debezium的文档,没有根据表中的表自动创建主题 . 数据库 . 如果是某个模式中的表A和表B,我假设在Kafka中隐式创建了2个主题 . RUNNING,连接器和任务的状态是RUNNING,下面提到的是我为连接器所做的配置,
{
"name": "geo-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": <dbHostName>,
"database.port": <dbPort>,
"database.user": <dbUser>,
"database.password":<dbPassword> ,
"database.dbname" : <dbName>,
"database.server.name": <logicalName>,
"database.history.kafka.bootstrap.servers":<>,
"database.history.kafka.topic": "schema-changes.inventory",
"plugin.name":"wal2json",
"config.storage.replication.factor": "3",
"offset.storage.replication.factor" : "3",
"auto.create.topics.enable" : "true",
"snapshot.mode" : "always"
}
}
我在Connect日志中看到的错误是,
2018-08-09 15:28:50,409 - DEBUG [KafkaBasedLog工作线程 - kconnect-offsets:Fetcher @ 199] - [Consumer clientId = consumer-1,groupId = 1]发送READ_UNCOMMITTED IncrementalFetchRequest(toSend =(),toForget =( ),暗示=(kconnect-offsets-10,kconnect-offsets-4,kconnect-offsets-16,kconnect-offsets-7,kconnect-offsets-19,kconnect-offsets-13,kconnect-offsets-22,kconnect-offsets -1))经纪人kafka-02.hotel02.pro06.eu.idealo.com:9092(id:2002 rack:pro06)2018-08-09 15:28:50,465 - DEBUG [kafka-producer-network-thread | producer-6:NetworkClient $ DefaultMetadataUpdater @ 927] - [Producer clientId = producer-6]向节点kafka-01.hotel02.pro05.eu.idealo.com发送元数据请求(type = MetadataRequest,topics = dbserver1.public.spatial_ref_sys) :9092(id:2004 rack:pro05)2018-08-09 15:28:50,467 - WARN [kafka-producer-network-thread | producer-6:NetworkClient $ DefaultMetadataUpdater @ 882] - [Producer clientId = producer-6]获取具有相关ID 23856的元数据时出错: 2018-08-09 15:28:50,467 - DEBUG [ kafka-producer-network-thread | producer-6:Metadata @ 270] - 更新了集群元数据版本23852到Cluster(id = BwqlZApfT-ygzWr_wPcdng,nodes = [kafka-03.hotel02.pro05.eu.idealo.com:9092(id:2003 rack:pro05), kafka-01.hotel02.pro05.eu.idealo.com:9092(id:2004 rack:pro05),kafka-02.hotel02.pro06.eu.idealo.com:9092(id:2002 rack:pro06)],partition = [])
1 回答
您在日志中看到的消息是警告,而不是错误 . 您能否尝试使用
kafka-topics.sh
实用程序列出可用的主题?