首页 文章

Kafka与Debezium联系

提问于
浏览
0

我使用'Postgresql'数据库设置环境,使用'Debezium'连接器与Kafka Connect和Kafka . Kafka有多个实例(3)正在运行,并且它配置了Zookeeper(3) . 整个管道中的连接正在工作,但是根据Debezium的文档,没有根据表中的表自动创建主题 . 数据库 . 如果是某个模式中的表A和表B,我假设在Kafka中隐式创建了2个主题 . RUNNING,连接器和任务的状态是RUNNING,下面提到的是我为连接器所做的配置,

{
   "name": "geo-connector",
   "config": {
    "connector.class":     "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": <dbHostName>,
    "database.port": <dbPort>,
    "database.user": <dbUser>,
    "database.password":<dbPassword> ,
    "database.dbname" : <dbName>,
    "database.server.name": <logicalName>,
    "database.history.kafka.bootstrap.servers":<>,
    "database.history.kafka.topic": "schema-changes.inventory",
    "plugin.name":"wal2json",
    "config.storage.replication.factor": "3",
    "offset.storage.replication.factor" : "3",
    "auto.create.topics.enable" : "true",
    "snapshot.mode" : "always"
  }
}

我在Connect日志中看到的错误是,

2018-08-09 15:28:50,409 - DEBUG [KafkaBasedLog工作线程 - kconnect-offsets:Fetcher @ 199] - [Consumer clientId = consumer-1,groupId = 1]发送READ_UNCOMMITTED IncrementalFetchRequest(toSend =(),toForget =( ),暗示=(kconnect-offsets-10,kconnect-offsets-4,kconnect-offsets-16,kconnect-offsets-7,kconnect-offsets-19,kconnect-offsets-13,kconnect-offsets-22,kconnect-offsets -1))经纪人kafka-02.hotel02.pro06.eu.idealo.com:9092(id:2002 rack:pro06)2018-08-09 15:28:50,465 - DEBUG [kafka-producer-network-thread | producer-6:NetworkClient $ DefaultMetadataUpdater @ 927] - [Producer clientId = producer-6]向节点kafka-01.hotel02.pro05.eu.idealo.com发送元数据请求(type = MetadataRequest,topics = dbserver1.public.spatial_ref_sys) :9092(id:2004 rack:pro05)2018-08-09 15:28:50,467 - WARN [kafka-producer-network-thread | producer-6:NetworkClient $ DefaultMetadataUpdater @ 882] - [Producer clientId = producer-6]获取具有相关ID 23856的元数据时出错: 2018-08-09 15:28:50,467 - DEBUG [ kafka-producer-network-thread | producer-6:Metadata @ 270] - 更新了集群元数据版本23852到Cluster(id = BwqlZApfT-ygzWr_wPcdng,nodes = [kafka-03.hotel02.pro05.eu.idealo.com:9092(id:2003 rack:pro05), kafka-01.hotel02.pro05.eu.idealo.com:9092(id:2004 rack:pro05),kafka-02.hotel02.pro06.eu.idealo.com:9092(id:2002 rack:pro06)],partition = [])

1 回答

  • 0

    您在日志中看到的消息是警告,而不是错误 . 您能否尝试使用 kafka-topics.sh 实用程序列出可用的主题?

相关问题