Keywords: Kafka - Google Cloud Platform - Technical issue - Other
Description:
Hello, I’m using elasticsearch sink connector to ingest data to my elasticsearch from secured Apache Kafka. The Kafka cluster is deployed using Bitnami image in GCP. Using the following security configuration (see below) with KSQL server and Golang SDK (Sarama), I have no issue connecting, publish and consume the data. However with elasticsearch sink connector, using the same security configuration it will spit the following message infinitely.
WARN [Consumer clientId=connector-consumer-elasticsearch-sink-0, groupId=connect-elasticsearch-sink] Bootstrap broker kafka-cluster-sk-kafka-1.c.data-analytics-mizal.internal:9092 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1019)
And the Kafka server log will stated
Failed authentication with /10.148.0.2 (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)
I followed the confluent documentation to also specifying the security configuration in the connector configuration properties itself and prefix it with the ‘consumer’ but still the same failure.
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“user”
password=“xxxxxxx”;
Does anyone have any idea how to fix or debug this issue?