Can't connect to kafka cluster on GCP from App Engine

[ehsmeng]

Hi,

We’re running Bitnami Kafka cluster on GCP, and try to use it from App Engine.
(django pyhon3.7 with kafka-python 1.4.7)
We can use it from other Compute Engine instances, both with local hostname and private ip.
Django/App engine seems to connect to the ip, but then we have a tedious loop

2020-02-07 09:05:29 api[20200207t095108]  WARNING:kafka.conn:DNS lookup failed for kafka-cluster-1-kafka-2:9092, exception was [Errno -2] Name or service not known. Is your advertised.listeners (called advertised.host.name before Kafka 9) correct and resolvable?
2020-02-07 09:05:29 api[20200207t095108]  ERROR:kafka.conn:DNS lookup failed for kafka-cluster-1-kafka-2:9092 (AddressFamily.AF_UNSPEC)

We’re not giving kafka-python any plaintext hostnames, only ip. This must be smth the server has returned?

What are we supposed to do here?

Thanks!
Marcus

Software
Java (1.8.232)
Apache ZooKeeper (3.5.6)
Gonit (0.2.2)
Apache Kafka (2.4.0)
Bnsupport Tool (0.8.2)

[vikram-bitnami]

Hi @ehsmeng

We have some documentation on how to connect to the cluster. You mentioned that you’re using the Bitnami Kafka cluster, so I assume you are using the multi-tier version.

You can find documentation on how to connect to Kafka from the same or different network at https://docs.bitnami.com/google-templates/infrastructure/kafka/administration/connect-remotely/ and how to connect to individual nodes at https://docs.bitnami.com/google-templates/infrastructure/kafka/administration/connect-nodes/.

Hope this helps,

Vikram