Not able to connect to Kafka cluster created from Azure marketplace

samanta.avijit 2018-04-04 05:38:48 UTC #1

Keywords: Kafka - Microsoft Azure - Technical issue - Other
Description:
I have created a simple 2 node Kafka cluster from Azure marketplace for my dev test. I am able to follow the instructions as given in - https://docs.bitnami.com/azure-templates/infrastructure/kafka/ and able to test producer/consumer successfully.

I am now trying to connect my .NET application to Kafka cluster for sending/receiving messages. But when I try to send the client application gets hanged.

I have enabled Azure firewall to allow inbound traffic to 9092, edited server.properties to advertise the public ip address for the testing broker node. The auth mechanism is "SASL_PlainText". Do I need to use username/password for to connect with my kafka cluster? or am I missing something else?

Key changes in server.properties

listeners=SASL_PLAINTEXT://0.0.0.0:9092
...
advertised.listeners=SASL_PLAINTEXT://[public ip address of the broker node]:9092
advertised.host.name=[public ip address of the broker node]
advertised.port=9092
...
zookeeper.connect=<private ip address of zookeeper node>:2181

With above changes I am able to test producer/consumer with commandline inside the cluster.

Please help

Not able to connect Azure Kafka cluster using Confluent .NET library

Using Confluent.net with bitnami kafka on Microsoft Azure

jsalmeron 2018-04-04 11:41:20 UTC #2

Hi,

So, with these changes, you can connect to your Kafka cluster using command line when using an another instance to the cluster. However, are you able to use command line from outside the cluster? Plus, from inside the cluster, are you able to access using the public IP?

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-04-06 09:02:14 UTC #3

Hi Javier, I am trying to use .NET application to access the cluster from outside and it is not working.Application just gets hanged up. I just tested if my cluster setup is fine or not using command line inside one of the node in the cluster. I need help from Bitnami to make my .NET application work with the Kafka cluster accessing from

Thanks,
Avi

jsalmeron 2018-04-10 11:11:30 UTC #4

Hi,

I've been tinkering with the official .NET library from kafka, and it looks like it could be an issue with the authentication. If you do not set any authentication, it seems hanged indeed. However, when trying to set jaas (just like the instance is configured with), I get the following:

Unhandled Exception: System.ArgumentException: Java JAAS configuration is not supported, see https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka for more information.
   at Confluent.Kafka.Impl.SafeConfigHandle.Set(String name, String value) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Impl/SafeConfigHandle.cs:line 116
   at Confluent.Kafka.Producer.<>c__DisplayClass23_0.<.ctor>b__2(KeyValuePair`2 kvp) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 282
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Confluent.Kafka.Producer..ctor(IEnumerable`1 config, Boolean manualPoll, Boolean disableDeliveryReports) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 282
   at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 1044
   at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 1069
   at Confluent.Kafka.Examples.SimpleProducer.Program.Main(String[] args) in /tmp/confluent-kafka-dotnet/examples/SimpleProducer/Program.cs:line 47

I've been using this library: https://github.com/confluentinc/confluent-kafka-dotnet

It looks like the alternative would be setting a kerberos mechanism, which I do not have much expertise. I think it is an issue in the library, so I contacted the official developers for advice. You can follow the status here: https://github.com/confluentinc/confluent-kafka-dotnet/issues/504

Hope it helps

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-04-10 12:00:26 UTC #5

Thanks, I was al so doubting on the same link. Will watch at github link

jsalmeron 2018-04-11 08:30:59 UTC #6

Hi,

I was able to make it work. For the client to work, you need to make all the brokers externally accessible and be able access them not using their IP address, but their hostnames. So this is what I did:

I added a public IP to all of the brokers' NIC using Azure portal configuration. Go to each nic, then open IP Settings and add a Public IP.
I added each hostname to my /etc/hosts

After that the client worked without issues.

Hope it helps

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-04-12 05:19:32 UTC #7

Thanks Javier.
I had already added public ip address to my broker, but tried to access them by IP address in stead of hostname. So, will try with hostname.
I need to know exactly what changes you have made in Kafka cluster configuration and "/etc/hosts". Please share that details with me.

jsalmeron 2018-04-12 07:43:50 UTC #8

Hi,

Actually, I did not change anything in the cluster, I just only added the entries for the different hosts in my /etc/hosts file. Example:

51.144.126.30 kafkatest-kafka-0
51.144.241.10 kafkatest-kafka-1

Apart from that, I nothing else.

Hope it helps

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-04-13 10:00:05 UTC #9

Thanks Javier. It may be very dummy question, but I would like to know in which node I need to change the "/etc/hosts" file. Should I update it for each broker node as well as zookeeper node with their respective IP and hostname?

jsalmeron 2018-04-13 10:11:02 UTC #10

Hi,

You need to change it in the one where the .net application is running.

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-04-13 11:06:09 UTC #11

I am running it from a windows machine. Is it possible?

jsalmeron 2018-04-16 07:38:19 UTC #12

Hi,

Yes, see this example: https://gist.github.com/zenorocha/18b10a14b2deb214dc4ce43a2d2e2992

Hope it helps

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-04-17 10:26:45 UTC #13

Thanks Javier. will test now and check

system 2018-05-01 10:26:55 UTC #14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

jota 2018-05-11 09:44:46 UTC #15

samanta.avijit 2018-05-11 09:45:00 UTC #16

Hi, I am using one Azure market place Bitnami Kafka cluster. I want to test producer/consumer application with official Kafka .NET library (https://github.com/confluentinc/confluent-kafka-dotnet). I can test messaging inside the cluster. But, I am unable to test from my .NET application running locally. I am already tested Broker node ip address and port no. 9092 to check if is it's accessible and open (with telnet) and it is fine. But from application I am facing issue.

I had earlier raised the same issue with - https://community.bitnami.com/t/not-able-to-connect-to-kafka-cluster-created-from-azure-marketplace/56631 and Javier(community.bitnami.com/users/jsalmeron) was working on that. He was able to do it successfully. So, I would like to know if it's possible to share the code or any steps which I am missing

jsalmeron 2018-05-14 07:28:57 UTC #17

Hi,

Did this work for your case?

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

samanta.avijit 2018-05-14 13:19:10 UTC #18

Hi Javier. I have figured out the issue. It is due to the authentication problem. I was not passing proper username/password to authenticate with my cluster and it is working now.

crhernandez 2018-05-15 07:38:59 UTC #19

Hi,

We are glad that you were able to fix the issue. We are marking the previous answer as “Solution” and this topic as “Closed".
If you have any other questions, please do not hesitate to let us know. Feel free to create a new topic referencing to this one if necessary.

Best regards,
Carlos R. Hernández

crhernandez 2018-05-15 07:39:04 UTC #20

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!