# RASA Core integration issue with Kafka event broker - "Error - NoBrokersAvailable"

**URL:** <https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678>\
**Category:** Rasa Open Source\
**Created:** [August 17, 2020, 5:36pm UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678 "2020-08-17T17:36:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![hari](https://avatars.discourse-cdn.com/v4/letter/h/cdc98d/32.png) [@hari](https://forum.rasa.com/u/hari)\
**Post date:** [August 17, 2020, 5:36pm UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/1 "2020-08-17T17:36:21Z")

</div>

Hello,

I’m receiving “kafka.errors.NoBrokersAvailable: NoBrokersAvailable” error while trying to connect to Kafka event broker from RASA Core server. The RasaX server seems to connect fine.

So basically the consumer service (RasaX) can connect to Kafka broker successfully but the producer server (Rasa Core) is failing to connect to Kafka broker.

The endpoint and environment variables configuration is the same for both Rasa Core and RasaX servers.

Attached Rasa Core server (primary server) log, RasaX server log and RasaX server debug logs. [RasaCoreLog.txt](https://forum.rasa.com/uploads/short-url/2pWdJ1tqinw1WvtnuaycTPbCrlO.txt) (17.1 KB) [RasaXServerDebugLog.txt](https://forum.rasa.com/uploads/short-url/6cMGLxldBdSmLt5Ba1i5Ym3ylUQ.txt) (82.0 KB) [RasaXServerLog.txt](https://forum.rasa.com/uploads/short-url/v5L6KuAJaxXhlZivGGOtJaDMF30.txt) (2.0 KB)

Endpoint configuration:-

```
event_broker:
url: ${KAFKA_HOST}
sasl_username: ${SASL_USERNAME}
sasl_password: ${SASL_PASSWORD}
topic: ${KAFKA_TOPIC}
security_protocol: ${KAFKA_SECURITY_PROTOCOL}
type: kafka

```

RasaCore primary server & RasaX environment variables:-

```
- KAFKA_HOST=${KAFKA_HOST:-kafka:9092}
- KAFKA_TOPIC=${KAFKA_TOPIC:-rasa_core_events}
- KAFKA_SECURITY_PROTOCOL=${KAFKA_SECURITY_PROTOCOL:-SASL_PLAINTEXT}
- SASL_USERNAME=${SASL_USERNAME:-test}
- SASL_PASSWORD=${SASL_PASSWORD:-test}

```

Kafka docker container:-

```
kafka:
     image: confluentinc/cp-kafka:5.3.1
     container_name: gds_kafka
     ports:
         - "9092:9092"
     environment:
         KAFKA_BROKER_ID: 1
         KAFKA_ADVERTISED_HOST_NAME: ${KAFKA_ADVERTISED_HOST_NAME:-kafka}
         KAFKA_CREATE_TOPICS: "rasa_core_events:1:1"
         KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181           
         KAFKA_SASL_USERNAME: test
         KAFKA_SASL_PASSWORD: test
         KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_NAME:-kafka}:9092
         KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

```

Any suggestions would be greatly appreciated.

Thanks Hari

---

<div class="post-metadata">

**Author:** ![stephens](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/stephens/32/1632_2.png) [@stephens](https://forum.rasa.com/u/stephens)\
**Post date:** [August 24, 2020, 5:24am UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/2 "2020-08-24T05:24:55Z")

</div>

Are you sure the value for `KAFKA_HOST` is set for the `rasa-production` container?

---

<div class="post-metadata">

**Author:** ![hari](https://avatars.discourse-cdn.com/v4/letter/h/cdc98d/32.png) [@hari](https://forum.rasa.com/u/hari)\
**Post date:** [August 24, 2020, 2:45pm UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/3 "2020-08-24T14:45:01Z")

</div>

@stephens Thanks for your reply. The KAFKA\_HOST is set for rasa-production, rasa-worker and RasaX.

The problem is RASA X and RASA core are not configured to use the same security protocol. There is a mismatch between the code for the two components (RasaX security\_protocol=“PLAINTEXT” vs RasaCore security\_protocol=“SASL\_PLAINTEXT”).

Rasa core server is not working If Kafka server is configured with PLAINTEXT and RasaX is not working if Kafka server is configured with SASL\_PLAINTEXT.

This requires fix in code. It would be great if RASA team address this issue. The preferable fix is to add an else condition to RasaX consumer code to send security\_protocol =SASL\_PLAINTEXT which is secure than PLAINTEXT protocol for the Production environment.

Please note I’ve verified the latest image 0.32.0a1 and the mismatch is still there.

rasax\community\services\event\_consumers\kafka\_consumer.py

```
def _create_consumer(self) -> None:
	# noinspection PyPackageRequirements
	import kafka

	if "plaintext" in self.security_protocol.lower():
	    self.consumer = kafka.KafkaConsumer(
		self.topic,
		bootstrap_servers=self.host,
		security_protocol="PLAINTEXT",
		sasl_mechanism="PLAIN",
		sasl_plain_username=self.sasl_username,
		sasl_plain_password=self.sasl_password,
		ssl_check_hostname=False,
	    )
	elif "ssl" in self.security_protocol.lower():
	    self.consumer = kafka.KafkaConsumer(
		self.topic,
		bootstrap_servers=self.host,
		security_protocol="SSL",
		ssl_cafile=self.ssl_cafile,
		ssl_certfile=self.ssl_certfile,
		ssl_keyfile=self.ssl_keyfile,
		ssl_check_hostname=self.ssl_check_hostname,
	    )
	else:
	    raise ValueError(
		f"Cannot initialise `kafka.KafkaConsumer` "
		f"with security protocol '{self.security_protocol}'."
	    )

```

rasa\core\brokers\kafka.py

```
def _create_producer(self) -> None:
	import kafka

	if self.security_protocol == "SASL_PLAINTEXT":
	    self.producer = kafka.KafkaProducer(
		bootstrap_servers=[self.host],
		value_serializer=lambda v: json.dumps(v).encode(DEFAULT_ENCODING),
		sasl_plain_username=self.sasl_username,
		sasl_plain_password=self.sasl_password,
		sasl_mechanism="PLAIN",
		security_protocol=self.security_protocol,
	    )
	elif self.security_protocol == "SSL":
	    self.producer = kafka.KafkaProducer(
		bootstrap_servers=[self.host],
		value_serializer=lambda v: json.dumps(v).encode(DEFAULT_ENCODING),
		ssl_cafile=self.ssl_cafile,
		ssl_certfile=self.ssl_certfile,
		ssl_keyfile=self.ssl_keyfile,
		ssl_check_hostname=False,
		security_protocol=self.security_protocol,
	    )
```

---

<div class="post-metadata">

**Author:** ![hari](https://avatars.discourse-cdn.com/v4/letter/h/cdc98d/32.png) [@hari](https://forum.rasa.com/u/hari)\
**Post date:** [August 31, 2020, 6:57pm UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/4 "2020-08-31T18:57:21Z")

</div>

@stephensI’ve manually updated consumer code in RasaX service container to support SASL\_PLAINTEXT protocol and then everything works fine. It is a simple fix to RasaX code. All we need is an elif condition below to support SASL\_PLAINTEXT protocol. It would be great if Rasa team consider this fix in next release. I’m not sure if I can submit a PR request as RasaX is not OSS. Please let me know if there is any way that I can submit a PR.

rasax\community\services\event\_consumers\kafka\_consumer.py

```auto
def _create_consumer(self) -> None:
	# noinspection PyPackageRequirements
	import kafka

	elif "sasl_plaintext" in self.security_protocol.lower():
	    self.consumer = kafka.KafkaConsumer(
		self.topic,
		bootstrap_servers=self.host,
		security_protocol="SASL_PLAINTEXT",
		sasl_mechanism="PLAIN",
		sasl_plain_username=self.sasl_username,
		sasl_plain_password=self.sasl_password,
		ssl_check_hostname=False,
	    )

```

---

<div class="post-metadata">

**Author:** ![tyd](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tyd/32/3030_2.png) [@tyd](https://forum.rasa.com/u/tyd)\
**Post date:** [September 1, 2020, 9:16am UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/5 "2020-09-01T09:16:30Z")

</div>

Hi @hari. Would you be able to [create an issue for this on GitHub](https://github.com/RasaHQ/rasa/issues/new?assignees=&labels=area%3Arasa-oss+%3Aferris_wheel%3A%2Ctype%3Abug+%3Abug%3A&template=bug_report.md&title=), so we can get it fixed?

---

<div class="post-metadata">

**Author:** ![hari](https://avatars.discourse-cdn.com/v4/letter/h/cdc98d/32.png) [@hari](https://forum.rasa.com/u/hari)\
**Post date:** [September 1, 2020, 3:37pm UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/6 "2020-09-01T15:37:25Z")

</div>

@tyd submitted an issue on GitHub and here is the link [Update RasaX Kafka event broker consumer code to support SSAL\_PLAINTEXT protocol · Issue #6539 · RasaHQ/rasa · GitHub](https://github.com/RasaHQ/rasa/issues/6539)

I created the issue by following your link and it is showing under rasa issues. Please let me know if I have to submit on different page for RasaX. Please note the code needs to be fixed in RasaX.

Thanks Hari

---

<div class="post-metadata">

**Author:** ![tyd](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tyd/32/3030_2.png) [@tyd](https://forum.rasa.com/u/tyd)\
**Post date:** [September 2, 2020, 8:05am UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/7 "2020-09-02T08:05:02Z")

</div>

@hari Thanks! That works. All Rasa Open Source and Rasa X issues from the community belong in that repo

---

<div class="post-metadata">

**Author:** ![hari](https://avatars.discourse-cdn.com/v4/letter/h/cdc98d/32.png) [@hari](https://forum.rasa.com/u/hari)\
**Post date:** [September 2, 2020, 1:02pm UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/8 "2020-09-02T13:02:05Z")

</div>

@tyd Thank you. Hope it will be fixed soon.

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [September 29, 2020, 8:36am UTC](https://forum.rasa.com/t/rasa-core-integration-issue-with-kafka-event-broker-error-nobrokersavailable/32678/9 "2020-09-29T08:36:19Z")

</div>

Just an update, This was fixed in version 0.32.2
