Cannot connect rasax to AWS RabbitMQ

I’m facing some issues when i try to connect rasax to AWS RabbitMQ, rasa event service can create new Broker in AWS RabbitMQ but rasa-worker and rasa-production cannot connect to it. I’m using rasax helm chart, here is the logs from rasa-worker and rasa-production:

 2021-03-11 15:06:09 DEBUG    sanic_jwt.configuration  - validating provided secret
2021-03-11 15:06:09 DEBUG    sanic_jwt.configuration  - validating keys (if needed)
2021-03-11 15:06:09 DEBUG    sanic_jwt.configuration  - loading secret and/or keys (if needed)
2021-03-11 15:06:09 DEBUG    rasa.core.utils  - Available web server routes:
/conversations/<conversation_id:path>/messages     POST                           add_message
/conversations/<conversation_id:path>/tracker/events POST                           append_events
/auth                                              OPTIONS                        auth_bp.AuthenticateEndpoint
/auth/me                                           OPTIONS                        auth_bp.RetrieveUserEndpoint
/auth/verify                                       OPTIONS                        auth_bp.VerifyEndpoint
/webhooks/rasa                                     GET                            custom_webhook_RasaChatInput.health
/webhooks/rasa/webhook                             POST                           custom_webhook_RasaChatInput.receive
/webhooks/rest                                     GET                            custom_webhook_RestInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_RestInput.receive
/model/test/intents                                POST                           evaluate_intents
/model/test/stories                                POST                           evaluate_stories
/conversations/<conversation_id:path>/execute      POST                           execute_action
/domain                                            GET                            get_domain
/                                                  GET                            hello
/model                                             PUT                            load_model
/model/parse                                       POST                           parse
/conversations/<conversation_id:path>/predict      POST                           predict
/conversations/<conversation_id:path>/tracker/events PUT                            replace_events
/conversations/<conversation_id:path>/story        GET                            retrieve_story
/conversations/<conversation_id:path>/tracker      GET                            retrieve_tracker
/status                                            GET                            status
/model/predict                                     POST                           tracker_predict
/model/train                                       POST                           train
/conversations/<conversation_id:path>/trigger_intent POST                           trigger_intent
/model                                             DELETE                         unload_model
/version                                           GET                            version
2021-03-11 15:06:09 INFO     root  - Starting Rasa server on http://localhost:5005
2021-03-11 15:06:09 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2021-03-11 15:06:09 INFO     root  - Enabling coroutine debugging. Loop id 93909152029232.
2021-03-11 15:06:09 DEBUG    root  - Could not load interpreter from 'None'.
2021-03-11 15:06:09 INFO     rasa.core.brokers.pika  - Connecting to RabbitMQ ...
2021-03-11 15:06:10 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:15 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:20 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:25 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:30 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:35 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:40 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:45 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:50 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:55 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:00 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:05 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:10 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:15 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:20 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:25 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:30 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:35 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:40 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:45 DEBUG    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:50 ERROR    rasa.core.brokers.pika  - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'.
[2021-03-11 15:07:50 +0000] [1] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
  File "/opt/venv/lib/python3.7/site-packages/sanic/app.py", line 1170, in run
    serve(**server_settings)
  File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 832, in serve
    trigger_events(before_start, loop)
  File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 648, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 254, in load_agent_on_start
    _broker = await EventBroker.create(endpoints.event_broker, loop=loop)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 25, in create
Starting Rasa X in production mode... 🚀
    return await _create_from_endpoint_config(obj, loop)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 74, in _create_from_endpoint_config
    broker = await PikaEventBroker.from_endpoint_config(endpoint_config, event_loop)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 138, in from_endpoint_config
    await broker.connect()
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 144, in connect
    self._connection = await self._connect()
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 188, in _connect
    raise last_exception
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
    return await self.task
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 174, in _connect
    ssl_options=ssl_options,
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 284, in connect_robust
    **kwargs
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 333, in connect
    timeout=timeout, client_properties=client_properties, loop=loop,
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 128, in connect
    timeout=timeout, **self.connect_kwargs
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 121, in connect
    self._make_connection(**kwargs), timeout=timeout,
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
    return await fut
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 105, in _make_connection
    connection = await aiormq.connect(self.url, **kwargs)
  File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 542, in connect
    await connection.connect(client_properties or {​}​)
  File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 168, in wrap
    return await self.create_task(func(self, *args, **kwargs))
  File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
    return await self.task
  File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 255, in connect
    response=credentials.value(self).marshal(),
  File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 212, in __rpc
    raise err
aiormq.exceptions.ProbableAuthenticationError: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
Traceback (most recent call last):
  File "/opt/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/opt/venv/lib/python3.7/site-packages/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/opt/venv/lib/python3.7/site-packages/rasa/cli/x.py", line 338, in rasa_x
    run_in_production(args)
  File "/opt/venv/lib/python3.7/site-packages/rasa/cli/x.py", line 397, in run_in_production
    _rasa_service(args, endpoints, None, credentials_path)
  File "/opt/venv/lib/python3.7/site-packages/rasa/cli/x.py", line 100, in _rasa_service
    ssl_password=args.ssl_password,
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 226, in serve_application
    workers=number_of_workers,
  File "/opt/venv/lib/python3.7/site-packages/sanic/app.py", line 1170, in run
    serve(**server_settings)
  File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 832, in serve
    trigger_events(before_start, loop)
  File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 648, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 254, in load_agent_on_start
    _broker = await EventBroker.create(endpoints.event_broker, loop=loop)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 25, in create
    return await _create_from_endpoint_config(obj, loop)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 74, in _create_from_endpoint_config
    broker = await PikaEventBroker.from_endpoint_config(endpoint_config, event_loop)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 138, in from_endpoint_config
    await broker.connect()
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 144, in connect
    self._connection = await self._connect()
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 188, in _connect
    raise last_exception
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
    return await self.task
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 174, in _connect
    ssl_options=ssl_options,
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 284, in connect_robust
    **kwargs
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 333, in connect
    timeout=timeout, client_properties=client_properties, loop=loop,
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 128, in connect
    timeout=timeout, **self.connect_kwargs
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 121, in connect
    self._make_connection(**kwargs), timeout=timeout,
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
    return await fut
  File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 105, in _make_connection
    connection = await aiormq.connect(self.url, **kwargs)
  File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 542, in connect
    await connection.connect(client_properties or {​}​)
  File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 168, in wrap
    return await self.create_task(func(self, *args, **kwargs))
  File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
    return await self.task
  File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 255, in connect
    response=credentials.value(self).marshal(),
  File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 212, in __rpc
    raise err
aiormq.exceptions.ProbableAuthenticationError: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
/opt/venv/lib/python3.7/site-packages/sentry_sdk/worker.py:118: ResourceWarning: unclosed <ssl.SSLSocket fd=16, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.109.5.23', 58806), raddr=('35.188.42.15', 443)>
  callback = self._queue.get()
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Here is my endpoint Configuration:

 event_broker:
      type: "pika"
      url: "amqps://b-HOST.amazonaws.com"
      username: "rasa"
      password: ${RABBITMQ_PASSWORD}
      port: 5672
      queues:
      - ${RABBITMQ_QUEUE}

Rasa event service logs:

Check for database migrations completed.
INFO:__main__:Starting event service (standalone: True).
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
[2021-03-11 21:14:04 +0000] [28] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2021-03-11 21:14:04 +0000] [28] [INFO] Starting worker [28]

Versions:

rasa-event-service
"rasa/rasa-x:0.34.0"

rasa-rasa-worker
'rasa/rasa:2.1.2-full'

rasa-rasa-x
"rasa/rasa-x:0.34.0"

The error message reads For details see the broker logfile.. Just to check, you’ve been able to inspect these files?

Rasa tries to connect with the username guest although we used rasa username :

PLAIN login refused: user 'guest' - invalid credentials

Just to confirm. You are able to use the same credentials manually to connect to RabbitMQ? Since it reads “invalid credentials” I’d like to double-check that it isn’t due to a typo.

Yes using rasa as username and rasa+123 as passwords works when manually connecting to RabbitMQ. We used rasa/rasa:2.1.2-full but the same error persists

RabbitMQ Logs:

021-03-12 15:44:38.737 [info] <0.16866.16> accepting AMQP connection <0.16866.16> (10.0.0.95:3579 -> 172.18.0.3:5671)
 2021-03-12 15:44:38.738 [error] <0.16866.16> Error on AMQP connection <0.16866.16> (10.0.0.95:3579 -> 172.18.0.3:5671, state: starting):
 PLAIN login refused: user 'guest' - invalid credentials
2021-03-12 15:44:38.740 [warning] <0.16866.16> closing AMQP connection <0.16866.16> (10.0.0.95:3579 -> 172.18.0.3:5671):
 client unexpectedly closed TCP connection
 2021-03-12 15:44:43.766 [info] <0.16984.16> accepting AMQP connection <0.16984.16> (10.0.0.95:4822 -> 172.18.0.3:5671)
 2021-03-12 15:44:43.768 [error] <0.16984.16> Error on AMQP connection <0.16984.16> (10.0.0.95:4822 -> 172.18.0.3:5671, state: starting):
PLAIN login refused: user 'guest' - invalid credentials
2021-03-12 15:44:43.769 [warning] <0.16984.16> closing AMQP connection <0.16984.16> (10.0.0.95:4822 -> 172.18.0.3:5671):
client unexpectedly closed TCP connection
2021-03-12 15:44:48.796 [info] <0.17134.16> accepting AMQP connection <0.17134.16> (10.0.0.95:53673 -> 172.18.0.3:5671)
2021-03-12 15:44:48.798 [error] <0.17134.16> Error on AMQP connection <0.17134.16> (10.0.0.95:53673 -> 172.18.0.3:5671, state: starting):
PLAIN login refused: user 'guest' - invalid credentials
2021-03-12 15:44:48.800 [warning] <0.17134.16> closing AMQP connection <0.17134.16> (10.0.0.95:53673 -> 172.18.0.3:5671):
client unexpectedly closed TCP connection
2021-03-12 15:44:53.828 [info] <0.17262.16> accepting AMQP connection <0.17262.16> (10.0.0.95:5457 -> 172.18.0.3:5671)
2021-03-12 15:44:53.830 [error] <0.17262.16> Error on AMQP connection <0.17262.16> (10.0.0.95:5457 -> 172.18.0.3:5671, state: starting):
PLAIN login refused: user 'guest' - invalid credentials
2021-03-12 15:44:53.832 [warning] <0.17262.16> closing AMQP connection <0.17262.16> (10.0.0.95:5457 -> 172.18.0.3:5671):
client unexpectedly closed TCP connection
2021-03-12 15:44:58.859 [info] <0.17396.16> accepting AMQP connection <0.17396.16> (10.0.0.95:14011 -> 172.18.0.3:5671)
2021-03-12 15:44:58.860 [error] <0.17396.16> Error on AMQP connection <0.17396.16> (10.0.0.95:14011 -> 172.18.0.3:5671, state: starting):
PLAIN login refused: user 'guest' - invalid credentials

We found the problem, we should pass the username and password in the RabbitMQ URL because rasa event service needs the URL, username and password to connect to RabbitMQ unlike rasa worker and rasa production which need the username and password embedded in the RabbitMQ URL:

event_broker:
  type: pika
  url: "amqps://USER:PASSWORD@host"
  username: USER
  password: PASSWORD
  queues:
  - my_queue