I’m facing some issues when i try to connect rasax to AWS RabbitMQ, rasa event service can create new Broker in AWS RabbitMQ but rasa-worker and rasa-production cannot connect to it. I’m using rasax helm chart, here is the logs from rasa-worker and rasa-production:
2021-03-11 15:06:09 DEBUG sanic_jwt.configuration - validating provided secret
2021-03-11 15:06:09 DEBUG sanic_jwt.configuration - validating keys (if needed)
2021-03-11 15:06:09 DEBUG sanic_jwt.configuration - loading secret and/or keys (if needed)
2021-03-11 15:06:09 DEBUG rasa.core.utils - Available web server routes:
/conversations/<conversation_id:path>/messages POST add_message
/conversations/<conversation_id:path>/tracker/events POST append_events
/auth OPTIONS auth_bp.AuthenticateEndpoint
/auth/me OPTIONS auth_bp.RetrieveUserEndpoint
/auth/verify OPTIONS auth_bp.VerifyEndpoint
/webhooks/rasa GET custom_webhook_RasaChatInput.health
/webhooks/rasa/webhook POST custom_webhook_RasaChatInput.receive
/webhooks/rest GET custom_webhook_RestInput.health
/webhooks/rest/webhook POST custom_webhook_RestInput.receive
/model/test/intents POST evaluate_intents
/model/test/stories POST evaluate_stories
/conversations/<conversation_id:path>/execute POST execute_action
/domain GET get_domain
/ GET hello
/model PUT load_model
/model/parse POST parse
/conversations/<conversation_id:path>/predict POST predict
/conversations/<conversation_id:path>/tracker/events PUT replace_events
/conversations/<conversation_id:path>/story GET retrieve_story
/conversations/<conversation_id:path>/tracker GET retrieve_tracker
/status GET status
/model/predict POST tracker_predict
/model/train POST train
/conversations/<conversation_id:path>/trigger_intent POST trigger_intent
/model DELETE unload_model
/version GET version
2021-03-11 15:06:09 INFO root - Starting Rasa server on http://localhost:5005
2021-03-11 15:06:09 DEBUG rasa.core.utils - Using the default number of Sanic workers (1).
2021-03-11 15:06:09 INFO root - Enabling coroutine debugging. Loop id 93909152029232.
2021-03-11 15:06:09 DEBUG root - Could not load interpreter from 'None'.
2021-03-11 15:06:09 INFO rasa.core.brokers.pika - Connecting to RabbitMQ ...
2021-03-11 15:06:10 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:15 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:20 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:25 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:30 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:35 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:40 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:45 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:50 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:06:55 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:00 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:05 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:10 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:15 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:20 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:25 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:30 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:35 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:40 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:45 DEBUG rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.
2021-03-11 15:07:50 ERROR rasa.core.brokers.pika - Connecting to 'amqps://b-HOST.amazonaws.com' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'.
[2021-03-11 15:07:50 +0000] [1] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
File "/opt/venv/lib/python3.7/site-packages/sanic/app.py", line 1170, in run
serve(**server_settings)
File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 832, in serve
trigger_events(before_start, loop)
File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 648, in trigger_events
loop.run_until_complete(result)
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 254, in load_agent_on_start
_broker = await EventBroker.create(endpoints.event_broker, loop=loop)
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 25, in create
Starting Rasa X in production mode... 🚀
return await _create_from_endpoint_config(obj, loop)
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 74, in _create_from_endpoint_config
broker = await PikaEventBroker.from_endpoint_config(endpoint_config, event_loop)
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 138, in from_endpoint_config
await broker.connect()
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 144, in connect
self._connection = await self._connect()
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 188, in _connect
raise last_exception
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
return await self.task
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 174, in _connect
ssl_options=ssl_options,
File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 284, in connect_robust
**kwargs
File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 333, in connect
timeout=timeout, client_properties=client_properties, loop=loop,
File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 128, in connect
timeout=timeout, **self.connect_kwargs
File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 121, in connect
self._make_connection(**kwargs), timeout=timeout,
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
return await fut
File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 105, in _make_connection
connection = await aiormq.connect(self.url, **kwargs)
File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 542, in connect
await connection.connect(client_properties or {})
File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 168, in wrap
return await self.create_task(func(self, *args, **kwargs))
File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
return await self.task
File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 255, in connect
response=credentials.value(self).marshal(),
File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 212, in __rpc
raise err
aiormq.exceptions.ProbableAuthenticationError: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
Traceback (most recent call last):
File "/opt/venv/bin/rasa", line 8, in <module>
sys.exit(main())
File "/opt/venv/lib/python3.7/site-packages/rasa/__main__.py", line 116, in main
cmdline_arguments.func(cmdline_arguments)
File "/opt/venv/lib/python3.7/site-packages/rasa/cli/x.py", line 338, in rasa_x
run_in_production(args)
File "/opt/venv/lib/python3.7/site-packages/rasa/cli/x.py", line 397, in run_in_production
_rasa_service(args, endpoints, None, credentials_path)
File "/opt/venv/lib/python3.7/site-packages/rasa/cli/x.py", line 100, in _rasa_service
ssl_password=args.ssl_password,
File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 226, in serve_application
workers=number_of_workers,
File "/opt/venv/lib/python3.7/site-packages/sanic/app.py", line 1170, in run
serve(**server_settings)
File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 832, in serve
trigger_events(before_start, loop)
File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 648, in trigger_events
loop.run_until_complete(result)
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 254, in load_agent_on_start
_broker = await EventBroker.create(endpoints.event_broker, loop=loop)
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 25, in create
return await _create_from_endpoint_config(obj, loop)
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/broker.py", line 74, in _create_from_endpoint_config
broker = await PikaEventBroker.from_endpoint_config(endpoint_config, event_loop)
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 138, in from_endpoint_config
await broker.connect()
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 144, in connect
self._connection = await self._connect()
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 188, in _connect
raise last_exception
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 630, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
return await self.task
File "/opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py", line 174, in _connect
ssl_options=ssl_options,
File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 284, in connect_robust
**kwargs
File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 333, in connect
timeout=timeout, client_properties=client_properties, loop=loop,
File "/opt/venv/lib/python3.7/site-packages/aio_pika/robust_connection.py", line 128, in connect
timeout=timeout, **self.connect_kwargs
File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 121, in connect
self._make_connection(**kwargs), timeout=timeout,
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
return await fut
File "/opt/venv/lib/python3.7/site-packages/aio_pika/connection.py", line 105, in _make_connection
connection = await aiormq.connect(self.url, **kwargs)
File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 542, in connect
await connection.connect(client_properties or {})
File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 168, in wrap
return await self.create_task(func(self, *args, **kwargs))
File "/opt/venv/lib/python3.7/site-packages/aiormq/base.py", line 25, in __inner
return await self.task
File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 255, in connect
response=credentials.value(self).marshal(),
File "/opt/venv/lib/python3.7/site-packages/aiormq/connection.py", line 212, in __rpc
raise err
aiormq.exceptions.ProbableAuthenticationError: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
/opt/venv/lib/python3.7/site-packages/sentry_sdk/worker.py:118: ResourceWarning: unclosed <ssl.SSLSocket fd=16, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.109.5.23', 58806), raddr=('35.188.42.15', 443)>
callback = self._queue.get()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Here is my endpoint Configuration:
event_broker:
type: "pika"
url: "amqps://b-HOST.amazonaws.com"
username: "rasa"
password: ${RABBITMQ_PASSWORD}
port: 5672
queues:
- ${RABBITMQ_QUEUE}
Rasa event service logs:
Check for database migrations completed.
INFO:__main__:Starting event service (standalone: True).
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
[2021-03-11 21:14:04 +0000] [28] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2021-03-11 21:14:04 +0000] [28] [INFO] Starting worker [28]
Versions:
rasa-event-service
"rasa/rasa-x:0.34.0"
rasa-rasa-worker
'rasa/rasa:2.1.2-full'
rasa-rasa-x
"rasa/rasa-x:0.34.0"