Rasa X doesn't respond to messages

Naktibalda · April 2, 2020, 6:51am

Hi,

After some strugles I managed to get Rasa X working on Minikube, sync with repo on Github and train the model. However it still doesn’t respond to my messages (rasa shell responds with the same model). In interactive training mode I see that my message is classified correctly and correct utter action is selected, but no response comes back, all I get is a blinking … icon. On other occasions the conversation is cleared and bird image with “Talk to your assistant and hit the strict mode button to validate every step of the conversation.” is displayed again. No errors is displayed on screen, in the browser console, or in the logs of any Rasa pods.

While I was typing this message, Rasa started responding to some questions in shared chat window, but after responding to 7 or 8 messages, it stopped even displaying my new messages. Restart button doesn’t work at all. Is there a limit of how many messages can be entered in one test session?

I need advice how to start debugging, because Rasa X looks like black box to me.

Naktibalda · April 2, 2020, 4:30pm

Now I had a really good chat session using shared chat window, but clicking Restart button ruined it.

Interactive Learning still doesn’t work.

First time it showed … icon for a long time, after reloading the page and sending message it waits for about 1 seconds and then returns to bird page:

pandaxar · April 2, 2020, 4:46pm

what are your rasa , rasa x and rasa sdk versions just for reference ?

Naktibalda · April 2, 2020, 6:28pm

I’m not at my computer now. Whatever versions were available 2 days ago. Rasa X 0.27.3 Rasa 1.9.4 I built a custom rasa docker image with spacy and Lt language pack, based on spacy-de image.

Naktibalda · April 2, 2020, 7:15pm

My docker changes: Build rasa with spacy-lt · Naktibalda/rasa@491555f · GitHub My training data: GitHub - Naktibalda/rasa-koronavirusas

It seems that shared chat window works fine until I send message using Interactive learning page, then shared chat breaks too.

I found these errors in the logs of event service:

INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-02 19:02:57 +0000] [13] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2020-04-02 19:02:57 +0000] [13] [INFO] Starting worker [13]
ERROR:__main__:Caught an exception while consuming events. Will retry in 5 s.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1391, in continuously_consume
    consumer.consume()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
    self.channel.start_consuming()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 834, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 566, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1494, in _dispatch_events
    evt.properties, evt.body)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 116, in _callback
    properties
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/event_consumer.py", line 148, in log_event
    on_save=lambda event: self._on_event_saved(data, event),
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 199, in save_event
    self._update_conversation_metadata(event)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 359, in _update_conversation_metadata
    event_confidence, conversation.minimum_action_confidence
TypeError: '<' not supported between instances of 'NoneType' and 'int'
INFO:rasax.community.services.event_consumers.event_consumer:Terminated event consumer liveness endpoint process with PID '13'.
[2020-04-02 19:08:31 +0000] [13] [INFO] Stopping worker [13]
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-02 19:08:31 +0000] [23] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2020-04-02 19:08:31 +0000] [23] [INFO] Starting worker [23]
[2020-04-02 19:08:31 +0000] [13] [INFO] Server Stopped
ERROR:__main__:Caught an exception while consuming events. Will retry in 5 s.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1391, in continuously_consume
    consumer.consume()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
    self.channel.start_consuming()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 834, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 566, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1494, in _dispatch_events
    evt.properties, evt.body)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 116, in _callback
    properties
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/event_consumer.py", line 148, in log_event
    on_save=lambda event: self._on_event_saved(data, event),
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 199, in save_event
    self._update_conversation_metadata(event)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 359, in _update_conversation_metadata
    event_confidence, conversation.minimum_action_confidence
TypeError: '<' not supported between instances of 'NoneType' and 'float'
INFO:rasax.community.services.event_consumers.event_consumer:Terminated event consumer liveness endpoint process with PID '23'.

There is nothing in the logs of rasa services:

k logs -n rasa-koronavirusas rasa-x-1585756598-rasa-x-67775965db-bnr65
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
Starting Rasa X server... 🚀

k logs -n rasa-koronavirusas rasa-x-1585756598-rasa-production-84dfd49f6d-mmqjr
2020-04-02 19:03:42.239132: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

k logs -n rasa-koronavirusas rasa-x-1585756598-rasa-worker-57f854b65c-hh7fr
2020-04-02 19:03:42.070227: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

akelad · April 3, 2020, 1:09pm

@Naktibalda it seems like you’re using version 0.26.x of Rasa X, based on the error logs. Could you try upgrading to 0.27.4? Also, I saw you’re building a bot about the coronavirus - if upgrading doesn’t work as intended, feel free to email us at covid@rasa.com and we’ll help you get set up.

Naktibalda · April 3, 2020, 4:13pm

I upgraded to 0.27.4, but it made no difference.

This time I was watching event-service logs, so I noticed that the error happened right after clicking on Interactive Learning link.

$ kubectl logs -n rasa-koronavirusas rasa-x-1585756598-event-service-68b6f7fdc6-drknd  -f
INFO:__main__:Waiting until database migrations have been executed...
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:__main__:Check for database migrations completed.
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-03 16:06:24 +0000] [13] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2020-04-03 16:06:24 +0000] [13] [INFO] Starting worker [13]




ERROR:__main__:Caught an exception while consuming events. Will retry in 5 s.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1391, in continuously_consume
    consumer.consume()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
    self.channel.start_consuming()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 834, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 566, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1494, in _dispatch_events
    evt.properties, evt.body)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 116, in _callback
    properties
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/event_consumer.py", line 148, in log_event
    on_save=lambda event: self._on_event_saved(data, event),
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 199, in save_event
    self._update_conversation_metadata(event)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 359, in _update_conversation_metadata
    event_confidence, conversation.minimum_action_confidence
TypeError: '<' not supported between instances of 'NoneType' and 'int'
INFO:rasax.community.services.event_consumers.event_consumer:Terminated event consumer liveness endpoint process with PID '13'.
[2020-04-03 16:10:21 +0000] [13] [INFO] Stopping worker [13]
[2020-04-03 16:10:21 +0000] [13] [INFO] Server Stopped
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
[2020-04-03 16:10:21 +0000] [23] [INFO] Goin' Fast @ http://0.0.0.0:5673
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-03 16:10:21 +0000] [23] [INFO] Starting worker [23]

Naktibalda · April 4, 2020, 10:06am

I raised issue about event service crash at Event Service crashes when message is sent using Interactive Learning page · Issue #5577 · RasaHQ/rasa · GitHub

Naktibalda · April 6, 2020, 3:48pm

My issue was fixed in 0.27.4, but I only upgraded rasa-x pod,
I had to upgrade eventService and nginx pods too by adding these lines to values.yml:

eventService:
    tag: "0.27.4"
nginx:
    tag: "0.27.4"

Topic		Replies	Views
Rasa X not showing responses [Deprecated] Rasa X Community Edition	7	3036	January 24, 2022
Rasa X is not answering nor recognizing intents [Deprecated] Rasa X Community Edition	3	778	April 3, 2021
Responses not showing up, but model working in interactive mode RasaX: 0.33 [Deprecated] Rasa X Community Edition	14	1798	June 17, 2021
Rasa X not working properly [Deprecated] Rasa X Community Edition	32	6617	December 15, 2020
The answers of bot don't appear on Rasa X [Deprecated] Rasa X Community Edition	9	1035	March 24, 2021

Rasa X doesn't respond to messages

Related topics