Rasa X doesn't respond to messages

Hi,

After some strugles I managed to get Rasa X working on Minikube, sync with repo on Github and train the model. However it still doesn’t respond to my messages (rasa shell responds with the same model). In interactive training mode I see that my message is classified correctly and correct utter action is selected, but no response comes back, all I get is a blinking … icon. On other occasions the conversation is cleared and bird image with “Talk to your assistant and hit the strict mode button to validate every step of the conversation.” is displayed again. No errors is displayed on screen, in the browser console, or in the logs of any Rasa pods.

While I was typing this message, Rasa started responding to some questions in shared chat window, but after responding to 7 or 8 messages, it stopped even displaying my new messages. Restart button doesn’t work at all. Is there a limit of how many messages can be entered in one test session?

I need advice how to start debugging, because Rasa X looks like black box to me.

Now I had a really good chat session using shared chat window, but clicking Restart button ruined it.

Interactive Learning still doesn’t work. image

First time it showed … icon for a long time, after reloading the page and sending message it waits for about 1 seconds and then returns to bird page:

what are your rasa , rasa x and rasa sdk versions just for reference ?

I’m not at my computer now. Whatever versions were available 2 days ago. Rasa X 0.27.3 Rasa 1.9.4 I built a custom rasa docker image with spacy and Lt language pack, based on spacy-de image.

My docker changes: Build rasa with spacy-lt · Naktibalda/rasa@491555f · GitHub My training data: GitHub - Naktibalda/rasa-koronavirusas

It seems that shared chat window works fine until I send message using Interactive learning page, then shared chat breaks too.

I found these errors in the logs of event service:

INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-02 19:02:57 +0000] [13] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2020-04-02 19:02:57 +0000] [13] [INFO] Starting worker [13]
ERROR:__main__:Caught an exception while consuming events. Will retry in 5 s.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1391, in continuously_consume
    consumer.consume()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
    self.channel.start_consuming()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 834, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 566, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1494, in _dispatch_events
    evt.properties, evt.body)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 116, in _callback
    properties
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/event_consumer.py", line 148, in log_event
    on_save=lambda event: self._on_event_saved(data, event),
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 199, in save_event
    self._update_conversation_metadata(event)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 359, in _update_conversation_metadata
    event_confidence, conversation.minimum_action_confidence
TypeError: '<' not supported between instances of 'NoneType' and 'int'
INFO:rasax.community.services.event_consumers.event_consumer:Terminated event consumer liveness endpoint process with PID '13'.
[2020-04-02 19:08:31 +0000] [13] [INFO] Stopping worker [13]
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-02 19:08:31 +0000] [23] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2020-04-02 19:08:31 +0000] [23] [INFO] Starting worker [23]
[2020-04-02 19:08:31 +0000] [13] [INFO] Server Stopped
ERROR:__main__:Caught an exception while consuming events. Will retry in 5 s.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1391, in continuously_consume
    consumer.consume()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
    self.channel.start_consuming()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 834, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 566, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1494, in _dispatch_events
    evt.properties, evt.body)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 116, in _callback
    properties
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/event_consumer.py", line 148, in log_event
    on_save=lambda event: self._on_event_saved(data, event),
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 199, in save_event
    self._update_conversation_metadata(event)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 359, in _update_conversation_metadata
    event_confidence, conversation.minimum_action_confidence
TypeError: '<' not supported between instances of 'NoneType' and 'float'
INFO:rasax.community.services.event_consumers.event_consumer:Terminated event consumer liveness endpoint process with PID '23'.

There is nothing in the logs of rasa services:

k logs -n rasa-koronavirusas rasa-x-1585756598-rasa-x-67775965db-bnr65
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
Starting Rasa X server... 🚀

k logs -n rasa-koronavirusas rasa-x-1585756598-rasa-production-84dfd49f6d-mmqjr
2020-04-02 19:03:42.239132: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

k logs -n rasa-koronavirusas rasa-x-1585756598-rasa-worker-57f854b65c-hh7fr
2020-04-02 19:03:42.070227: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

@Naktibalda it seems like you’re using version 0.26.x of Rasa X, based on the error logs. Could you try upgrading to 0.27.4? Also, I saw you’re building a bot about the coronavirus - if upgrading doesn’t work as intended, feel free to email us at covid@rasa.com and we’ll help you get set up.

I upgraded to 0.27.4, but it made no difference.

This time I was watching event-service logs, so I noticed that the error happened right after clicking on Interactive Learning link.

$ kubectl logs -n rasa-koronavirusas rasa-x-1585756598-event-service-68b6f7fdc6-drknd  -f
INFO:__main__:Waiting until database migrations have been executed...
INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.
INFO:__main__:Check for database migrations completed.
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-03 16:06:24 +0000] [13] [INFO] Goin' Fast @ http://0.0.0.0:5673
[2020-04-03 16:06:24 +0000] [13] [INFO] Starting worker [13]




ERROR:__main__:Caught an exception while consuming events. Will retry in 5 s.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1391, in continuously_consume
    consumer.consume()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
    self.channel.start_consuming()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
    self._process_data_events(time_limit=None)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
    self.connection.process_data_events(time_limit=time_limit)
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 834, in process_data_events
    self._dispatch_channel_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 566, in _dispatch_channel_events
    impl_channel._get_cookie()._dispatch_events()
  File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1494, in _dispatch_events
    evt.properties, evt.body)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 116, in _callback
    properties
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/event_consumer.py", line 148, in log_event
    on_save=lambda event: self._on_event_saved(data, event),
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 199, in save_event
    self._update_conversation_metadata(event)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 359, in _update_conversation_metadata
    event_confidence, conversation.minimum_action_confidence
TypeError: '<' not supported between instances of 'NoneType' and 'int'
INFO:rasax.community.services.event_consumers.event_consumer:Terminated event consumer liveness endpoint process with PID '13'.
[2020-04-03 16:10:21 +0000] [13] [INFO] Stopping worker [13]
[2020-04-03 16:10:21 +0000] [13] [INFO] Server Stopped
INFO:rasax.community.services.event_consumers.event_consumer:Started Sanic liveness endpoint at port '5673'.
[2020-04-03 16:10:21 +0000] [23] [INFO] Goin' Fast @ http://0.0.0.0:5673
INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rasa-x-1585756598-rabbit'.
[2020-04-03 16:10:21 +0000] [23] [INFO] Starting worker [23]

I raised issue about event service crash at Event Service crashes when message is sent using Interactive Learning page · Issue #5577 · RasaHQ/rasa · GitHub

My issue was fixed in 0.27.4, but I only upgraded rasa-x pod,
I had to upgrade eventService and nginx pods too by adding these lines to values.yml:

eventService:
    tag: "0.27.4"
nginx:
    tag: "0.27.4"
2 Likes