Identical model behaving differently on Rasa VS Rasa X

Hey guys :wave:

I have a trained model, which works perfectly fine on Rasa Open Source.

But when I upload the same model to Rasa X, it produces completely different outputs on Rasa X.

Rasa Open Source (shell):

Rasa X (talk to your bot / shared link for testers):

Further info:

  • The model files are definetly identical
  • rasa x is installed via docker-compose on an EC2 instance
  • a github repo is connected

Any ideas about the cause of this problem?

Or how to solve it? :smiley:

Hey @ClemensJ,

Can you check/answer a few things?

  • go to the models screen and double check that the correct model is made active.
  • are you using any custom components in your pipeline that may not be referenced correctly in docker compose?
  • are you running any custom actions at this point?
  • can you check the rasa worker container logs to see if there are any errors loading the latest model? (make sure debug mode is turned on if you donโ€™t see anything)

Thanksfor your answer @desmarchris. Here are the answers to your question and the logs.

  • go to the models screen and double check that the correct model is made active. โ†’ :white_check_mark:
  • are you using any custom components in your pipeline that may not be referenced correctly in docker compose? โ†’ Iโ€™m still using the default pipeline
  • are you running any custom actions at this point? โ†’ no, I donโ€™t
  • can you check the rasa worker container logs to see if there are any errors loading the latest model? (make sure debug mode is turned on if you donโ€™t see anything) โ†’ logs:
  1. Uploading model to rasa-x:
rasa-x_1           | [2021-06-06 07:52:27 +0000] - (sanic.access)[INFO][172.19.0.11:60566]: POST http://the.server.com/api/projects/default/models?api_token=76f2fc6c41ddec83ad6c472396aeabead2ae75cb  201 199
rasa-x_1           | DEBUG:urllib3.connectionpool:https://api.segment.io:443 "POST /v1/track HTTP/1.1" 200 21
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | [2021-06-06 07:52:29 +0000] - (sanic.access)[INFO][172.19.0.8:34328]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=7M7C2z0bWM6ImVC  404 160
rasa-x_1           | INFO:sanic.access:
rasa-x_1           | DEBUG:websockets.protocol:server > Frame(fin=True, opcode=9, data=b'cm\xb3v', rsv1=False, rsv2=False, rsv3=False)
rasa-x_1           | DEBUG:websockets.protocol:server < Frame(fin=True, opcode=10, data=b'cm\xb3v', rsv1=False, rsv2=False, rsv3=False)
rasa-x_1           | DEBUG:websockets.protocol:server - received solicited pong: 636db376
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | [2021-06-06 07:52:31 +0000] - (sanic.access)[INFO][172.19.0.9:35820]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=7M7C2z0bWM6ImVC  404 160
  1. Tagging model as production:
> rasa-x_1           | [2021-06-06 07:53:52 +0000] - (sanic.access)[INFO][172.19.0.11:60738]: PUT http://the.server.com/api/projects/default/models/20210529-170445/tags/production  204 0
rasa-x_1           | DEBUG:urllib3.connectionpool:https://api.segment.io:443 "POST /v1/track HTTP/1.1" 200 21
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | INFO:sanic.access:
rasa-x_1           | [2021-06-06 07:53:59 +0000] - (sanic.access)[INFO][172.19.0.8:34498]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=7M7C2z0bWM6ImVC  204 0
  1. Sending message in interactive mode:
> rasa-x_1           | [2021-06-06 08:05:15 +0000] - (sanic.access)[INFO][172.19.0.11:33602]: POST http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879/messages?environment=production  200 2
rasa-x_1           | INFO:sanic.access:
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:15 +0000] "POST /api/conversations/27a1aefa744345b29cdfb37f12f1f879/messages?environment=production HTTP/1.1" 200 2 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user", "timestamp": 1622966715.9585028, "text": "Hello", "parse_data": {"intent": {"name": "nlu_fallback", "confidence": 0.843318983912468}, "entities": [], "text": "Hello", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}, "intent_ranking": [{"name": "nlu_fallback", "confidence": 0.843318983912468}, {"id": 7616751426105194017, "name": "deny", "confidence": 0.15668101608753204}, {"id": 6548359479492351688, "name": "mood_great", "confidence": 0.1550668627023697}, {"id": 384799227093601941, "name": "bot_challenge", "confidence": 0.12006856501102448}, {"id": 6208300640583772273, "name": "describe_outcome", "confidence": 0.11965557932853699}, {"id": -2598160608433049275, "name": "describe_small_step", "confidence": 0.11938878148794174}, {"id": 6436635119084146062, "name": "describe_goal_criteria", "confidence": 0.09934554249048233}, {"id": 4371941254989265915, "name": "greet", "confidence": 0.07762385159730911}, {"id": -5762783295493080525, "name": "describe_resource", "confidence": 0.05361383408308029}, {"id": 4789708915851480043, "name": "affirm", "confidence": 0.051674239337444305}, {"id": -6058483853826046540, "name": "mood_unhappy", "confidence": 0.04688167944550514}], "response_selector": {"all_retrieval_intents": [], "default": {"response": {"id": null, "responses": null, "response_templates": null, "confidence": 0.0, "intent_response_key": null, "utter_action": "utter_None", "template_name": "utter_None"}, "ranking": []}}}, "input_channel": "rasa", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}}'
rasa-x_1           | DEBUG:rasax.community.services.websocket_service:Broadcasting message: {'topic': 'messages', 'name': 'message_sent', 'data': {'channel': 'rasa'}, 'recipient_id': 'ALL', 'scopes': None}
rasa-x_1           | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.segment.io:443
rasa-x_1           | DEBUG:rasax.community.services.logs_service:Could not find a model in the supplied NLU parse data. Will try to associate the message log with the current production model instead.
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | DEBUG:rasax.community.services.logs_service:Saving to NLU logs:
rasa-x_1           | <rasax.community.database.conversation.MessageLog object at 0x7f38bd9d2e50>
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user", "timestamp": 1622966715.9585028, "text": "Hello", "parse_data": {"intent": {"name": "nlu_fallback", "confidence": 0.843318983912468}, "entities": [], "text": "Hello", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}, "intent_ranking": [{"name": "nlu_fallback", "confidence": 0.843318983912468}, {"id": 7616751426105194017, "name": "deny", "confidence": 0.15668101608753204}, {"id": 6548359479492351688, "name": "mood_great", "confidence": 0.1550668627023697}, {"id": 384799227093601941, "name": "bot_challenge", "confidence": 0.12006856501102448}, {"id": 6208300640583772273, "name": "describe_outcome", "confidence": 0.11965557932853699}, {"id": -2598160608433049275, "name": "describe_small_step", "confidence": 0.11938878148794174}, {"id": 6436635119084146062, "name": "describe_goal_criteria", "confidence": 0.09934554249048233}, {"id": 4371941254989265915, "name": "greet", "confidence": 0.07762385159730911}, {"id": -5762783295493080525, "name": "describe_resource", "confidence": 0.05361383408308029}, {"id": 4789708915851480043, "name": "affirm", "confidence": 0.051674239337444305}, {"id": -6058483853826046540, "name": "mood_unhappy", "confidence": 0.04688167944550514}], "response_selector": {"all_retrieval_intents": [], "default": {"response": {"id": null, "responses": null, "response_templates": null, "confidence": 0.0, "intent_response_key": null, "utter_action": "utter_None", "template_name": "utter_None"}, "ranking": []}}}, "input_channel": "rasa", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user_featurization", "timestamp": 1622966715.9646637, "use_text_for_featurization": false}'
rasa-x_1           | [2021-06-06 08:05:16 +0000] - (sanic.access)[INFO][172.19.0.11:33606]: GET http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false  200 310
rasa-x_1           | INFO:sanic.access:
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:16 +0000] "GET /api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false HTTP/1.1" 200 310 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | DEBUG:rasax.community.services.websocket_service:Broadcasting message: {'topic': 'messages', 'name': 'message_sent', 'data': {'channel': 'rasa'}, 'recipient_id': 'ALL', 'scopes': None}
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user_featurization", "timestamp": 1622966715.9646637, "use_text_for_featurization": false}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9646747, "name": "action_restart", "policy": "policy_1_TEDPolicy", "confidence": 0.342068612575531, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9646747, "name": "action_restart", "policy": "policy_1_TEDPolicy", "confidence": 0.342068612575531, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "restart", "timestamp": 1622966715.9646838}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "restart", "timestamp": 1622966715.9646838}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648616, "name": "action_session_start", "policy": "followup_action", "confidence": 1.0, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648616, "name": "action_session_start", "policy": "followup_action", "confidence": 1.0, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "session_started", "timestamp": 1622966715.9648757}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "session_started", "timestamp": 1622966715.9648757}'
rasa-x_1           | INFO:sanic.access:
rasa-x_1           | [2021-06-06 08:05:16 +0000] - (sanic.access)[INFO][172.19.0.11:33612]: GET http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false  200 362
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:16 +0000] "GET /api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false HTTP/1.1" 200 362 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | [2021-06-06 08:05:16 +0000] - (sanic.access)[INFO][172.19.0.11:33610]: GET http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=true  200 487
rasa-x_1           | INFO:sanic.access:
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:16 +0000] "GET /api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=true HTTP/1.1" 200 487 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648893, "name": "action_listen", "policy": null, "confidence": null, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648893, "name": "action_listen", "policy": null, "confidence": null, "action_text": null}'
  1. sudo docker-compose logs rasa-worker:
> Attaching to rasa_rasa-worker_1
rasa-worker_1      | 2021-06-06 07:49:13.922770: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
rasa-worker_1      | 2021-06-06 07:49:13.922882: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
rasa-worker_1      | 2021-06-06 07:49:16.419032: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
rasa-worker_1      | 2021-06-06 07:49:16.419077: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
rasa-worker_1      | 2021-06-06 07:49:16.419137: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2849706c3427): /proc/driver/nvidia/version does not exist
rasa-worker_1      | 2021-06-06 07:49:34.180135: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
rasa-worker_1      | To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
rasa-worker_1      | 2021-06-06 07:49:34.187709: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300060000 Hz
rasa-worker_1      | 2021-06-06 07:49:34.188044: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x44723c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
rasa-worker_1      | 2021-06-06 07:49:34.188175: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

Does that help?

@ClemensJ thanks for posting the info! nothing seems out of the ordinary hereโ€ฆ are you able to share the repo so I can test locally? Along with your Rasa X and Rasa open source versions?