Identical model behaving differently on Rasa VS Rasa X

ClemensJ · June 1, 2021, 8:57am

Hey guys

I have a trained model, which works perfectly fine on Rasa Open Source.

But when I upload the same model to Rasa X, it produces completely different outputs on Rasa X.

Rasa Open Source (shell):

Rasa X (talk to your bot / shared link for testers):

Further info:

The model files are definetly identical
rasa x is installed via docker-compose on an EC2 instance
a github repo is connected

Any ideas about the cause of this problem?

Or how to solve it?

desmarchris · June 1, 2021, 8:28pm

Hey @ClemensJ,

Can you check/answer a few things?

go to the models screen and double check that the correct model is made active.
are you using any custom components in your pipeline that may not be referenced correctly in docker compose?
are you running any custom actions at this point?
can you check the rasa worker container logs to see if there are any errors loading the latest model? (make sure debug mode is turned on if you don’t see anything)

ClemensJ · June 6, 2021, 8:20am

Thanksfor your answer @desmarchris. Here are the answers to your question and the logs.

go to the models screen and double check that the correct model is made active. →
are you using any custom components in your pipeline that may not be referenced correctly in docker compose? → I’m still using the default pipeline
are you running any custom actions at this point? → no, I don’t
can you check the rasa worker container logs to see if there are any errors loading the latest model? (make sure debug mode is turned on if you don’t see anything) → logs:

Uploading model to rasa-x:

rasa-x_1           | [2021-06-06 07:52:27 +0000] - (sanic.access)[INFO][172.19.0.11:60566]: POST http://the.server.com/api/projects/default/models?api_token=76f2fc6c41ddec83ad6c472396aeabead2ae75cb  201 199
rasa-x_1           | DEBUG:urllib3.connectionpool:https://api.segment.io:443 "POST /v1/track HTTP/1.1" 200 21
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | [2021-06-06 07:52:29 +0000] - (sanic.access)[INFO][172.19.0.8:34328]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=7M7C2z0bWM6ImVC  404 160
rasa-x_1           | INFO:sanic.access:
rasa-x_1           | DEBUG:websockets.protocol:server > Frame(fin=True, opcode=9, data=b'cm\xb3v', rsv1=False, rsv2=False, rsv3=False)
rasa-x_1           | DEBUG:websockets.protocol:server < Frame(fin=True, opcode=10, data=b'cm\xb3v', rsv1=False, rsv2=False, rsv3=False)
rasa-x_1           | DEBUG:websockets.protocol:server - received solicited pong: 636db376
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | [2021-06-06 07:52:31 +0000] - (sanic.access)[INFO][172.19.0.9:35820]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=7M7C2z0bWM6ImVC  404 160

Tagging model as production:

> rasa-x_1           | [2021-06-06 07:53:52 +0000] - (sanic.access)[INFO][172.19.0.11:60738]: PUT http://the.server.com/api/projects/default/models/20210529-170445/tags/production  204 0
rasa-x_1           | DEBUG:urllib3.connectionpool:https://api.segment.io:443 "POST /v1/track HTTP/1.1" 200 21
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | INFO:sanic.access:
rasa-x_1           | [2021-06-06 07:53:59 +0000] - (sanic.access)[INFO][172.19.0.8:34498]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=7M7C2z0bWM6ImVC  204 0

Sending message in interactive mode:

> rasa-x_1           | [2021-06-06 08:05:15 +0000] - (sanic.access)[INFO][172.19.0.11:33602]: POST http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879/messages?environment=production  200 2
rasa-x_1           | INFO:sanic.access:
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:15 +0000] "POST /api/conversations/27a1aefa744345b29cdfb37f12f1f879/messages?environment=production HTTP/1.1" 200 2 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user", "timestamp": 1622966715.9585028, "text": "Hello", "parse_data": {"intent": {"name": "nlu_fallback", "confidence": 0.843318983912468}, "entities": [], "text": "Hello", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}, "intent_ranking": [{"name": "nlu_fallback", "confidence": 0.843318983912468}, {"id": 7616751426105194017, "name": "deny", "confidence": 0.15668101608753204}, {"id": 6548359479492351688, "name": "mood_great", "confidence": 0.1550668627023697}, {"id": 384799227093601941, "name": "bot_challenge", "confidence": 0.12006856501102448}, {"id": 6208300640583772273, "name": "describe_outcome", "confidence": 0.11965557932853699}, {"id": -2598160608433049275, "name": "describe_small_step", "confidence": 0.11938878148794174}, {"id": 6436635119084146062, "name": "describe_goal_criteria", "confidence": 0.09934554249048233}, {"id": 4371941254989265915, "name": "greet", "confidence": 0.07762385159730911}, {"id": -5762783295493080525, "name": "describe_resource", "confidence": 0.05361383408308029}, {"id": 4789708915851480043, "name": "affirm", "confidence": 0.051674239337444305}, {"id": -6058483853826046540, "name": "mood_unhappy", "confidence": 0.04688167944550514}], "response_selector": {"all_retrieval_intents": [], "default": {"response": {"id": null, "responses": null, "response_templates": null, "confidence": 0.0, "intent_response_key": null, "utter_action": "utter_None", "template_name": "utter_None"}, "ranking": []}}}, "input_channel": "rasa", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}}'
rasa-x_1           | DEBUG:rasax.community.services.websocket_service:Broadcasting message: {'topic': 'messages', 'name': 'message_sent', 'data': {'channel': 'rasa'}, 'recipient_id': 'ALL', 'scopes': None}
rasa-x_1           | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.segment.io:443
rasa-x_1           | DEBUG:rasax.community.services.logs_service:Could not find a model in the supplied NLU parse data. Will try to associate the message log with the current production model instead.
rasa-x_1           | DEBUG:rasax.community.services.model_service:Model discovery completed.
rasa-x_1           | DEBUG:rasax.community.services.logs_service:Saving to NLU logs:
rasa-x_1           | <rasax.community.database.conversation.MessageLog object at 0x7f38bd9d2e50>
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user", "timestamp": 1622966715.9585028, "text": "Hello", "parse_data": {"intent": {"name": "nlu_fallback", "confidence": 0.843318983912468}, "entities": [], "text": "Hello", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}, "intent_ranking": [{"name": "nlu_fallback", "confidence": 0.843318983912468}, {"id": 7616751426105194017, "name": "deny", "confidence": 0.15668101608753204}, {"id": 6548359479492351688, "name": "mood_great", "confidence": 0.1550668627023697}, {"id": 384799227093601941, "name": "bot_challenge", "confidence": 0.12006856501102448}, {"id": 6208300640583772273, "name": "describe_outcome", "confidence": 0.11965557932853699}, {"id": -2598160608433049275, "name": "describe_small_step", "confidence": 0.11938878148794174}, {"id": 6436635119084146062, "name": "describe_goal_criteria", "confidence": 0.09934554249048233}, {"id": 4371941254989265915, "name": "greet", "confidence": 0.07762385159730911}, {"id": -5762783295493080525, "name": "describe_resource", "confidence": 0.05361383408308029}, {"id": 4789708915851480043, "name": "affirm", "confidence": 0.051674239337444305}, {"id": -6058483853826046540, "name": "mood_unhappy", "confidence": 0.04688167944550514}], "response_selector": {"all_retrieval_intents": [], "default": {"response": {"id": null, "responses": null, "response_templates": null, "confidence": 0.0, "intent_response_key": null, "utter_action": "utter_None", "template_name": "utter_None"}, "ranking": []}}}, "input_channel": "rasa", "message_id": "1c9d13e6462c4e76aa2279ec1ccbd72d", "metadata": {}}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user_featurization", "timestamp": 1622966715.9646637, "use_text_for_featurization": false}'
rasa-x_1           | [2021-06-06 08:05:16 +0000] - (sanic.access)[INFO][172.19.0.11:33606]: GET http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false  200 310
rasa-x_1           | INFO:sanic.access:
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:16 +0000] "GET /api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false HTTP/1.1" 200 310 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | DEBUG:rasax.community.services.websocket_service:Broadcasting message: {'topic': 'messages', 'name': 'message_sent', 'data': {'channel': 'rasa'}, 'recipient_id': 'ALL', 'scopes': None}
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "user_featurization", "timestamp": 1622966715.9646637, "use_text_for_featurization": false}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9646747, "name": "action_restart", "policy": "policy_1_TEDPolicy", "confidence": 0.342068612575531, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9646747, "name": "action_restart", "policy": "policy_1_TEDPolicy", "confidence": 0.342068612575531, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "restart", "timestamp": 1622966715.9646838}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "restart", "timestamp": 1622966715.9646838}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648616, "name": "action_session_start", "policy": "followup_action", "confidence": 1.0, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648616, "name": "action_session_start", "policy": "followup_action", "confidence": 1.0, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "session_started", "timestamp": 1622966715.9648757}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "session_started", "timestamp": 1622966715.9648757}'
rasa-x_1           | INFO:sanic.access:
rasa-x_1           | [2021-06-06 08:05:16 +0000] - (sanic.access)[INFO][172.19.0.11:33612]: GET http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false  200 362
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:16 +0000] "GET /api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=false HTTP/1.1" 200 362 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | [2021-06-06 08:05:16 +0000] - (sanic.access)[INFO][172.19.0.11:33610]: GET http://the.server.com/api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=true  200 487
rasa-x_1           | INFO:sanic.access:
nginx_1            | 77.185.102.56 - - [06/Jun/2021:08:05:16 +0000] "GET /api/conversations/27a1aefa744345b29cdfb37f12f1f879?test_story=true HTTP/1.1" 200 487 "https://the.server.com/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
rasa-x_1           | DEBUG:rasax.community.services.event_service:Saving event from origin 'production' to event service:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648893, "name": "action_listen", "policy": null, "confidence": null, "action_text": null}'
rasa-x_1           | DEBUG:rasax.community.services.analytics_service:Saving to AnalyticsService:
rasa-x_1           | b'{"sender_id": "27a1aefa744345b29cdfb37f12f1f879", "event": "action", "timestamp": 1622966715.9648893, "name": "action_listen", "policy": null, "confidence": null, "action_text": null}'

sudo docker-compose logs rasa-worker:

> Attaching to rasa_rasa-worker_1
rasa-worker_1      | 2021-06-06 07:49:13.922770: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
rasa-worker_1      | 2021-06-06 07:49:13.922882: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
rasa-worker_1      | 2021-06-06 07:49:16.419032: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
rasa-worker_1      | 2021-06-06 07:49:16.419077: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
rasa-worker_1      | 2021-06-06 07:49:16.419137: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2849706c3427): /proc/driver/nvidia/version does not exist
rasa-worker_1      | 2021-06-06 07:49:34.180135: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
rasa-worker_1      | To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
rasa-worker_1      | 2021-06-06 07:49:34.187709: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300060000 Hz
rasa-worker_1      | 2021-06-06 07:49:34.188044: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x44723c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
rasa-worker_1      | 2021-06-06 07:49:34.188175: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

Does that help?

desmarchris · June 7, 2021, 1:18pm

@ClemensJ thanks for posting the info! nothing seems out of the ordinary here… are you able to share the repo so I can test locally? Along with your Rasa X and Rasa open source versions?

r3c0nf1gur3d · August 23, 2021, 11:50am

Were you able to resolve the issue? I am running into the same problems and would be interested in any hints on what is happening…

desmarchris · August 23, 2021, 5:52pm

@r3c0nf1gur3d do you see any errors in the Rasa X, rasa production or worker logs? my initial hunch is that the model is not being loaded in correctly

r3c0nf1gur3d · August 24, 2021, 12:08am

I found one error, but I am not sure if this is related to this issue: Rasa X form not working: IndexError - List index out of range · Issue #9437 · RasaHQ/rasa · GitHub

The rest of the logs however look fine to me so far. If the issue doesn’t give any insights I will copy the logs tomorrow. Out of curiosity, how can a model not be load correctly? I am just wondering as I am using the “upload model” function of rasa x after I build the model locally. Could there be a problem with my workflow/setup? I thought as long as the bot does something the model should be loaded and ready

Thanks for the help!

desmarchris · August 24, 2021, 1:20pm

there would only be an issue with your workflow if you are uploading a different model or not tagging your new model as production

so most likely, it’s failing to upload/tag the new model but not actually telling you that. Did everything look ok when you uploaded in the UI? Otherwise, I think I’d need my hands on a full repo where this happens so I can try to reproduce locally.

r3c0nf1gur3d · August 24, 2021, 8:06pm

The problem is a mismatch of the rasa version used (at least I think). My rasa x instance uses rasa 2.6.x while my local machine uses 2.8.x. So I think the model is not backward compatible? The result was that predictions/actions were completely off and forms did not work correctly.

I upgraded my rasa x to rasa 2.8.x and have no issues so far.

Thanks a lot for the support! Describing my workflow made me realize my error

Topic		Replies	Views
Rasa X, I don't know what's wrong? [Deprecated] Rasa X Community Edition	7	870	April 17, 2020
Rasa X Not Loading Model [Deprecated] Rasa X Community Edition	2	918	June 24, 2021
Rasax and rasa shell descrepancy [Deprecated] Rasa X Community Edition	0	236	June 15, 2021
Rasa X model training [Deprecated] Rasa X Community Edition	6	797	May 19, 2021
Can't upload to model to Rasa X - Docker Compose [Deprecated] Rasa X Community Edition	11	701	April 6, 2022

Identical model behaving differently on Rasa VS Rasa X

Related topics