Hello everyone,
I have encountered several wrong responses when starting Rasa X. They are “401” or “Unauthorized” errors and the request are always the same. It doesn’t happen all the time, but 10-20 % and is a bit random. It only occurs at the beginning of Rasa X (the first time you enter the URL in a browser). Rasa X works fine if you log in, but the user experience is not quite good if this happens (Rasa X will be used by a customer)
- Error detail
For example, the following images show the “data URLs” loaded when accessing to Rasa X before logging in (they are temporally sorted in ascending order). I believe the error happens because Rasa X “thinks” there is a user already logged in. I have found that the image “SaraWizard.23524.ae2.svg” is loaded when the error occurs, but should be loaded after you put the credentials (it is not in OK -1 and is in OK -2). Also, I noticed that “https://api/version?skip_external={value}” is set up to False when it fails and to True when it is OK. And I also checked that all failing requests are responded correctly when there is no failure.
- Example error
- OK - 1
- OK - 2
I’ve installed Rasa X locally inside an Ubuntu 20.04 Docker container and run it with the following command:
/usr/bin/python3 /usr/local/bin/rasa x --no-prompt --log-file log_rasa_x.log
The installed versions are:
# rasa --version
Rasa Version : 2.8.15
Minimum Compatible Version: 2.8.9
Rasa SDK Version : 2.8.2
Rasa X Version : 1.0.0
Python Version : 3.8.10
Operating System : Linux-3.10.0-1160.45.1.el7.x86_64-x86_64-with-glibc2.29
Python Path : /usr/bin/python3
Here is the log of Rasa X. I have checked that it appears as “sanic_jwt.authentication - (‘Signature verification failed’,)”. And as you can see in the log, it is a little after “New websocket connected”. I’m not sure if the error “rasax.community.services.stack_service - Error retrieving Rasa Open Source version. Cannot connect to host stuff:5005 ssl:default [Name or service not known]” is also involved because sometimes it appears when Rasa X is running correctly.
The server is running at http://localhost:5002/login?username=admin&password=pass
[2022-03-10 13:39:02 +0100] [1299] [INFO] Starting worker [1299]
2022-03-10 13:39:02 INFO sanic.root - Starting worker [1299]
2022-03-10 13:39:03 DEBUG rasa.core.agent - Requesting model from server http://localhost:5002/api/projects/default/models/tags/production...
/usr/local/lib/python3.8/dist-packages/rasax/community/api/blueprints/models.py:134: FutureWarning: The "GET /projects/<project_id>/models/tags/<tag>" endpoint is deprecated. Please use "GET /models/tags/<tag>" instead".
2022-03-10 13:39:03 DEBUG rasax.community.services.model_service - Checking the model discovery from model_for_tag.
2022-03-10 13:39:03 DEBUG rasax.community.services.model_service - Model discovery completed.
2022-03-10 13:39:03 DEBUG rasax.community.database.utils - Turned SQLite foreign key enforcement on.
2022-03-10 13:39:03 DEBUG rasax.community.database.utils - Turned on SQLite WAL mode.
2022-03-10 13:39:06 DEBUG rasa.core.agent - Unzipped model to '/tmp/tmpwwctn4bq'
2022-03-10 13:39:06 DEBUG rasa.core.agent - Found new model with fingerprint f450d5312097f1e395aca253c61e13d2. Loading...
2022-03-10 13:39:06 DEBUG rasa.utils.tensorflow.models - Loading the model from /tmp/tmpwwctn4bq/nlu/component_5_DIETClassifier.tf_model with finetune_mode=False...
2022-03-10 13:39:06 DEBUG rasa.nlu.classifiers.diet_classifier - Following metrics will be logged during training:
2022-03-10 13:39:06 DEBUG rasa.nlu.classifiers.diet_classifier - t_loss (total loss)
2022-03-10 13:39:06 DEBUG rasa.nlu.classifiers.diet_classifier - i_acc (intent acc)
2022-03-10 13:39:06 DEBUG rasa.nlu.classifiers.diet_classifier - i_loss (intent loss)
2022-03-10 13:39:19 DEBUG rasax.community.api.blueprints.websocket - New websocket connected.
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned SQLite foreign key enforcement on.
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned on SQLite WAL mode.
2022-03-10 13:39:19 DEBUG rasax.community.services.stack_service - Error when retrieving version from Rasa Open Source. Cannot connect to host stuff:5005 ssl:default [Name or service not known]
2022-03-10 13:39:19 DEBUG sanic_jwt.authentication - ('Signature verification failed',)
2022-03-10 13:39:19 DEBUG urllib3.connectionpool - Starting new HTTPS connection (1): o251570.ingest.sentry.io:443
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned SQLite foreign key enforcement on.
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned on SQLite WAL mode.
2022-03-10 13:39:19 DEBUG sanic_jwt.authentication - ('Signature verification failed',)
2022-03-10 13:39:19 DEBUG rasax.community.services.stack_service - Error when retrieving version from Rasa Open Source. Cannot connect to host stuff:5005 ssl:default [Name or service not known]
2022-03-10 13:39:19 DEBUG sanic_jwt.authentication - ('Signature verification failed',)
2022-03-10 13:39:19 DEBUG urllib3.connectionpool - https://o251570.ingest.sentry.io:443 "POST /api/5870793/store/ HTTP/1.1" 200 41
2022-03-10 13:39:19 DEBUG sanic_jwt.authentication - ('Signature verification failed',)
2022-03-10 13:39:19 DEBUG urllib3.connectionpool - https://o251570.ingest.sentry.io:443 "POST /api/5870793/store/ HTTP/1.1" 200 41
2022-03-10 13:39:19 DEBUG sanic_jwt.authentication - ('Signature verification failed',)
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned SQLite foreign key enforcement on.
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned on SQLite WAL mode.
2022-03-10 13:39:19 DEBUG urllib3.connectionpool - https://o251570.ingest.sentry.io:443 "POST /api/5870793/store/ HTTP/1.1" 200 41
2022-03-10 13:39:19 DEBUG sanic_jwt.authentication - ('Signature verification failed',)
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned SQLite foreign key enforcement on.
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned on SQLite WAL mode.
2022-03-10 13:39:19 DEBUG rasax.community.services.stack_service - Error when retrieving version from Rasa Open Source. Cannot connect to host stuff:5005 ssl:default [Name or service not known]
2022-03-10 13:39:19 DEBUG urllib3.connectionpool - https://o251570.ingest.sentry.io:443 "POST /api/5870793/store/ HTTP/1.1" 200 41
2022-03-10 13:39:19 DEBUG rasa.utils.tensorflow.models - Finished loading the model.
/usr/local/lib/python3.8/dist-packages/rasa/utils/train_utils.py:641: UserWarning: constrain_similarities is set to `False`. It is recommended to set it to `True` when using cross-entropy loss. It will be set to `True` by default, Rasa Open Source 3.0.0 onwards.
rasa.shared.utils.io.raise_warning(
2022-03-10 13:39:19 DEBUG rasa.nlu.classifiers.diet_classifier - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpwwctn4bq/nlu' doesn't exist?
2022-03-10 13:39:19 DEBUG rasa.utils.tensorflow.models - Loading the model from /tmp/tmpwwctn4bq/core/policy_1_TEDPolicy/ted_policy.tf_model with finetune_mode=False...
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned SQLite foreign key enforcement on.
2022-03-10 13:39:19 DEBUG rasax.community.database.utils - Turned on SQLite WAL mode.
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/cond_grad/Identity_1:0", shape=(None,), dtype=int64), values=Tensor("gradients/cond_grad/Identity:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/cond_grad/Identity_2:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:447: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/cond_1_grad/Identity_1:0", shape=(None,), dtype=int64), values=Tensor("gradients/cond_1_grad/Identity:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/cond_1_grad/Identity_2:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
^CGoodbye 👋
Goodbye 👋
[2022-03-10 13:39:23 +0100] [1299] [INFO] Stopping worker [1299]
2022-03-10 13:39:23 INFO sanic.root - Stopping worker [1299]
2022-03-10 13:39:23 DEBUG rasax.community.api.blueprints.websocket - WebSocket connection to 'None' was closed.
2022-03-10 13:39:23 WARNING rasax.community.local - Stopping event service.
2022-03-10 13:39:34 DEBUG rasa.utils.tensorflow.models - Finished loading the model.
2022-03-10 13:39:34 DEBUG rasa.core.agent - Finished updating agent to new model.
[2022-03-10 13:39:35 +0100] [1299] [INFO] Server Stopped
2022-03-10 13:39:35 INFO sanic.root - Server Stopped
root@e21ea3dd9c1e:~/app#`
And here all the python packages installed:
root@16b98d1c023b:~/app# pip3 list
Package Version
----------------------- ------------
absl-py 0.13.0
aio-pika 6.8.0
aiofiles 0.7.0
aiohttp 3.7.4
aioredis 2.0.0
aiormq 3.3.1
alembic 1.4.3
APScheduler 3.7.0
astunparse 1.6.3
async-generator 1.10
async-timeout 3.0.1
attrs 19.3.0
bidict 0.21.3
boto3 1.18.42
botocore 1.21.42
CacheControl 0.12.10
cachetools 4.2.2
certifi 2021.5.30
cffi 1.14.6
chardet 3.0.4
clang 5.0
cloudpickle 1.6.0
colorclass 2.2.0
coloredlogs 15.0.1
colorhash 1.0.3
croniter 1.0.15
cryptography 3.4.8
cycler 0.10.0
decorator 4.4.2
dm-tree 0.1.6
dnspython 1.16.0
docopt 0.6.2
fbmessenger 6.0.0
flatbuffers 1.12
future 0.18.2
gast 0.4.0
gitdb 4.0.7
GitPython 3.1.20
google-auth 1.35.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
greenlet 1.1.1
grpcio 1.40.0
h11 0.9.0
h2 3.2.0
h5py 3.1.0
hpack 3.0.0
hstspreload 2021.9.1
httptools 0.3.0
httpx 0.11.1
humanfriendly 9.2
hyperframe 5.2.0
idna 2.10
ipaddress 1.0.23
isodate 0.6.0
jmespath 0.10.0
joblib 1.0.1
jsonpickle 2.0.0
jsonschema 3.2.0
kafka-python 2.0.2
keras 2.6.0
Keras-Preprocessing 1.1.2
kiwisolver 1.3.1
lxml 4.6.3
Mako 1.1.5
Markdown 3.3.4
MarkupSafe 2.0.1
matplotlib 3.3.4
mattermostwrapper 2.2
msgpack 1.0.3
multidict 4.7.6
networkx 2.5.1
nob 0.6.1
numpy 1.19.5
oauthlib 3.1.1
opt-einsum 3.3.0
packaging 20.9
pamqp 2.3.0
pika 1.2.0
Pillow 8.3.2
pip 22.0.4
prompt-toolkit 2.0.10
protobuf 3.18.0
psycopg2-binary 2.9.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.20
pydot 1.4.2
PyJWT 2.0.0
pykwalify 1.8.0
pymongo 3.10.1
pyparsing 2.4.7
pyrsistent 0.18.0
pyTelegramBotAPI 3.8.3
python-crfsuite 0.9.7
python-dateutil 2.8.2
python-editor 1.0.4
python-engineio 4.2.1
python-socketio 5.4.0
python3-saml 1.12.0
pytz 2021.1
questionary 1.5.2
rasa 2.8.15
rasa-sdk 2.8.2
rasa-x 1.0.0
redis 3.5.3
regex 2021.7.6
requests 2.25.1
requests-oauthlib 1.3.0
requests-toolbelt 0.9.1
rfc3986 1.5.0
rocketchat-API 1.16.0
rsa 4.7.2
ruamel.yaml 0.16.13
ruamel.yaml.clib 0.2.6
s3transfer 0.5.0
sanic 20.3.0
Sanic-Cors 0.10.0.post3
sanic-jwt 1.6.0
Sanic-Plugins-Framework 0.9.5
scikit-learn 0.24.2
scipy 1.5.4
sentry-sdk 1.2.0
setuptools 45.2.0
six 1.15.0
sklearn-crfsuite 0.3.6
slackclient 2.9.3
smmap 4.0.0
sniffio 1.2.0
SQLAlchemy 1.4.23
tabulate 0.8.9
tarsafe 0.0.3
tensorboard 2.6.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
tensorflow 2.6.2
tensorflow-addons 0.14.0
tensorflow-estimator 2.6.0
tensorflow-hub 0.12.0
tensorflow-probability 0.13.0
tensorflow-text 2.6.0
termcolor 1.1.0
terminaltables 3.1.0
threadpoolctl 2.2.0
tqdm 4.62.2
twilio 6.50.1
typeguard 2.12.1
typing-extensions 3.7.4.3
tzlocal 2.1
ujson 1.35
urllib3 1.26.6
uvloop 0.14.0
wcwidth 0.2.5
webexteamssdk 1.6
websockets 8.0.2
Werkzeug 2.0.1
wheel 0.37.0
wrapt 1.12.1
xmlsec 1.3.12
yarl 1.6.3
I would like to know if anyone can help me figure out what is going on.
Thanks