Version: Rasa X 0.37.1
Rasa-X Helm Chart installation
rasax:
# tag refers to the Rasa X image tag
tag: "0.37.1"
# rasa: Settings common to all Rasa containers
rasa:
# tag refers to the Rasa image tag
tag: "2.3.1-full"
additionalChannelCredentials:
rest:
When I set a model to train over the UI (Training → Update Model → Train Model), I get an Error Message telling me training failed and nothing else:
But if I check the logs in the production container, I see that the training keeps on going.
If I check in the Rasa X Models section after a while I see a new Model there. If I activate it, it works perfectly even though it “Failed”.
2021-05-31 10:57:34 INFO rasa.validator - Validating utterances…
/srv/shared/huijh03/venv_rasa2.3/lib64/python3.6/site-packages/rasa/shared/utils/io.py:93: UserWarning: The action ‘utter_ict_faq’ is used in the stories, but is not a valid utterance action. Please make sure the action is listed in your domain and there is a template defined with its name.
More info at Actions
Project validation completed with errors.
UserWarning: Action 'utter_faq' is listed as a response action in the domain file, but there is no matching response defined. Please check your domain.
I’ve upgraded Rasa-x to 0.40 and OS to 2.6 and the problem got worse. Now I get the “Training failed” message, but the model doesn’t appear in the models list after a while
.
I tried to do some digging and found following errors and logs.
In the browser dev tools:
Request URL:
https://chatbot.url/api/projects/default/models/jobs
Request Method:
POST
Status Code:
504 Gateway Time-out
Remote Address:
10.10.43.190:443
Referrer Policy:
strict-origin-when-cross-origin
Note that at the time the ingress-controller logs a 504 (Gateway Timeout) there has not been a response from rasa-x-nginx yet.
The 500 (Internal Server Error) logged by the rasa-x-nginx is probably unrelated. It only started happening recently and happens when the rasa-x-worker pod crashes (17 minutes later). I will open a separate post about this.