Hello ppl,
I have RasaX deployed in a k8s cluster, overall the deploy is fine and I can update and train new models with no issues, but I found some strange behaviour regarding repository sync and model training.
Issue: RasaX stops training new models despite being synced with git repository.
Let me desribe my environment first:
- RasaX running on a k8s cluster (used the Helm template)
- RasaX Version 0.40.1
- Rasa 2.6.2-full
How to replicate the issue:
- Have a working, trained and active model
- Make a “mistake” in the domain.yml in the git repository (a few spaces in an utter to break indentation will do the trick)
- Add something to domain.yml (like a new answer for an utter)
- Refresh RasaX. It will say it’s synced with git repository
- Try to train the “new” model. It says “nothing changed”
- Check domain.yml in RasaX, it’s not up to date with the repository (maybe because the domain.yml in the repository is wrong and RasaX knows?).
- In git repository, “fix” the mistake.
- Refresh RasaX to check if it’s synced with git repository again 9 . Try to train the new “fixed” model. It will say “nothing changed”
- Check domain.yml in RasaX, it’s still not up to date with the repository (this is the strange part)
- No new model trained
- Oh nooo
The "Fix"
The “fix” I found for this was to delete the current model to force RasaX do train a new and updated one.
My questions:
- Was this supposed to happen?
- The current model is working and it’s current domain.yml is fine, since RasaX didn’t train a new model with the wrong domain.yml. If that’s the case, why when the mistake is fixed in the repository RasaX won’t train a new one?
Hope you guys can replicate the problem and tell me if this is an expected behaviour or not.
Best regards.
