Is there any preferred approach to automated testing, or any recommended framework available?
We are considering employing RASA for a chatbot but would like the confidence of a TDD based approach. After one has sorted out a test runner, I believe the process would be similar to any other model. One would:
Does anyone have any suggestions or thoughts?
Many thanks in advance, Tyler
Hi @jabberlope! I think you will struggle to be able to use a TDD-based approach because it is really hard to anticipate how users will use your bot and for you to know everything in advance. I think you will be better off designing its capabilities and then getting it in the hands of real people as soon as possible.
The way I would approach it is by coming up with a dozen or so utterance examples per intent and some stories that you think users will follow. Then, I would use the share your bot feature in Rasa X and get your friends/coworkers to play around with the bot. From there, you can review the real conversations they have with it, correct any NLU or Core mistakes, add them to your training data, and retrain your model.
Once it gets good enough, then you can follow this same process with real users. I personally think automated testing is more important as a way to ensure that as you add more and more training data, you do not end up with regressions. I would also recommend checking out End-to-End Evaluation in the Rasa Docs
@tyd man thanks for the feedback. I had my suspicions about the limits to an automated workflow, but it was good to have them confirmed.
Also, big props for the E2E evaluation link you posted, I had no idea that there was a rasa test command. In addition, that page provides a whole lot to digest in regards to evaluating models programatically.
I went on this journey as well and found a testing framework called Botium (https://botium.com)
I love it and it was exactly what I was looking for. Totally integrates with Rasa with a connector you add (GitHub - codeforequity-at/botium-connector-rasa: Botium Connector for Rasa) It has an emulator that will run your bot like rasa shell but in a nicer window. There’s also a GUI tool you can install called Botium Box, there’s both a community edition and a paid version. I haven’t tried those out yet, but they’re on my list.
You should check it out.
Hope it helps
I’m having some problems with ESOCKETTIMEDOUT exception using botium . Some functions take almost 1 minute and botium returns that exception. Did you have the same problem?
Ooo no I didn’t have that problem, but then again I was using it quite a while ago. I haven’t tried their latest version. I think I was using the Rasa API not socket.io
We’ve shifted to Rasa’s built in tests, but I’d like to get back into using Botium
Sorry I couldn’t be of help.