Hi - I’ve been looking for some guidance / best practices for “batch testing” with Rasa. I want to be able to save a bunch of inputs to Rasa in a file (the utterances a user would provide), feed them to Rasa one by one, and pipe the responses from Rasa to another file. The immediate use case is simple regression testing to ensure that updates to Python custom actions don’t break existing function.
Any recommendations on an approach to doing this kind of batch testing? Thanks.
Could you not just use the existing test features built into Rasa to accomplish the same thing? It basically runs through all your stories that you build as tests and if anything fails it writes it to a file to review. I have a post about this here, How to handle failed_stories in Evaluating core models
Would this give you what you want?
Thanks - I’ll try it out. I want to exercise a very broad variety of utterances within each story, so I’m not sure whether I can get the coverage I’m looking for iterating through stories. I’ll give it a shot and report back if it doesn’t do the trick. Thanks again for responding - much appreciated!
The approach described here worked for me: Communicating with Rasa in a local Python file
Basic working example: chatbot/test_client.py at master · ryanmark1867/chatbot · GitHub