Testing your bot

Rasa Open Source
1

How do you go about testing your bot beyond a simple proof of concept? Is there any testing tool/process you follow?

I am building a bot for our internal sales people. So far I have tested it by putting all possible conversation flow in the stories file. We want to go to a best test phase with our customer and want to make sure we are covering all the corners cases.

Please suggest.

1 Like
2

A colleague shared some experimental testing framework for rasa-core : https://github.com/mrbot-ai/rasa-addons

I’m sure he would be happy to have feedback.

2 Likes
3

Hi, we are also starting out on our test strategy for chatbots( It is a bit tricky). So far I have got this

  • Unit Tests( We use botkit and some custom integration for which we have unit tests)
  • Integration Tests ( For now we do it at botkit level, we write tons of stories in a simple JSON file and run them all together against our stack to see if anything breaks in terms of connections only. trying to cover all the happy flows)
  • Continuos Model Integration Tests - Probably the most tricky one, as understandably not all tests will pass but ideally to benchmark a model performance ( this is separate from cross-validation). we plan to do this at the time of deploying a model to the chatbot for use with actual human testers putting the stories in a JSON file( possibly via UI) written by business/functional testers and evaluate the performance of the bot in every sense (disambiguation, fallback etc etc). You need creative testers with a bit of customer profile understanding to test the most likely scenarios. It is impossible to cover all cases. Another great way to test this is with your beta testers( actual customers) who can provide feedback on a rolling basis so probably an even distribution of customers on each update

Of course this is all theory (well some is already implemented) but I am sharing just so to get feedback on the strategy as well. :slight_smile: we built the tool ourselves but I like the above idea as well. Quite similar to what we did

3 Likes
4

@souvikg10 your ‘Continuos Model Integration Tests’ approach will help a lot but seems will add delay in the process and not scalable. How will you get the actual human tester? Your own developers? or Amazon Mturk or similar?

5

We are a big organisation. We are still brainstorming and i brainstorm with the people of the world :smiley:

We are looking at multilingual bots and probably for more than one internal business looking towards their own customer so the idea is to build a funnel model and provide chatbots as a platform powered by Rasa. So we believe the businesses should fund the testers for functional integration each time they wish to add a new conversation to their bot. The core model integration remains with the central team meaning we are most likely to manage our pipelines in terms of model training and deployment however when it comes to testing a business level integration on whether the supposed conversation actually works or not, it will be the businesses

1 Like
6

HI I am getting following error during testing. Do you know how to resolve?\

## CHITCHAT.GREET
* /chitchat.greet

  - Sorry, I am unable to understand that. (exp: utter_greet)
TEST CASE INTERRUPTED)

## CHITCHAT.GOODBYE
* /chitchat.goodbye

  - Sorry, I am unable to understand that. (exp: utter_goodbye)
TEST CASE INTERRUPTED)