A/B testing multiple bots

Rasa Open Source
1

Is there a way to A/B test bots built with rasa?

I have different flows for a product and I want to try and evaluate those.

Is there any native way to do A/B testing?

More specifically, A/B testing for dialogue model only not NLU.

Thanks,

2

@akelad @Tobias_Wochinger Any ideas?

3

One way to do this is:

  1. Set up multiple rasa instances (e.g. rasa A, rasa B)
  2. Send user messages to a different rasa instance based on their user id / session (one user should only speak to one rasa instance)
  3. After your A/B testing analyse the tracker store contents
4

Thanks.

Wondering if there’s a scope to support A/B testing natively in Rasa. Can help with this if anybody is interested.

1 Like