Run Test that should fallback

Rasa Open Source
1

Hello,

I run test on my Bot with rasa test command

The Need :

I have some tests that are true negative; i.e sentences that should trigger a fallback, because it doesn’t have sense for the bot.

In order to do that I would like to write a test like these one :

True negative test

  • Fallback: These sentence has no sense
    • action_default_fallback

Problem :

there is no way to specify that the wanted intent is a fallback. The test will fail and here is what i get int the failed_stories.md

 ##  True negative testf (/tmp/tmpovhf9r0x/06160c3f9e394a3cb5f8d1cff7614337_conversation_tests.md)
 * Fallback: These sentence has no sense   <!-- predicted: Intent1: These sentence has no sense -->
     - action_default_fallback

Where Intent1 is an existing intent.

We can see with the following line that the test doesn’t succeed because it’s waiting for the first intent that match ( i.e.: Intent1) (even if the confidence is under the fallback threshold)

<!-- predicted: Intent1: These sentence has no sense -->

Is it a correct behaviour of the Rasa test function ? Any idea of a solution to bypass that ?

2

That is the behaviour expected - it predicts the most likely intent, regardless of confidence score. It still correctly predicted action_default_fallback though?

3

Yes it correctly predict action_default_fallback.

However, the test fails because of the predicted intent. I do not get why it is an expected behaviour, without taking into account the need to know if the bot can fallback correctly.

4

up

5

Sorry I missed this. That’s because in your test file, you are defining what is the expected output. “Fallback” is (probably) not a real intent, and that isn’t what is predicted. You could, if you wanted, write a test story with the actual intent, but put text you are sure will be categorized with low confidence, and then the fallback action. This should pass in testing, but I wouldn’t recommend it because you don’t want to test that some specific sentence will always fall back; you should rather check that fallback, in general, does what you expect given unexpected input. Fallback is really there for the cases that you can’t test, if that makes sense.