Why does TwoStageFallbackPolicy keep asking for second affirmation even when the intent is correct?

Rasa Open Source
1

I have set up a TwoStagePolicy and it works as expected except in the following case:

image
image946×276 7.88 KB

In this example, the user affirms the rephrased intent but rasa keeps on asking for affirmation even when the user says “yes”. I expected the conversation to continue because as per the documentation:

  • If the user affirms the intent, the story continues as if the user had this intent from the beginning.

This is what the logs look like if I enabled debugging:

two-stage-infinite-loop
two-stage-infinite-loop1818×874 114 KB

This is how I set up the TwoStageFallbackPolicy in config.yml:

- name: TwoStageFallbackPolicy
  nlu_threshold: 0.8
  fallback_nlu_action_name: "action_start_over"

In domain.yml, I have added out_of_scope in intent, action_start_over in action, and utter_ask_rephrase in templates.

I expected the conversation to follow this story:

## happy path no relevant audience            
* greet              
  - utter_greet
* recommend_session
  - utter_ask_relevant_audience
* inform{"relevant_audience":"Data Scientists"}
  - action_recommend_session
  - slot{"speaker":"Justina"}
  - slot{"length":"5 min"}
  - slot{"abstract":"Workshop on chatbots"}
* length
  - utter_length
* speaker
  - utter_speaker
* thanks
  - utter_thanks

Is this a bug with TwoStageFallbackPolilcy or is it due to insufficient training data?

Please let me know if you need more information.

By the way, if I set core_threshold to 0.01, this is what happens:

01
011819×509 37.5 KB

There is no loop but the action is wrong.

I installed rasa using:

pip install rasa-x==0.19.6.dev336+gc771b11 --extra-index-url https://pypi.rasa.com/simple

2

Hey @hsm207. I think your should add more data to your NLU and Core models. When you test just an NLU model, does it perform well?

3

@Juste Testing just the NLU model, the performance is acceptable when inputs are similar to the training data.

So, I should add more stories like ## happy path no relevant audience so that the Core model knows to predict -utter_ask_relevant_audience if the intent is recommend_session with confidence higher than 0.30 (the default for core_threshold when TwoStageFallbackPolicy is used)?

4

I would suggest first improving the NLU model so that it learns to generalise better. So try adding more NLU examples first, test the performance and once it gets better in predicting inputs that are not in your training data, focus on stories.

5

@Juste thank you for your suggestion.

You don’t think this is a bug with TwoStageFallbackPolicy since the way it is behaving is not consistent with what is described in the documentation?

6

Hey @hsm207. To really tell I should see the files of your project and try to reproduce. Also, I can see that your NLU threshold is 0.8 which is very high for a fallback. Did you test it on a lower number for example 0.3?

7

@Juste Here are the project files.

Setting the NLU threshold to 0.30 also works most of the time.

This project isn’t something for production. It’s actually for a talk about rasa I plan to give so the data is deliberately kept small, like in the strata workshop you gave recently. I’m planning to go through the main features that make it easy to develop chatbots using rasa and just want to be sure if a feature does not work as intended, it really is due to lack of training data and not a bug in the code.

8

hey @Juste. Have you had a chance to look at the project files?