Rasa Fallback action failing for Faq Bot

Rasa Open Source
1

Hi , I have developed simple faq bot with rasa using Response selector and It is working fine . But when I implement Fallback Policy in the same and if a user asks anything which is not in the training questions it does not return with the default answer. Instead it gives one answer of faq question. Please let me know how to implement it correctly.

Stories file:

happy path

  • Greet
    • utter_greet
  • faq
    • respond_faq
  • Bye
    • utter_bye

Some question from FAQ

  • faq
    • respond_faq
  • Bye
    • utter_bye

Some random question

  • Greet
    • utter_greet
  • faq
    • utter_default
  • Bye
    • utter_bye

In Responses.md file I have written all the rsponses

Configuration for Rasa NLU.

Components

language: en pipeline:

  • name: SpacyNLP model: “en_core_web_md”
  • name: WhitespaceTokenizer
  • name: RegexFeaturizer
  • name: LexicalSyntacticFeaturizer
  • name: CountVectorsFeaturizer
  • name: CountVectorsFeaturizer analyzer: “char_wb” min_ngram: 1 max_ngram: 4
  • name: DIETClassifier epochs: 100
  • name: EntitySynonymMapper
  • name: ResponseSelector epochs: 200

Configuration for Rasa Core.

Policies

policies:

  • name: TEDPolicy
  • name: MemoizationPolicy max_history: 1
  • name: MappingPolicy
  • name: FallbackPolicy nlu_threshold: 0.90 core_threshold: 0.80

–Domain File
intents:

  • Greet
  • Bye
  • faq

actions:

  • respond_faq
  • utter_greet
  • utter_bye
  • utter_default

responses: utter_greet:

  • text: “Hey! How are you?”

utter_bye:

  • text: “Good bye , Have an Nice Day”

utter_default:

  • text: “Sorry I dont have answer for that.”

session_config: session_expiration_time: 01 carry_over_slots_to_new_session: false

Kindly let me know what changes I have to do to have fallback implemented

2

Hi @prapends. Could you try by increasing the max_history of MemoizationPolicy to 3 or 5 and see?

3

Hi,

I think the problem is with your stories: In the happy path and some random question, the intents are exactly the same. So when reaching the faq intent, your bot can follow either of those stories, they both match exactly.

What I would recommend is to remove the some random question story. Then, whenever the user’s question is confidently classified as faq intent, the bot will do respond_faq. And whenever the user’s question is too far away from the faq training data (i.e. not much closer to the faq examples than to e.g. the greet examples) and the nlu confidence falls below 0.90, the bot will fall back (due to the FallbackPolicy) and do utter_default.

It is possible that after that you will still have the same problem: If the user question is not in the training data but somewhat similar to the training data (e.g. also starting with “what”), it might still get classified confidently as faq.

To prevent this, you can add another intent out_of_scope and include a story

## some random question
* out_of_scope
  - utter_default

Then you just need to add some random questions that should not be covered by the faq as examples for the out_of_scope intent. Hopefully, random new questions will then be recognised as more similar to the out_of_scope than to faq. If not, this is not a problem of the policies but of the nlu that doesn’t recognise the correct intent. The best way to solve this is to simply add more training data, or you could also play around with different Intent Classifiers.

Does this help with your question?

Also, a general tip on how to investigate such issues: To understand what’s going on, it can often be helpful to see the nlu confidence and which policy was used to choose the next action. For that, you can run rasa shell --debug. This will show you information like

DEBUG    rasa.core.processor  - Received user message 'hi' with intent '{'name': 'greet', 'confidence': 0.9999992847442627}' and entities '[]'

and

DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_MemoizationPolicy

If you have any further questions, let me know!

1 Like