How can set a fallback policy if the response selector has a lower confidence level?

Hello everyone

We are currently using a Response Selector to use FAQs for our bot, but we have an issue where the user said a message and Rasa matches the message with the intent faq, but when I look the confidence in the ranking of the FAQs is below 0.3, how can we add a fallback that checks the confidence level of the actual FAQ that it’s being selected and not only the intent itself? Do we have to use an out-of-scope intent?

Thank you and let me know if you need more information

Hi @DanielOlarte . Yes, in general, you’d need a fallback intent. Have you tried including any of the Fallback handling techniques in your assistant?

Yes, I already have this in my config.yml

language: en
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
    retrieval_intent: faq
  - name: FallbackClassifier
    threshold: 0.75
policies:
  - name: MemoizationPolicy
    max_history: 5
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: RulePolicy
    core_fallback_threshold: 0.8
    core_fallback_action_name: action_default_fallback
    enable_fallback_prediction: True

It seems that with the requests it detects FAQ intent, but the ranking of the actual responses are very low

Hi! Just wanted to know if someone has any idea on how to solve the issue? Thanks!

@Juste Hi! Sorry to bother you again, can you please take a look at this issue? Thank you!

Looking for the same thing! @dakshvar22

@DanielOlarte @zack This is currently not possible. Can you one of you please open a feature enhancement request on rasa open source repo?

@dakshvar22 Yes, sure. I’ll do it. We created a custom response selector based on the default ResponseSelector to check for confidence inside the selector.

Also, could you take a look at this: Contradicting rules or stories found with categorical slots.

Thanks