Hello everyone We are currently using a Response Selector to use FAQs for our bot, but we have an issue where the user said a message and Rasa matches the message with the intent faq, but when I look the confidence in the ranking of the FAQs is below 0.3, how can we add a fallback that checks the confidence level of the actual FAQ that it’s being selected and not only the intent itself? Do we have to use an out-of-scope intent? Thank you and let me know if you need more information

Hi @DanielOlarte . Yes, in general, you’d need a fallback intent. Have you tried including any of the Fallback handling techniques in your assistant?

Yes, I already have this in my config.yml language: en pipeline: - name: WhitespaceTokenizer - name: RegexFeaturizer - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 - name: EntitySynonymMapper - name: ResponseSelector epochs: 100 retrieval_intent: faq - name: FallbackClassifier threshold: 0.75 policies: - name: MemoizationPolicy max_history: 5 - name: TEDPolicy max_history: 5 epochs: 100 - name: RulePolicy core_fallback_threshold: 0.8 core_fallback_action_name: action_default_fallback enable_fallback_prediction: True It seems that with the requests it detects FAQ intent, but the ranking of the actual responses are very low

Hi! Just wanted to know if someone has any idea on how to solve the issue? Thanks!

@Juste Hi! Sorry to bother you again, can you please take a look at this issue? Thank you!

Looking for the same thing! @dakshvar22

@DanielOlarte @zack This is currently not possible. Can you one of you please open a feature enhancement request on rasa open source repo?

@dakshvar22 Yes, sure. I’ll do it. We created a custom response selector based on the default ResponseSelector to check for confidence inside the selector. Also, could you take a look at this: Contradicting rules or stories found with categorical slots . Thanks

I am interested in this also. Do you have a feature request issue number?

I have the same issue for an FAQ bot use case. @dakshvar22 Is there a plan for implementing this feature? @DanielOlarte Can you share your custom response selector?

@ali.karimi I didn’t create a feature request, my bad. We created a custom component that extends the regular ResponseSelector. We override the process function and we use the same thing as the base code that you can find on their source code, and then, at the end of the code, we added validation for the top_label confidence key and if it was lower than our set threshold, we changed the confidence of that given FAQ. I don’t have access to the code now, but this is the main idea.

How can set a fallback policy if the response selector has a lower confidence level?

Rasa Open Source

ChineseZheng (Yutin) July 6, 2022, 1:34am 16

There are some method I’ve verified.You can use any or all of them to deal with your problems
1\use utter_default

2\use out of scope

3\use nlu fallback
4\use rephrase

Topic		Replies	Views
Is there a way to add fallback for the retrieval intents (Response Selectors) when the confidennce is Low Rasa Open Source	0	384	February 11, 2022
Response Selector Fallback Mechanism Rasa Open Source	2	379	October 6, 2023
Response Selector Fallback action Rasa Open Source	2	612	September 28, 2021
Rasa Fallback action failing for Faq Bot Rasa Open Source	2	1342	July 28, 2020
FallbackClassifier not working when using faq and chitchat Rasa Open Source	4	893	August 17, 2023

How can set a fallback policy if the response selector has a lower confidence level?

Related topics