How can set a fallback policy if the response selector has a lower confidence level?

Rasa Open Source
1

Hello everyone

We are currently using a Response Selector to use FAQs for our bot, but we have an issue where the user said a message and Rasa matches the message with the intent faq, but when I look the confidence in the ranking of the FAQs is below 0.3, how can we add a fallback that checks the confidence level of the actual FAQ that it’s being selected and not only the intent itself? Do we have to use an out-of-scope intent?

Thank you and let me know if you need more information

1 Like
2

Hi @DanielOlarte . Yes, in general, you’d need a fallback intent. Have you tried including any of the Fallback handling techniques in your assistant?

3

Yes, I already have this in my config.yml

language: en
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
    retrieval_intent: faq
  - name: FallbackClassifier
    threshold: 0.75
policies:
  - name: MemoizationPolicy
    max_history: 5
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: RulePolicy
    core_fallback_threshold: 0.8
    core_fallback_action_name: action_default_fallback
    enable_fallback_prediction: True

It seems that with the requests it detects FAQ intent, but the ranking of the actual responses are very low

4

Hi! Just wanted to know if someone has any idea on how to solve the issue? Thanks!

5

@Juste Hi! Sorry to bother you again, can you please take a look at this issue? Thank you!

6

Looking for the same thing! @dakshvar22

7

@DanielOlarte @zack This is currently not possible. Can you one of you please open a feature enhancement request on rasa open source repo?

8

@dakshvar22 Yes, sure. I’ll do it. We created a custom response selector based on the default ResponseSelector to check for confidence inside the selector.

Also, could you take a look at this: Contradicting rules or stories found with categorical slots.

Thanks

9

I am interested in this also. Do you have a feature request issue number?

10

I have the same issue for an FAQ bot use case. @dakshvar22 Is there a plan for implementing this feature?

@DanielOlarte Can you share your custom response selector?

11

@ali.karimi I didn’t create a feature request, my bad.

We created a custom component that extends the regular ResponseSelector. We override the process function and we use the same thing as the base code that you can find on their source code, and then, at the end of the code, we added validation for the top_label confidence key and if it was lower than our set threshold, we changed the confidence of that given FAQ.

I don’t have access to the code now, but this is the main idea.

1 Like
12

FYI: https://github.com/RasaHQ/rasa/issues/9751

1 Like
13

I have the same issue for an FAQ bot use case. @dakshvar22 Is there a plan for implementing this feature?

@ali.karimi @DanielOlarte I really appreciate all the input here and issue. We will not be able to fit this on the roadmap before the end of 2021 and are still figuring out what problems we plan to solve in 2022, so we do not have plans to solve this problem yet

1 Like
14

If somewhere is a poll for important problems to solve, I would vote on this :smiley:

15

Hi everyone! Faced same problem and with the help of an advice from @DanielOlarte I did extend ResponseSelector class and it works.

For those who wanted to see some snipped(as myself):

  • in custom_component.py created class CustomResponseSelector extension of class ResponseSelector(which is here)

  • did overwrite process function - just added these lines at the end, replacing FAQ intent confidence with its value from response selector, applied only if FAQ intent was predicted from intent classfier:

        predicted_intent = message.get(INTENT)
        if predicted_intent and predicted_intent["name"] == selector_key:
            predicted_intent["confidence"] = top_label[PREDICTED_CONFIDENCE_KEY]```
  • in config.yml use it as:
     - name: custom_components.CustomResponseSelector
16

There are some method I’ve verified.You can use any or all of them to deal with your problems
1\use utter_default
image

2\use out of scope
image

3\use nlu fallback
image 4\use rephrase
image

17

I totally agree with you, I am voting for this also!