When implementing my chatbot I came across the need to define a fallback intent, an intent that is classified when confidence is below threshold. I was surprised not find any way to do this with the normal components and policies. Fallback policy only lets us define a fallback action, not an intent. 2-stage fallback is able to do that but only on the last stage. Weird design decision in my opinion.
I found this forum post with a custom component in the NLU pipeline to implement such behavior. Anyway it is from 2018 and I’m not sure if its still up to date. Why does rasa not implement such a feature? Is there a better (official, supported) way to implement it 2 years later?
What do you guys think about using that custom component? Do you have any concerns about how to combine it with the Fallback / 2-Stage-Fallback Policy?
Hi Vale, great question! I wasn’t around for the original choice, but there’s some discussion of the general design decision in this github issue
To quickly summarize: the initial proposed approach was to have a null intent classification for utterances that fell out of scope & the confidence threshold and fallback policy model was adopted instead.
To me, this makes sense because an intent is supposed to be capturing something your users want to do, whereas actions are things your assistant does. Your users (hopefully!) don’t want to provide utterances that are confusing for your assistant, and the difference between something falling above and below the confidence threshold is in what your assistant does. You could add your own very broad intent and use that as the fallback but I would guess that that would dramatically reduce overall intent classification accuracy, especially if you had a lot of intents.
To answer your more general question about doing X and then doing Y, there’s a branch in development to add rule-based behavior.
Quick update about having an intent for things that are out of scope: my colleagues say that it can improve performance, but should be a little narrower than I was envisioning it. I was thinking extremely high entropy/basically random examples so that it would catch anything that didn’t fall into another category. A better approach is to have specific things someone might ask that would be out of scope.
For example, a financial bot that can only currently check someone’s account balance may have out of scope examples like “what’s my credit score?”, “can I make a wire transfer” or “can you reset my password” if it can’t do any of those things.
