I really like the idea of the two stage fallback, however, if you have a lot of intents that are often only differing in the depth of the details. We have a lot of customer processes for which we want to be able to allow our customers jump right into instead of starting with generic stories only, so we have quite a bunch of similar intents. Of course that leads to situations in which the distinction between the intents is hard, especially if users hit phrases that are combined from multiple similar intent examples.
I found out that creating “disambiguation intents” does not make the situation better, as the NLU data gets just more data which is quite alike.
Instead, I thought about creating an intent naming scheme like maintopic_topic_detail and create a pipeline component that grabs the recognized intents (comparable to the NluFallbackClassifier) and checks if the distance between the top 2 intents is below threshold. But instead of creating a generic nlu_fallback intent, I check the list of recognized intents and try to find a common (or majority vote) root intent, say 3/4 of the recognized intents start with the same maintopic_ - in that case I can issue an intent of maintopic_fallback to ask the customer if the topic was correctly recognized and start asking detail questions from there.
This approach however has issues with using SoftMax on the confidences, because that makes the measurement of distances between the utterance and the intents even harder.
Do you, @koaning , see any major issues with that approach? I am going to experiment with triplet-loss once spare time allows, to see if I can get good thresholds for both ambiguity and minimum confidence.