I have built an NLU model that does great for the true positive phrases. All the phrases and their variations from replacing the placeholders with the accepted entities are being detected. However, when I input a random phrase expecting it return a fallback, it’s predicting as one of the known intents. When testing it against the trained phrases and all combinations of those phrases with different entities in them, I got 100% accuracy for both the intent and entity side with a minimum 88% confidence. Does this mean I increase the fallback threshold for confidence score or do I decrease my training epochs? After training under 300 epochs these are the stats I got: [00:28<00:00, 10.37it/s, loss=0.009, acc=1.000]
I think the best way to increase your detection of negatives is to have a true negative intent, i.e. something like out_of_scope where you collect phrases that should lead to fallback. Then you can just include that intent leading to fallback in your stories as well.
Thanks for your reply. However, this can be very problematic because of the infinite possibilities. I had luck falling back on phrases with keywords that aren’t part of my training data vocabulary. However, I need to know how to fallback for tricky test cases where majority of the words are matching but a keyword is replaced with a random word. Is there a way to achieve fallback based on semantic meaning of the phrases as well?
I’m not sure how semantic in general the NLU gets – you may need to look into changing configuration parameters (using bi- or trigrams etc) or creating your own custom component to parse more semantic information to handle those edge cases.
I tried out your suggestion and it worked surprisingly well when training the fallback intent in similar phrases with different keywords. Ex: “My top deals” and “My top cars” would predict to TopDeals intent even though “My top cars” was never a training phrase and is out of my NLU’s scope, but when training “my top cars” or “my top players” under fallback, it forces the NLU to predict TopDeals only when “top deals” exists in the phrase and “my top ___” variants predict under the fallback intent.
Any other unrecognizable phrase would then be too different from the trained phrases so they would either predict as none or fall below my confidence score threshold to be considered a fallback. Therefore, only training the fallback intent with the phrases similar to the ones from all my other intents will allow it to keep the training file cleaner. This method fortunately didn’t affect the known phrases from predicted under their target intents.