I have an intent
need_registration with 40 examples (e.g., do I need to be registered in service_x to pay with it?, do I need to have an account in service_x to use it?’), where 12 out of those 40 examples contain the word “need”. The other intents have many more examples (~ 600).
Using the DIETClassifier with word and character n-grams sparse features, and training for 40 epochs, the single word “need” is classified as intent
need_registration with high confidence (0.94). If I only train for 20 epochs, the confidence is still relatively high (0.62), but below 0.7.
I am trying to understand if this is something to be expected in special cases like this one or in general. Or if this is otherwise something unexpected. In any case, I would appreciate strategies to prevent it.
My intuition right now tells me this could be related to the use of balanced batches when training the DIETClassifier. Since the problematic intent has many fewer examples, the same examples might be being sampled too often. Could this be the reason? How exactly are batches balanced?