Hey everyone,
We are working on a chatbot use case in which the data is not so clean and balanced. Our initial attempt was to try out Rasa DIET for NLU and got some decent results but the issue we identified of DIET was with false positive being inferred with very high accuracy we can reduce this by using out_of_scope intent to an extend and accuracy issues due to class imbalance. So we then tried out a custom classification model using distilled BERT with tuned class weights and the results were more generalised and better compared to DIET.
Now the scenario is to use our own pytorch based Intent classifier for intent classification and rest of rasa components like entity extraction, tracker and dialogue management with RASA. The rules/story has to work based on the intent identified from the custom model instead of rasa NLU.
Is there any feasible solutions for the same ? Came across an blog post