I am trying to Use FAQ and REsponse Selectors to handle FAQ’s related to Medicare. I have more than 3000 Q&A Pairs and some of them are very similar to each other.
When I load the FAQ’s, RASA always returns the last Response in the list - irrespective of the question I type.
I read the following topic - Is there any recommendation on how to resolve this?
Do you have train/test results from the response selector?
Please see training output for 3000 qNa’s:
2021-12-30 11:49:31 INFO rasa.nlu.model - Starting to train component ResponseSelector Epochs: 100%|██████████████████████████████████| 100/100 [46:38<00:00, 27.98s/it, t_loss=6.65, r_acc=0.0464] 2021-12-30 12:36:48 INFO rasa.nlu.model - Finished training component.
For 500 QNa’s please see training output:
2021-12-30 13:05:17 INFO rasa.nlu.model - Starting to train component ResponseSelector Epochs: 100%|███████████████████████████████████| 100/100 [03:15<00:00, 1.95s/it, t_loss=22.4, r_acc=0.733] 2021-12-30 13:08:36 INFO rasa.nlu.model - Finished training component.
The accuracy increases when I reduce to 500 and also the responses are better.
Hope this helps. Let me know if you need any other information.
Workaround that I implemented - RASA accurately predicts it is a FAQ but is unable to predict the correct instance of the FAQ (like faq/1234 ). So I am using the NLU to get the intent FAQ and then instead of using RASA for uttering the response, I use a custom action and just send the entire Text entered by the user to Azure QnA service via their API. Azure QnA does a great job of returning the correct response and I just utter that response in the custom action.
While this works, I would really like to move everything to rasa from Azure. Any help in resolving this will be truly apprecited.