RASA FAQ - Similarity and Upper Limit for QnA pairs

I am trying to Use FAQ and REsponse Selectors to handle FAQ’s related to Medicare. I have more than 3000 Q&A Pairs and some of them are very similar to each other.

When I load the FAQ’s, RASA always returns the last Response in the list - irrespective of the question I type.

I read the following topic - Is there any recommendation on how to resolve this?

Do you have train/test results from the response selector?

Please see training output for 3000 qNa’s:

2021-12-30 11:49:31 INFO rasa.nlu.model - Starting to train component ResponseSelector Epochs: 100%|██████████████████████████████████| 100/100 [46:38<00:00, 27.98s/it, t_loss=6.65, r_acc=0.0464] 2021-12-30 12:36:48 INFO rasa.nlu.model - Finished training component.

For 500 QNa’s please see training output:

2021-12-30 13:05:17 INFO rasa.nlu.model - Starting to train component ResponseSelector Epochs: 100%|███████████████████████████████████| 100/100 [03:15<00:00, 1.95s/it, t_loss=22.4, r_acc=0.733] 2021-12-30 13:08:36 INFO rasa.nlu.model - Finished training component.

The accuracy increases when I reduce to 500 and also the responses are better.

Hope this helps. Let me know if you need any other information.

Workaround that I implemented - RASA accurately predicts it is a FAQ but is unable to predict the correct instance of the FAQ (like faq/1234 ). So I am using the NLU to get the intent FAQ and then instead of using RASA for uttering the response, I use a custom action and just send the entire Text entered by the user to Azure QnA service via their API. Azure QnA does a great job of returning the correct response and I just utter that response in the custom action.

While this works, I would really like to move everything to rasa from Azure. Any help in resolving this will be truly apprecited.