I am using it only for a single intent (chitchat) but I have many sentence pairs (> 4000).
I was looking in the documentation that it is similar to DIETClassifier, but I could’t figure what it does, and whether the default values of the network ( hidden_layers_sizes, embedding_dimension, etc.) are enough for my case…
How is the performance of the DIETClassifier? It might be that the language model you are using is not the best. Did you tried any other language model or even training without any pre-trained language model? Also it is quite hard to say what is going wrong without looking at the data. Could you maybe share an excerpt of your data?
## intent: chitchat/greetings0en02
- Good Morning
- Good Morning.
My responses.md has:
## greetings0en02
* chitchat/greetings0en02
- Good day.
- Good Morning.
I am using bert multilingual embeddings (“bert-base-multilingual-cased”).
In rasa shell, I type “good morning”.
2020-09-08 12:26:57 DEBUG rasa.core.policies.mapping_policy - The predicted intent 'chitchat' is mapped to action 'respond_chitchat' in the domain.
2020-09-08 12:26:57 DEBUG rasa.core.policies.form_policy - There is no active form
2020-09-08 12:26:57 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-09-08 12:26:57 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_2_MappingPolicy
2020-09-08 12:26:57 DEBUG rasa.core.processor - Predicted next action 'respond_chitchat' with confidence 1.00.
2020-09-08 12:26:57 DEBUG rasa.core.actions.action - Picking response from selector of type default
2020-09-08 12:26:57 DEBUG rasa.core.processor - Action 'respond_chitchat' ended with events '[BotUttered('Thank you!', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1599557217.3225248)]'
The intent is correct, “chitchat”. I am not sure if the “sub-intent” is correct.
But the response selector returns “thank you”.
Why isn’t selected one of “Good day.” or “Good Morning.”?
What follows ‘/’ in intent name is ignored?
This basically excludes features from RegexFeaturizer, LexicalSyntacticFeaturizer and char level CountVectorsFeaturizer from being used for the response selector.
What I did is simple: I map input into an intent (such as chitchat/action1), and then I select a response from all available responses in the responses.md file.
This works better than the rasa’s response selector (much fewer errors). although it still has some mis-classifications. And it solves the problem of not responding all responses.
I think in my case, where chitchat is huge (~23.000 questions) and almost as many answers, embedding both answers & questions in the same space and selecting through similarities, is not a good strategy.
@petasis Thanks for that feedback. We have actually made that possible now with the latest upcoming release of Rasa Open source 2.0. Expect it to be out sometime this week or early next week.
rasa --version
Rasa Version : 2.0.0rc3
Rasa SDK Version : 2.0.0rc1
Rasa X Version : None
Python Version : 3.8.5 (default, Aug 12 2020, 00:00:00)
Operating System : Linux-5.8.12-200.fc32.x86_64-x86_64-with-glibc2.2.5
Python Path : /usr/bin/python3
intent: flight_departure_info, training examples: 5953
intent: flight_arrival_info, training examples: 203
intent: inform, training examples: 2429
intent: affirm, training examples: 32
intent: deny, training examples: 19
intent: stop, training examples: 58
intent: search_encyclopedia, training examples: 32
intent: search_weather, training examples: 33
intent: chitchat_el, training examples: 21488
intent: insult, training examples: 236
intent: thank_you, training examples: 102
intent: chitchat_en, training examples: 1713
@dakshvar22 Is there a document describing how response selector works in rasa 2.0rc4?
Because the results are very low, and at the same time a simple DIET classifier (that simply maps input to a class like chitchat/ask) is ~0.98.
In reality it is much lower (i.e. when asking all questions, the final accuracy is 0.8), but still rasa’s response selector seems to not be able to handle the data. Perhaps a problem in the training format data?