Something like that the response of intent classifier is wrong.
There are some training data of intent “ask weather”, e.g.
After training, If I type in “what is the weather now”, it returns the right intent “ask_weather”. But if I type in “Open the door”, “Turn on the light” casually, the response is still “ask_weather”. How can I fix this problem?
I just set up a demo and there are only three intents similar to asking weather.
The following picture is my config, I use the spacyNLP with chinese model. And my training data are also chinese. I just translate them to English convenient for posting the question.
It’s recommended to have at least 10 examples, while you only have 2. Train-test split usually removes 20% of your training data for testing, and since you have only 2 examples it basically takes 50% instead, so you are left with one example only.
And quality is better than quantity of course, examples should be as dissimilar as possible (that does not mean having similar examples is bad, it just doesn’t help much).
Take a look at this example. You don’t have entities so you don’t need as much data.
@ChrisRahme Thanks for your very nice and detailed response. The keypoint is that if the input are other casual words, its output is also ‘ask_weather’. Of course, it works well if the input is really about weather.
@ChrisRahme Yeah. I know that this is due to the amount of training data. Because I have just set up a demo, so there are only several intents. If adding an out-scope intent here, the training examples of this intent are innumerable, since the users can say unexpected anything in the real application.
Actually you don’t need to add all of out_of_scope examples, DIET with good regularization will try to learn common patterns as long as you use contextual encoders (T5, BERT, …). Training examples just should be diverse as much as you can provide, 50 examples is a good start point. You can access them from rasa demo project. I usually translate this phrases into target language (e.g. russian).
Sure! You can play with learning_rate, drop_rate and regularization_constant as described in DIET hyperparams section. You can use cross validation to fairly check pipeline performance. Also checkout this repo to automate hyperparams search.