Anyone have experience with cutting down on fails of repeated intents confusing Rasa Core a lot?
For example, I have an intent, /find_services_gbv, that in my stories is only ever followed by two actions, an utter_confirm and then an utter_menu. We have half a dozen variants, in all of them, only ever this sequence, the intent never leads to anything else.
Then we run the model, and for some reason if the user repeats this phrase, by the third repetition, Core starts doing all sorts of strange things, like setting slots it shouldn’t be setting (often with spurious values), or uttering from wholly different stories.
Memorization is set at 8 or 10, augmentation at 20, on the train cross-val accuracy is 99%.
Any experience for how to basically enforce that if an intent is only ever followed by a certain action, that’s all that ever happens, not strange guessing / confusion.
You need to change memorisation or I suppose it is the max_history parameter you are using to 0 if you want that history don’t influence the conversation at all
Meaning given an intent X the answer is alway Y
Also set augmentation to 0 so it won’t randomly glue stories together to create longer stories
Yeah it was overfitting, actually intersecting with that augmentation point. So I dialed down augmentation (not all the way to 0, but down to 10) and it’s much better now.
The subtlety here that I didn’t realize is that cross-val set is generated from the aug process too, so if the overfitting is coming from the aug process then it won’t be picking up by losses on the validation run. I.e., learned to be careful with that aug factor and keep a close eye out for overfit when aug is over 0.