Intermittent prediction of action_listen and action_default_fallback

here’s my policy config:

policies:

  • name: AugmentedMemoizationPolicy
    max_history: 2
  • name: KerasPolicy
    epochs: 200
    max_history: 3
    augmentation_factor: 500
    validation_split: 0.2
  • name: FallbackPolicy
    core_threshold: 0.6
    fallback_action_name: ‘action_default_fallback’

Question1 : I’m using AugmentedMemoizationPolicy, KerasPolicy and FallbackPolicy so I expect AugmentedMemoizationPolicy to always work first in conditions where the story step is exactly matching and is present in the training data, but sometimes it just predicts action_default_fallback even if that same input to the model exists in the training data, any idea why this would happen? and how to make sure AugmentedMemoizationPolicy works 100% of the time when the input to rasa core exactly matches a story step in Training Data of the model and only use KerasPolicy for the inputs which are not part of training data.

The second issue is: Dialogue model always predicted action_listen, so in order to get around that issue, i have added the following lines of code to always add action_listen to tracker explicitly after every new prediction of dialogue model so the tracker assumes that action_listen is added and thus doesn’t predict it.

tracker.update(ActionExecuted('action_listen', 'policy_2_MemoizationPolicy', 1.0))
processor.tracker_store.save(tracker)

(I did this because action_listen prediction is of no use in my use-case and the workaround worked fine for me), However, after I added more training data for the dialog model (around 500 new stories), it started predicting action_listen intermittently. (although very rare, and cannot reproduce every time on the same input)

Question 2 : is there a way to completely remove the action_listen action from dialogue model prediction? i don’t want the dialogue model to ever predict action_listen

any help is greatly appreciated, Thanks!

1 Like

hi @kavan , welcome to the forum! my best guess is that your stories are not what you think, especially with augmentation of 500 I can imagine some stories getting created that you maybe didn’t want. Have you tried turning it off? As a sanity check you can remove all policies except augmentedmemo and see how that performs

1 Like