Fallback action triggered unnecessarily (observed after migrating to Rasa 1.x)

Rasa Open Source
1

I am seeing a difference in the behavior of predicting next actions after migrating from Rasa 0.13.x to Rasa 1.x. In the earlier version, it did not fall back unnecesarily after the converstation was complete.

Below is an example: (It has no slots and just small talk conversations)

Conversation example with Rasa 0.13.2:

  1. who are you ? (user message)
  2. I am a bot (Bot response)
  3. How old are you? (user message)
  4. I am not that old (bot response) …

Same conversation example with Rasa 1.0.9:

  1. who are you ? (user message)
  2. I am a bot (Bot response)
  3. How old are you? (user message)
  4. Sorry, please say that again (bot response) - which is the fallback response triggered by action_default_fallback …

Please note that the first prediction is always accurate. From the second response on wards its pretty random. Earlier this was working fine in Rasa 0.13.x.

Below is the dialog policy configuration:

policies:
- augmentation_factor: 0
  batch_size: 10
  epochs: 30
  featurizer:
  - max_history: 5
    name: MaxHistoryTrackerFeaturizer
    state_featurizer:
    - name: BinarySingleStateFeaturizer
  name: KerasPolicy
  validation_split: 0.2
- max_history: 6
  name: MemoizationPolicy
- core_threshold: 0.3
  fallback_action_name: action_default_fallback
  name: FallbackPolicy
  nlu_threshold: 0.3
- name: FormPolicy

Below is the story file:

## Generated Story for age
* agent_age
   - utter_agent_age
   
## Generated Story for information 
* agent_info
   - utter_agent_info

The other observation is that if I increase the epoch to 350+ than it starts predicting the next action more accurately. Hence not sure what is the actual reason behind the change in behavior. May be is it due to the upgraded versions of Keras Policy?

Any insight on the changed behavior or any other suggestions is appreciated.

2

If you want to go from the end of one story to another, then you probably want to use the AugmentedMemoizationPolicy instead of the MemoizationPolicy. Does that help?

2 Likes
3

Thanks for your response. I have a followup question in a scenario where we can have a mix of stories with and without slots. Based on my understanding, AugmentedMemoizationPolicy will still inherit the capability of “MemoizationPolicy” to consider slots for predictions in case it is being set. Additionally, it can also disregard slots to consider a decision if slots are not set. I hope my understanding is correct.

4

Hm, it depends. If slots being set or not affects your dialogue flow, you should have separate stories for these cases. If it doesn’t, I would just change them to the unfeaturized type so that the dialogue management doesn’t take that information into account. For any given slot, the model will either take it into account or not, it can’t do both.

5

Thanks once again for your response. Yes, we do have separate stories for small talk conversations (which would have no slots) as well as separate stories with slots used for getting information from the user.

6

Ah, I see, yes I think that should be fine, as you’re not expecting it to do the same story with and without slots.

7

Thank you for the help ! The basic sanity tests went fine.