Action_default_fallback often predicted as next action after correct action was predicted

Rasa Open Source
1

@souvikg10: Following up with the conversation we had here

I’ve recently found myself in a situation where my bot started to think that after executing the correct action, instead of following up with action_listen, he prefers to execute action_default_fallback. Any idea why this might be the case?

So e.g. a story could be like this:

  • greeting
    • utter_greeting

But then in reality the conversation is sometimes like this:

  • greeting
    • utter_greeting
    • action_default_fallback

It’s also only happening with some intents, not in general. I also didn’t experience it initially, but for the last week or two, it happened more frequently. Not sure what I changed or caused it though …

Does anyone have any ideas or had similar issues in the past?

2 Likes
2

Do you have the debug logs, normally it gives good indication to why this could be happening?

3

Let me check that. Was running in interactive mode which doesn’t output logs (at least for me right now), right!? Will get back to you with the logs.

4

@souvikg10 ran into this situation again right now, which should be this very obvious story:

  • smalltalk.greetings.hello
    • utter_smalltalk.greetings.hello

But again, after the utter action, the bot also executed the default_fallback_action. Here’s the log:

rasa-core_1      | 2018-11-18 19:03:00 DEBUG    rasa_core.processor  - Received user message 'Hi' with intent '{u'confidence': 0.9146153926849365, u'name': u'smalltalk.greetings.hello'}' and entities '[]'
rasa-core_1      | 2018-11-18 19:03:00 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 2 events
rasa-core_1      | 2018-11-18 19:03:00 DEBUG    rasa_core.processor  - Current slot values: 
...
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.memoization  - Current tracker state [None, None, None, {u'slot_auth.authenticated_0': 1.0}, {u'intent_smalltalk.greetings.hello': 1.0, u'prev_action_listen': 1.0, u'slot_auth.authenticated_0': 1.0}]
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.memoization  - There is a memorised next action '38'
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.form_policy  - There is no active form
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_1_MemoizationPolicy
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.processor  - Predicted next action 'utter_smalltalk.greetings.hello' with prob 1.00.
rasa-core_1      | 2018-11-18 19:03:01 INFO     rasa_core.channels.facebook  - Sending message: Hallo!
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.processor  - Action 'utter_smalltalk.greetings.hello' ended with events '[]'
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.processor  - Bot utterance 'BotUttered(text: Hallo!, data: {
rasa-core_1      |   "buttons": null, 
rasa-core_1      |   "elements": null, 
rasa-core_1      |   "attachment": null
rasa-core_1      | })'
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.memoization  - Current tracker state [None, None, {u'slot_auth.authenticated_0': 1.0}, {u'intent_smalltalk.greetings.hello': 1.0, u'prev_action_listen': 1.0, u'slot_auth.authenticated_0': 1.0}, {u'intent_smalltalk.greetings.hello': 1.0, u'prev_utter_smalltalk.greetings.hello': 1.0, u'slot_auth.authenticated_0': 1.0}]
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.form_policy  - There is no active form
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_2_FallbackPolicy
rasa-core_1      | 2018-11-18 19:03:01 DEBUG    rasa_core.processor  - Predicted next action 'action_default_fallback' with prob 1.00.

My policies.yml looks like this:

policies:
  - name: "KerasPolicy"
    featurizer:
    - name: MaxHistoryTrackerFeaturizer
      max_history: 5
      state_featurizer:
      - name: BinarySingleStateFeaturizer
    epochs: 50
  - name: "MemoizationPolicy"
    max_history: 5
  - name: "FallbackPolicy"
    nlu_threshold: 0.7
    core_threshold: 0.7
    fallback_action_name: "action_default_fallback"
  - name: "FormPolicy"

What I just realized is that the KerasPolicy doesn’t show up anymore (in the log). Is that normal? I think since I added the FormPolicy, I don’t get the Keras predictions for the ensemble. Does that make any sense?

1 Like
5

I have the same problem

6

It is weird that the Keras policy did not trigger. Possibly a bug

7

Yeah … just now I tried to run it without the form policy, but still no Keras policy popping up in the log. That’s what seems to cause the fallback policy to kick in every time there’s no memorized next action.

@akelad: Any idea what’s wrong with this setup?

UPDATE: KerasPolicy does show up during training though (with the same policies.yml)

UPDATE: Seems that it’s still active, just not logging anything except when its next action is taken:

rasa-core_1      | 2018-11-19 10:37:27 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
rasa-core_1      | 2018-11-19 10:37:27 DEBUG    rasa_core.policies.form_policy  - There is no active form
rasa-core_1      | 2018-11-19 10:37:27 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_0_KerasPolicy

I still don’t understand though why the default_fallback_action is executed every now and then …

8

I have the same problem with the right utter fired and the fallback for some intent Moreover, when I train the core I have different values Once

chat1$>rm models/dialogue -Rf chat1$>python3 -m rasa_core.train -d domain.yml -o models/dialogue/ -vv -s data/stories -c config.yaml 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Generated trackers will be deduplicated based on their unique last 2 states. 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Number of augmentation rounds is 3 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Starting data generation round 0 … (with 1 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:00<00:00, 788.89it/s, # trackers=1$ 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Finished phase (205 training samples found). 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Data generation rounds finished. 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Found 0 unused checkpoints 2018-11-19 08:25:55 DEBUG rasa_core.training.generator - Starting augmentation round 0 … (with 50 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:12<00:00, 16.50it/s, # trackers=50$ 2018-11-19 08:26:08 DEBUG rasa_core.training.generator - Finished phase (10455 training samples found). 2018-11-19 08:26:08 DEBUG rasa_core.training.generator - Starting augmentation round 1 … (with 50 trackers) Processed Story Blocks: 15%|█████████▌ | 31/213 [00:01<00:10, 17.36it/s, # trackers=47$ /NOBACKUP/centos/Python-3.5.2/lib/python3.5/site-packages/pykwalify/core.py:99: UnsafeLoaderWarning: The default ‘Loader’ for ‘load(stream)’ without further arguments can be unsafe. Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:12<00:00, 16.88it/s, # trackers=47$ 2018-11-19 08:26:21 DEBUG rasa_core.training.generator - Finished phase (20093 training samples found). 2018-11-19 08:26:21 DEBUG rasa_core.training.generator - Starting augmentation round 2 … (with 50 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:13<00:00, 15.48it/s, # trackers=50$ 2018-11-19 08:26:35 DEBUG rasa_core.training.generator - Finished phase (30343 training samples found). 2018-11-19 08:26:35 DEBUG rasa_core.training.generator - Found 30343 training trackers. 2018-11-19 08:26:35 DEBUG rasa_core.agent - Agent trainer got kwargs: {} 2018-11-19 08:26:35 DEBUG rasa_core.featurizers - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturi$er(BinarySingleStateFeaturizer))… Processed trackers: 100%|█████████████████████████████████████████████████████████████| 30343/30343 [00:25<00:00, 1173.30it/s, # actions=37714$2018-11-19 08:27:01 DEBUG rasa_core.featurizers - Created 37714 action examples. _________________________________________________________________ Layer (type) Output Shape Param #
================================================================= masking (Masking) (None, 2, 452) 0
_________________________________________________________________ lstm (LSTM) (None, 32) 62080
_________________________________________________________________ dense (Dense) (None, 209) 6897
_________________________________________________________________ activation (Activation) (None, 209) 0
================================================================= Total params: 68,977 Trainable params: 68,977 Non-trainable params: 0 _________________________________________________________________ 2018-11-19 08:27:03 DEBUG rasa_core.policies.keras_policy - None 2018-11-19 08:27:03 INFO rasa_core.policies.keras_policy - Fitting model with 37714 total samples and a validation split of 0.1 2018-11-19 08:27:03 DEBUG rasa_core.policies.policy - Parameters ignored by model.fit(...): {} Epoch 1/200 37714/37714 [==============================] - 1s 39us/step - loss: 5.2777 - acc: 0.1274

Repeated

chat1$>rm models/dialogue -Rf
chat1$>python3 -m rasa_core.train -d domain.yml -o models/dialogue/ -vv -s data/stories -c config.yaml
2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Generated trackers will be deduplicated based on their unique last 2 states. 2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Number of augmentation rounds is 3 2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Starting data generation round 0 … (with 1 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:00<00:00, 745.54it/s, # trackers=3] 2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Finished phase (205 training samples found). 2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Data generation rounds finished. 2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Found 0 unused checkpoints 2018-11-19 08:36:19 DEBUG rasa_core.training.generator - Starting augmentation round 0 … (with 50 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:13<00:00, 16.10it/s, # trackers=42] 2018-11-19 08:36:32 DEBUG rasa_core.training.generator - Finished phase (10047 training samples found). 2018-11-19 08:36:32 DEBUG rasa_core.training.generator - Starting augmentation round 1 … (with 50 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:13<00:00, 16.29it/s, # trackers=51] 2018-11-19 08:36:45 DEBUG rasa_core.training.generator - Finished phase (19889 training samples found). 2018-11-19 08:36:45 DEBUG rasa_core.training.generator - Starting augmentation round 2 … (with 50 trackers) Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████| 213/213 [00:13<00:00, 15.25it/s, # trackers=54] 2018-11-19 08:36:59 DEBUG rasa_core.training.generator - Finished phase (30139 training samples found). 2018-11-19 08:36:59 DEBUG rasa_core.training.generator - Found 30139 training trackers. 2018-11-19 08:36:59 DEBUG rasa_core.agent - Agent trainer got kwargs: {} 2018-11-19 08:36:59 DEBUG rasa_core.featurizers - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer))… Processed trackers: 100%|█████████████████████████████████████████████████████████████| 30139/30139 [00:25<00:00, 1172.40it/s, # actions=42863]2018-11-19 08:37:25 DEBUG rasa_core.featurizers - Created 42863 action examples. _________________________________________________________________ Layer (type) Output Shape Param #
================================================================= masking (Masking) (None, 2, 452) 0
_________________________________________________________________ lstm (LSTM) (None, 32) 62080
_________________________________________________________________ dense (Dense) (None, 209) 6897
_________________________________________________________________ activation (Activation) (None, 209) 0
================================================================= Total params: 68,977 Trainable params: 68,977 Non-trainable params: 0 _________________________________________________________________ 2018-11-19 08:37:27 DEBUG rasa_core.policies.keras_policy - None 2018-11-19 08:37:27 INFO rasa_core.policies.keras_policy - Fitting model with 42863 total samples and a validation split of 0.1 2018-11-19 08:37:27 DEBUG rasa_core.policies.policy - Parameters ignored by model.fit(...): {} Epoch 1/200 42863/42863 [==============================] - 2s 36us/step - loss: 5.2474 - acc: 0.2268

I do not know if it is normal, I did not change anything between training

9

@smn-snkl - i see you have the authentication slot, is it featurized? perhaps you should add them in the stories

10

@smn-snkl in what situations does the Fallback happen? could you post some logs?

11

@akelad: I posted this very simple flow and the logs above

@souvikg10: That could actually be it. Should I rather make it unfeaturized? Since authentication shouldn’t be a rule that core learns, but should be business logic that is checked in the actions, right?

12

if they don’t help in the conversation, you should unfeaturize it.

1 Like
13

I am facing the same problem i am using memoization and fallback policy , after returning correct action, it respond with fallback_action as well

2 Likes
14

I am having the same problem too any solution for this

2 Likes
15

im also facing same problem . any solution for this

1 Like
17

Any news on this? I’m experiencing the same problem. Setting action_listen as a hard trigger for the respective intent in the domain file also didn’t help.

1 Like
18

Hi, has this been resolved? I’ve got the same problem.

1 Like
19

Hi, has this been resolved? I’ve also got the same problem.

1 Like
20

YOu can try changing the priority of policies, in which the form policy should be the first one

21

Hey @souvikg10 many of us seem to be having the same problem. Could you please help!!!