Fallback policy nlu_threshold met when it should not

Rasa Open Source
1

Hi I have set the nlu_threshold at 0.8, the ambiguity_threshold at 0.55 and the core_threshold at 0.8 but sometimes even if the intent has confidence higher than 0.8 the action of my FallbackPolicy is called. I have checked my stories, my intents and my responses many times and there is nothing wrong with them. It worked fine just a few days ago and I haven’t change anything in my code

policies:

  • name: AugmentedMemoizationPolicy
  • name: TEDPolicy max_history: 5 epochs: 200
  • name: MappingPolicy
  • name: “FallbackPolicy” nlu_threshold: 0.8 ambiguity_threshold: 0.55 core_threshold: 0.8 fallback_action_name: action_default_ask_affirmation
2

did you take a look at your debug logs, what triggered fallback?

3

I changed core_threshold to 0.3

Received user message ‘SMS’ with intent ‘{‘name’: ‘multiSMS’, ‘confidence’: 0.9904831051826477}’ There is no memorised next action NLU confidence threshold met, confidence of fallback action set to core threshold (0.3). Predicted next action using policy_3_FallbackPolicy Predicted next action ‘action_default_ask_affirmation’ with confidence 0.30. Calling action endpoint to run action ‘action_default_ask_affirmation’

4

when i had core_threshold = 0.8

Received user message ‘SMS’ with intent ‘{‘name’: ‘multiSMS’, ‘confidence’: 0.9904831051826477}’ There is no memorised next action NLU confidence threshold met, confidence of fallback action set to core threshold (0.8). Predicted next action using policy_3_FallbackPolicy Predicted next action ‘action_default_ask_affirmation’ with confidence 0.80. Calling action endpoint to run action ‘action_default_ask_affirmation’

5

it looks like you don’t have trained TED policy in the running model

6

I have TED policy in my config.yml. So could you tell me what I am doing wrong and results in TED policy not to be trained?

7

I don’t know, just looking on your logs, it seemed like TED wasn’t present

8

Hi New to Rasa, I am getting the same issue, The first time even though the intent is threshold confidence is met. it triggers the fallbackpolicy and next minute it works fine by predicting TED policy. The version we are using is still 1.10.2. I have also attached the config as well.

2020-11-20 12:27:20 DEBUG rasa.core.policies.memoization - Current tracker state [{‘prev_action_address_confirmation’: 1.0, ‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_listen’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0, ‘prev_change_address’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_change_address’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_address_confirmation’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}] 2020-11-20 12:27:20 DEBUG rasa.core.policies.memoization - There is no memorised next action 2020-11-20 12:27:20 DEBUG rasa.core.policies.mapping_policy - There is no mapped action for the predicted intent, ‘affirm’. 2020-11-20 12:27:20 DEBUG rasa.core.policies.form_policy - There is no active form 2020-11-20 12:27:20 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7). 2020-11-20 12:27:20 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_1_TEDPolicy 2020-11-20 12:27:20 DEBUG rasa.core.processor - Predicted next action ‘action_listen’ with confidence 1.00. 2020-11-20 12:27:20 DEBUG rasa.core.processor - Action ‘action_listen’ ended with events ‘[]’. 2020-11-20 12:27:20 DEBUG rasa.core.lock_store - Deleted lock for conversation ‘xxxxxx xxxxx’. 2020-11-20 12:27:32 DEBUG rasa.core.lock_store - Issuing ticket for conversation ‘21915 5003’. 2020-11-20 12:27:32 DEBUG rasa.core.lock_store - Acquiring lock for conversation ‘21915 5003’. 2020-11-20 12:27:32 DEBUG rasa.core.lock_store - Acquired lock for conversation ‘21915 5003’. 2020-11-20 12:27:32 DEBUG rasa.core.tracker_store - Recreating tracker for id ‘21915 5003’ 2020-11-20 12:27:32 DEBUG rasa.nlu.classifiers.diet_classifier - There is no trained model for ‘ResponseSelector’: The component is either not trained or didn’t receive enough training data. 2020-11-20 12:27:32 DEBUG rasa.nlu.selectors.response_selector - Adding following selector key to message property: default 2020-11-20 12:27:32 DEBUG rasa.core.processor - Received user message ‘yes’ with intent ‘{‘name’: ‘affirm’, ‘confidence’: 0.9999963641166687}’ and entities ‘[]’ 2020-11-20 12:27:32 DEBUG rasa.core.processor - Logged UserUtterance - tracker now has 610 events. 2020-11-20 12:27:32 DEBUG rasa.core.policies.memoization - Current tracker state [{‘intent_deny’: 1.0, ‘prev_action_listen’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0, ‘prev_change_address’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_change_address’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_address_confirmation’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘prev_action_listen’: 1.0, ‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}] 2020-11-20 12:27:32 DEBUG rasa.core.policies.memoization - There is no memorised next action 2020-11-20 12:27:32 DEBUG rasa.core.policies.form_policy - There is no active form 2020-11-20 12:27:32 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7). 2020-11-20 12:27:32 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_4_FallbackPolicy 2020-11-20 12:27:32 DEBUG rasa.core.processor - Predicted next action ‘action_fallback’ with confidence 0.70. 2020-11-20 12:27:32 DEBUG rasa.core.actions.action - Calling action endpoint to run action ‘action_fallback’. 2020-11-20 12:27:32 DEBUG rasa.core.processor - Action ‘action_fallback’ ended with events ‘[BotUttered(‘I beg your pardon ! Could you please repeat it again ?’, {“elements”: null, “quick_replies”: null, “buttons”: null, “attachment”: null, “image”: null, “custom”: null}, {}, 1605875252.4759085), <rasa.core.events.SlotSet object at 0x7ff186d95cc0>]’. 2020-11-20 12:27:32 DEBUG rasa.core.processor - Current slot values:

Second time it uses TED Policy

2020-11-20 12:27:32 DEBUG rasa.core.policies.memoization - Current tracker state [{‘intent_deny’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0, ‘prev_change_address’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_change_address’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_address_confirmation’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘prev_action_listen’: 1.0, ‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0, ‘prev_action_fallback’: 1.0}] 2020-11-20 12:27:32 DEBUG rasa.core.policies.memoization - There is no memorised next action 2020-11-20 12:27:32 DEBUG rasa.core.policies.mapping_policy - There is no mapped action for the predicted intent, ‘affirm’. 2020-11-20 12:27:32 DEBUG rasa.core.policies.form_policy - There is no active form 2020-11-20 12:27:32 DEBUG rasa.core.policies.fallback - Predicted ‘action_listen’ after fallback action ‘action_fallback’ 2020-11-20 12:27:32 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_4_FallbackPolicy 2020-11-20 12:27:32 DEBUG rasa.core.processor - Predicted next action ‘action_listen’ with confidence 1.00. 2020-11-20 12:27:32 DEBUG rasa.core.processor - Action ‘action_listen’ ended with events ‘[]’. 2020-11-20 12:27:32 DEBUG rasa.core.lock_store - Deleted lock for conversation ‘07960021915 5003’. 2020-11-20 12:27:38 DEBUG rasa.core.lock_store - Issuing ticket for conversation ‘07960021915 5003’. 2020-11-20 12:27:38 DEBUG rasa.core.lock_store - Acquiring lock for conversation ‘07960021915 5003’. 2020-11-20 12:27:38 DEBUG rasa.core.lock_store - Acquired lock for conversation ‘07960021915 5003’. 2020-11-20 12:27:38 DEBUG rasa.core.tracker_store - Recreating tracker for id ‘07960021915 5003’ 2020-11-20 12:27:38 DEBUG rasa.nlu.classifiers.diet_classifier - There is no trained model for ‘ResponseSelector’: The component is either not trained or didn’t receive enough training data. 2020-11-20 12:27:38 DEBUG rasa.nlu.selectors.response_selector - Adding following selector key to message property: default 2020-11-20 12:27:38 DEBUG rasa.core.processor - Received user message ‘yes’ with intent ‘{‘name’: ‘affirm’, ‘confidence’: 0.9999963641166687}’ and entities ‘[]’ 2020-11-20 12:27:38 DEBUG rasa.core.processor - Logged UserUtterance - tracker now has 615 events. 2020-11-20 12:27:38 DEBUG rasa.core.policies.memoization - Current tracker state [{‘intent_deny’: 1.0, ‘prev_action_change_address’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘intent_deny’: 1.0, ‘prev_action_address_confirmation’: 1.0, ‘slot_registered_0’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘prev_action_listen’: 1.0, ‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}, {‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0, ‘prev_action_fallback’: 1.0}, {‘prev_action_listen’: 1.0, ‘slot_registered_0’: 1.0, ‘intent_affirm’: 1.0, ‘slot_Order_Type_0’: 1.0, ‘slot_Item_available_0’: 1.0}] 2020-11-20 12:27:38 DEBUG rasa.core.policies.memoization - There is no memorised next action 2020-11-20 12:27:38 DEBUG rasa.core.policies.form_policy - There is no active form 2020-11-20 12:27:38 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7). 2020-11-20 12:27:38 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_1_TEDPolicy 2020-11-20 12:27:38 DEBUG rasa.core.processor - Predicted next action ‘action_save_order’ with confidence 0.78. 2020-11-20 12:27:38 DEBUG rasa.core.actions.action - Calling action endpoint to run action ‘action_save_order’. 2020-11-20 12:27:40 DEBUG rasa.core.processor - Action ‘action_save_order’ ended with events ‘[BotUttered(‘The total bill amount is 23 pounds and 95 pence.How would you like to pay ? Cash or Card ?’, {“elements”: null, “quick_replies”: null, “buttons”: null, “attachment”: null, “image”: null, “custom”: null}, {“order_id”: “ivr121”, “pounds”: “23”, “pence”: “95”}, 1605875260.1129186), <rasa.core.events.SlotSet object at 0x7ff1a4cafeb8>, <rasa.core.events.SlotSet object at 0x7ff1a4caf6a0>, <rasa.core.events.SlotSet object at 0x7ff1d9d6b0f0>]’. 2020-11-20 12:27:40 DEBUG rasa.core.processor - Current slot values:

Config

policies:

  • name: MemoizationPolicy
  • name: TEDPolicy max_history: 10 epochs: 100
  • name: MappingPolicy
  • name: FormPolicy
  • name: FallbackPolicy nlu_threshold: 0.7 ambiguity_threshold: 0.1 core_threshold: 0.7 fallback_action_name: action_fallback
9

It seems like in the first case, TEDPolicy predicted next action with confidence less than 0.7

10

Actually, I have the same issue. The nlu intent confidence is high, but I don’t know why the confidence predicted by TEDPolicy is almost 0 (I tried to set core_threshold: 0.01), rasa.core.policies.ensemble Predicted next action wrongly. I want to disable the core_threshold checking, how could I do it?

Part of my log is as below Received user message ‘what is visual field’ with intent ‘{‘id’: 5760442712840671865, ‘name’: ‘find_condition_definitions’, ‘confidence’: 0.9999246597290039}’ and entities ‘[{‘entity’: ‘symptom_type’, ‘start’: 8, ‘end’: 20, ‘confidence_entity’: 0.9837067127227783, ‘value’: ‘visualfiled-synm’, ‘extractor’: ‘DIETClassifier’, ‘processors’: [‘EntitySynonymMapper’]}]’ … [state 0] user intent: find_condition_definitions | user entities: (‘disease_type’,) | previous action name: action_default_fallback [state 1] user intent: find_condition_definitions | user entities: (‘disease_type’,) | previous action name: action_listen [state 2] user intent: find_condition_definitions | user entities: (‘disease_type’,) | previous action name: action_default_fallback [state 3] user intent: find_condition_definitions | user entities: (‘symptom_type’,) | previous action name: action_listen [state 4] user intent: find_condition_definitions | user entities: (‘symptom_type’,) | previous action name: action_default_fallback 2021-01-08 16:10:46 DEBUG rasa.core.policies.memoization - There is no memorised next action 2021-01-08 16:10:46 DEBUG rasa.core.policies.mapping_policy - There is no mapped action for the predicted intent, ‘find_condition_definitions’. 2021-01-08 16:10:46 DEBUG rasa.core.policies.form_policy - There is no active form 2021-01-08 16:10:46 DEBUG rasa.core.policies.fallback - Predicted ‘action_listen’ after fallback action ‘action_default_fallback’ 2021-01-08 16:10:46 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_4_FallbackPolicy. 2021-01-08 16:10:46 DEBUG rasa.core.processor - Predicted next action ‘action_listen’ with confidence 1.00.