Bug when returning multiple FollowupAction events in action

Rasa Open Source
1

So, I’ve been developing a bot where a user can ask questions about items in their inventory, without going into any more detail. I have a flow where a user can ask a question getState{item_id}, which triggers “action_getState” and the bot replies with the state of the item with id item_id, and depending on the state, follows up with either “you can do X to item item_id, would you like to?” or “you can’t do X to item item_id”. I started out by setting a slot “can_do_X” and leading the model to the right actions based on that, and it works ok. Just when I ask repeated questions, the model sometimes fails on choosing that follow up action. I guess more stories + playing with model parameters such as history might help.

But, I looked into other, more deterministic approaches, and found the FollowupAction. So, trying that approach, in my “action_getState” I either return [FollowupAction("utter_can_do_X")] or return [FollowupAction("utter_cannot_do_X")] depending on the state. This kinda works, but now I sometimes (several exchanges in) get a double response, e.g. first “utter_can_do_X” with prob 1.00 (triggered by FollowupAction), and then “utter_cannot_do_X” with prob 0.39 by the KerasPolicy. This is my first concern, that the KerasPolicy actually tries to predict something that in the training data only ever was triggered as a FollowupAction… is this to be expected?

Assuming then that the KerasPolicy tries to predict the followup actions and can mess up there, I thought about enforcing another followup action, “action_listen”, by doing:

return [FollowupAction("utter_can_do_X"), FollowupAction("action_listen")]

Now, I can see both those events in the tracker after “action_getState”, but the utterance “utter_can_do_X” doesn’t appear. I’m guessing the returned “action_listen” prematurely forces the bot to stop uttering. Or am I doing it wrong? I guess I could change “utter_can_do_X” into an action, that itself has a FollowupAction(“action_listen”), but that sounds like a tedious workaround.

1 Like
2

I edited the title of the post to reflect the main issue.

TL;DR: It seems that returning two FollowupAction events, the first being an utterance and the second being “action_listen”, prevents the first FollowupAction from being properly executed and the utterance does not show up.

1 Like
3

Looks like this might be due to the way followup actions are implemented - tracker has a single followup_action attribute, which I guess gets overwritten by the second FollowupAction event in the return list.

Any ideas how one might go about specifying a sequence of followup actions without defining each of them as their own action with a single FollowupAction event returned? @akelad

1 Like
4

Hi @einarbmag, there’s actually a better way to achieve what you’re trying to do using the dispatcher. So you can do dispatcher.utter_template("utter_can_do_X", tracker) without returning any follow up actions. You shouldn’t need to do an action_listen either, as the template action won’t get logged in Rasa Cores tracker state

5

Right, but when you do dispatcher.utter_template("utter_can_do_X", tracker), it also doesn’t get logged in the tracker state (AFAICS), and therefore can’t affect the following predicted actions. So, I could get the unwanted following behaviour:

Dialogue 1 (correct):

  • get state of item X
    • state of item X is Y1
    • You can do Z to X, would you like to? (not logged in tracker)
  • yes
    • ok, did Z

Dialogue 2 (erronous):

  • get state of item X
    • state of item X is Y2
    • You can’t do Z to X (not logged in tracker)
  • yes
    • ok, did Z

Basically I need some state to steer the actions, whether it’s a slot or what the bot has uttered/actioned. What would you recommend - going back to the initial approach of using logic in the first action to set a slot value, and then predicting the following action/utterance depending on the slot value? Or refine the approach of using logic to enforce next action/utterance, and make sure that it’s logged in the tracker?

I am aware that maybe I was too quick to resort to these deterministic measures (FollowupAction etc.) instead of adding training stories.

1 Like
6

I think you can use FormAction , and play with requested_slots using logic.

8

I use a custom action function to decompose a multi-intent to its respective intents’ actions. However, enqueuing multiple FollowupActions in this custom action does not lead to execution of all FollowupActions. It seems that the first Followup action is properly acted upon, but other policies interrupt subsequently queued FollowupActions:

Here is an action for a multi-intent that is supposed to enqueue other actions:

class ActionMulti_VC_CT(Action):

    def name(self):
        return "action_VC_CT"

    def run(self, dispatcher, tracker, domain):

        intent = tracker.latest_message['intent'].get('name')
        list_of_actions = routeMultiIntent(intent)

        if len(list_of_actions) > 0:
            return [FollowupAction(x) for x in list_of_actions]
        else:
            return [FollowupAction("action_default_fallback")]

When it is triggered, the tracker correctly prints that two followup actions are registered for the particular multi-intent inform_partyVehics+inform_peopleCount :

2020-05-04 00:39:50 DEBUG rasa.core.processor - Action 'action_PV_PC' ended with events '[<rasa.core.events.FollowupAction object at 0x7f2c242c05d0>, <rasa.core.events.FollowupAction object at 0x7f2c24242c50>]'.

(these followup actions are action_inform_partyVehics and action_inform_peopleCount ):

However, as you can see, the bot only does the first FollowupAction:

Your input ->  3 adult and 2 kids. 2 cars                                                                                                                          
2020-05-04 00:39:50 DEBUG    rasa.core.tracker_store  - Recreating tracker for id 'c21e2c7a4d2640489471956fc08795e6'
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Received user message '3 adult and 2 kids. 2 cars' with intent '{'name': 'inform_partyVehics+inform_peopleCount', 'confidence': 0.5743247866630554}' and entities '[{'entity': 'number_people', 'start': 0, 'end': 1, 'extractor': 'DIETClassifier', 'value': '3'}, {'entity': 'number_people', 'start': 12, 'end': 13, 'extractor': 'DIETClassifier', 'value': '2'}, {'entity': 'number', 'start': 20, 'end': 21, 'extractor': 'DIETClassifier', 'value': '2'}, {'start': 0, 'end': 1, 'text': '3', 'value': 3, 'confidence': 1.0, 'additional_info': {'value': 3, 'type': 'value'}, 'entity': 'number', 'extractor': 'DucklingHTTPExtractor'}, {'start': 12, 'end': 13, 'text': '2', 'value': 2, 'confidence': 1.0, 'additional_info': {'value': 2, 'type': 'value'}, 'entity': 'number', 'extractor': 'DucklingHTTPExtractor'}, {'start': 20, 'end': 21, 'text': '2', 'value': 2, 'confidence': 1.0, 'additional_info': {'value': 2, 'type': 'value'}, 'entity': 'number', 'extractor': 'DucklingHTTPExtractor'}]'
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Current slot values: 
	age: None
	can_use_spacy: None
	countTime: 2020-05-04T12:00:00.000-07:00
	current_api: None
	data_stored: None
	entity_extractor: None
	feedback_message: None
	feedback_value: None
	howLong: None
	language: None
	name: None
	nlu_part: None
	onboarding: None
	package_manager: None
	partyVehics: None
	peopleCount: 3
	problem_description: None
	product: None
	requested_slot: partyVehics
	returnStatus: restart
	search_results: None
	seenBefore: False
	seenPastDay: None
	shown_privacy: None
	source: None
	step: None
	suggestion: None
	trailVisits: None
	unknown_nlu_part: None
	unknown_product: None
	vehicleCount: 7
	yearBorn: None
	zipCode: None
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 204 events.
2020-05-04 00:39:50 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'intent_hello': 1.0}, {'prev_action_setDefaultSlotValues': 1.0, 'intent_hello': 1.0}, {'prev_utter_hello': 1.0, 'intent_hello': 1.0}, {'prev_count_episode': 1.0, 'active_form_count_episode': 1.0, 'intent_hello': 1.0, 'slot_returnStatus_2': 1.0}, {'prev_action_listen': 1.0, 'active_form_count_episode': 1.0, 'entity_number_people': 1.0, 'intent_inform_partyVehics+inform_peopleCount': 1.0, 'entity_number': 1.0, 'slot_returnStatus_2': 1.0}]
2020-05-04 00:39:50 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-05-04 00:39:50 DEBUG    rasa.core.policies.mapping_policy  - The predicted intent 'inform_partyVehics+inform_peopleCount' is mapped to  action 'action_PV_PC' in the domain.
2020-05-04 00:39:50 DEBUG    rasa.core.policies.form_policy  - There is an active form 'count_episode'
2020-05-04 00:39:50 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-05-04 00:39:50 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_MappingPolicy
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Predicted next action 'action_PV_PC' with confidence 1.00.
2020-05-04 00:39:50 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'action_PV_PC'.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Action 'action_PV_PC' ended with events '[<rasa.core.events.FollowupAction object at 0x7f2c242c05d0>, <rasa.core.events.FollowupAction object at 0x7f2c24242c50>]'.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Current slot values: 
	age: None
	can_use_spacy: None
	countTime: 2020-05-04T12:00:00.000-07:00
	current_api: None
	data_stored: None
	entity_extractor: None
	feedback_message: None
	feedback_value: None
	howLong: None
	language: None
	name: None
	nlu_part: None
	onboarding: None
	package_manager: None
	partyVehics: None
	peopleCount: 3
	problem_description: None
	product: None
	requested_slot: partyVehics
	returnStatus: restart
	search_results: None
	seenBefore: False
	seenPastDay: None
	shown_privacy: None
	source: None
	step: None
	suggestion: None
	trailVisits: None
	unknown_nlu_part: None
	unknown_product: None
	vehicleCount: 7
	yearBorn: None
	zipCode: None
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Predicted next action 'action_inform_peopleCount' with confidence 1.00.
2020-05-04 00:39:50 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'action_inform_peopleCount'.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Action 'action_inform_peopleCount' ended with events '[<rasa.core.events.SlotSet object at 0x7f2c24018a50>]'.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Current slot values: 
	age: None
	can_use_spacy: None
	countTime: 2020-05-04T12:00:00.000-07:00
	current_api: None
	data_stored: None
	entity_extractor: None
	feedback_message: None
	feedback_value: None
	howLong: None
	language: None
	name: None
	nlu_part: None
	onboarding: None
	package_manager: None
	partyVehics: None
	peopleCount: 3
	problem_description: None
	product: None
	requested_slot: partyVehics
	returnStatus: restart
	search_results: None
	seenBefore: False
	seenPastDay: None
	shown_privacy: None
	source: None
	step: None
	suggestion: None
	trailVisits: None
	unknown_nlu_part: None
	unknown_product: None
	vehicleCount: 7
	yearBorn: None
	zipCode: None
2020-05-04 00:39:50 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, {}, {'prev_action_listen': 1.0, 'intent_hello': 1.0}, {'prev_action_setDefaultSlotValues': 1.0, 'intent_hello': 1.0}, {'prev_utter_hello': 1.0, 'intent_hello': 1.0}]
2020-05-04 00:39:50 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'count_episode'
2020-05-04 00:39:50 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'inform_partyVehics+inform_peopleCount'.
2020-05-04 00:39:50 DEBUG    rasa.core.policies.form_policy  - There is an active form 'count_episode'
2020-05-04 00:39:50 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-05-04 00:39:50 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_MemoizationPolicy
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Predicted next action 'count_episode' with confidence 1.00.
2020-05-04 00:39:50 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'count_episode'.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Action 'count_episode' ended with events '[BotUttered('How many vehicles did your party bring to the trailhead today?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {"countTime": "2020-05-04T12:00:00.000-07:00", "peopleCount": "3", "requested_slot": "partyVehics", "returnStatus": "restart", "vehicleCount": 7}, 1588552790.6476474), <rasa.core.events.SlotSet object at 0x7f2c24172290>]'.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Current slot values: 
	age: None
	can_use_spacy: None
	countTime: 2020-05-04T12:00:00.000-07:00
	current_api: None
	data_stored: None
	entity_extractor: None
	feedback_message: None
	feedback_value: None
	howLong: None
	language: None
	name: None
	nlu_part: None
	onboarding: None
	package_manager: None
	partyVehics: None
	peopleCount: 3
	problem_description: None
	product: None
	requested_slot: partyVehics
	returnStatus: restart
	search_results: None
	seenBefore: False
	seenPastDay: None
	shown_privacy: None
	source: None
	step: None
	suggestion: None
	trailVisits: None
	unknown_nlu_part: None
	unknown_product: None
	vehicleCount: 7
	yearBorn: None
	zipCode: None
2020-05-04 00:39:50 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, {}, {'prev_action_listen': 1.0, 'intent_hello': 1.0}, {'prev_action_setDefaultSlotValues': 1.0, 'intent_hello': 1.0}, {'prev_utter_hello': 1.0, 'intent_hello': 1.0}]
2020-05-04 00:39:50 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'count_episode'
2020-05-04 00:39:50 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'inform_partyVehics+inform_peopleCount'.
2020-05-04 00:39:50 DEBUG    rasa.core.policies.form_policy  - There is an active form 'count_episode'
2020-05-04 00:39:50 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-05-04 00:39:50 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_3_FormPolicy
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-05-04 00:39:50 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-05-04 00:39:50 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'c21e2c7a4d2640489471956fc08795e6'.

After the first action_inform_peopleCount is executed, the second followup action is nowhere to be seen, and the line 2020-05-04 00:39:50 DEBUG rasa.core.policies.memoization - There is a memorised next action 'count_episode' occurs.

How can you execute multiple enqueued followup actions?

This is the first “nested” action that is called as a FollowupAction:

class ActionStorePartyVehics(Action):
    """Stores the number of vehicles in a party in a slot"""

    def name(self):
        return "action_inform_partyVehics"

    def run(self, dispatcher, tracker, domain):

        intent = tracker.latest_message['intent'].get('name')
        partyVehics = next(tracker.get_latest_entity_values('number_vehic'), None)

        if not partyVehics:
            partyVehics = next(tracker.get_latest_entity_values('number'), None)


        if partyVehics:
            return [SlotSet('partyVehics', partyVehics)]
        elif intent in ["skip_please", "deny"]:
            return [SlotSet('partyVehics', "SKIPPED"), FollowupAction("action_skip_please")]
        else:
            return [SlotSet('partyVehics', tracker.latest_message.get('text'))]

Is the return statement in the action’s run() overwriting previously scheduled events? If so, how do I get around this and allow one multi-intent to call its respective component intents’ actions?

Thanks

9

@einarbmag Did you find any solution about it ?

10

@einar.bui