Button message not displayed during FormAction

Rasa Open Source
1

Hi, i try to validate a slot for which multiple entity inputs where found in a single message. In that case the variable ‘value’ in validate_{slot_to_validate} is a list of the extracted entities.

My idea was to let the user choose which entity he wants to map to the slot via a button message. Therefore I wrote this functions within my FormAction which gets called in the validate_{slot} function

 # result_list here contains the possible slot values
 def _ambiguous_result(self,dispatcher,tracker: Tracker,domain,result_list):

    logger.debug('Handling ambiguous results.')

    buttons = list()
    for result in result_list:
        payload = json.dumps({tracker.get_slot('requested_slot'): result})
        buttons.append({'title': result,
                        'payload': '/answer ' + payload,
                        'type': 'button-bot',
                        'style': 'inlineButton'})

    payload = json.dumps({tracker.get_slot('requested_slot'): None})

    buttons.append({'title': domain['templates'].get('ambigous_none_button'),
                    'payload': '/answer ' + payload,
                    'type': 'button-bot',
                    'style': 'inlineButton'})

    logger.debug(f'Number of buttons = {len(buttons)}')

    dispatcher.utter_message("This output gets uttered!")
    dispatcher.utter_button_message('This does not',buttons)

    return

While the pure text messages is uttered by the bot the button message is completely lost. However its mentioned in the debug output

DEBUG    rasa.core.processor  - Action 'move_form' ended with events '['BotUttered(text: This output gets uttered!, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, metadata: {})', 'BotUttered(text: This does not, data: {"elements": null, "quick_replies": null, "buttons": [{"payload": "/answer {\\"taxid\\": \\"1011121314152\\"}", "style": "inlineButton", "title": "1011121314152", "type": "button-bot"}, {"payload": "/answer {\\"taxid\\": \\"1011121314153\\"}", "style": "inlineButton", "title": "1011121314153", "type": "button-bot"}, {"payload": "{\\"taxid\\": null}", "style": "inlineButton", "title": [{"text": "Keine"}], "type": "button-bot"}], "attachment": null, "image": null, "custom": null}, metadata: {})', 'BotUttered(text: Wie lautet Ihre Steuernummer?, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, metadata: {"move_date": null, "requested_slot": "taxid", "residence": null, "taxid": null})', 'SlotSet(key: taxid, value: None)', 'SlotSet(key: requested_slot, value: taxid)']'

So, why is the button message not visible at all? It works in general (for example in the TwoStageFallbackPolicy).

What are the alternatives to handle two valid slot values? I might just throw away everything and ask the user for the same slot again. Any other ideas?

EDIT: Test above were done in the rasa shell on command line. Tried the Rasa X UI instead. There the message text of the button message gets displayed, but no buttons are displayed and it jumps right to the next action.

Also giving the validate action for completeness:

def validate_taxid(self, value,
                       dispatcher: CollectingDispatcher,
                       tracker: Tracker,
                       domain: Dict[Text, Any]) -> Dict[Text, Any]:

    logger.debug(f"Tracker latest message: {tracker.latest_message}")

    taxid_regex = re.compile(r'\d{13}')

    candidate_list = []

    # multiple entities end up here as a list
    if type(value) == type([]):
        logger.debug(f"Found multiple entities of taxid in message {tracker.latest_message.get('text')}")
        for taxid in value:
            if taxid_regex.match(taxid):
                candidate_list.append(taxid)
    # no entities found, trying to extract them from text
    elif tracker.latest_message.get('entities') == []:
        candidate_list = taxid_regex.findall(value)
    else:
        if taxid_regex.match(value):
            candidate_list = [value]

    logger.debug(f'Candidate list in taxid {candidate_list}')

    if len(candidate_list) == 0:
        dispatcher.utter_template('utter_not_understood',tracker)
        dispatcher.utter_template('utter_advice_taxid',tracker)
        return {'taxid' : None}
    elif len(candidate_list) > 1:
        self._ambiguous_result(dispatcher, tracker, domain, candidate_list)
        return {'taxid' : None}
    else:
        return{'taxid': candidate_list[0]}

rasa==1.1.5 rasa-sdk==1.1.0 rasa-x==0.19.4

2

Why is this line there?

payload = json.dumps({tracker.get_slot('requested_slot'): None})

3

Well, the user may choose between all recognized entities or none of them. In case the user doesnt want any of the recognized entities i want to reset the slot to None. The line in the loop is pretty much the same, but in these cases the chosen slot value should be delivered.

However I realized, that the payload is incomplete for the None button, since an intent is missing. So

buttons.append({'title': domain['templates'].get('ambigous_none_button'),
                    'payload': payload,

should become

buttons.append({'title': domain['templates'].get('ambigous_none_button'),
                    'payload': '/answer' + payload,

edit: code fixed above

4

@IgNoRaNt23 any solution to the problem you faced, related to button. My button are also not getting displayed though i am able to display the utter_message

5

Didnt work on further on that problem, but I think I understand it now. So, the problem is that you put that button in the dispatcher and the FormAction ends. Now, the MessageProcessor hands the message to the output channel, where it gets displayed. Now, the policy ensemble is asked to predict the next action and since we got an active form the FormAction is chosen. However, what you want in order to be able to answer is ‘action_listen’.

Got stuff to do now, but I think about to possible solution later. In general you want an ActionExecutionRejection to happen and then perform a custom action, that does your job.

6

@IgNoRaNt23 I had the same issue when adding buttons in validate function. I have created another action for fallback and here is a part of it

def name(self):
        return "action_fallback"

    def __init__(self):
        self.intent_mappings = {}
        # read the mapping from a csv and store it in a dictionary
        with open('intent_mapping.csv', newline='', encoding='utf-8') as file:
            csv_reader = csv.reader(file)
            for row in csv_reader:
                self.intent_mappings[row[0]] = row[1]

    def run(self, dispatcher, tracker, domain):
        test =[]
        for event in reversed(tracker.events):
            last_utter_action = event.get('name')
            if event.get('name') not in [ 'action_listen', None ]:
                test.append(last_utter_action)
        
        if tracker.get_slot('requested_slot'):            
            requested_slot = tracker.get_slot('requested_slot')
            slot_prompt = self.intent_mappings[requested_slot]
            latest_message = tracker.latest_message["text"]
            message = "Is this {} is the {}".format(latest_message,slot_prompt)
            
            buttons = [{'title': 'Yes',
                        f'payload': '/{{requested_slot}:{latest_message}}'},
                        {'title': 'No',
                        f'payload': '/{{requested_slot}:None}'}]
            dispatcher.utter_button_message(message, buttons=buttons)
            return[]

        if all(action == 'action_fallback' for action in test[0:2]):
            message = "Hey, I am confusing. You can proceed with"
            buttons = [{'title': 'Apply loan',
                        'payload': '/apply_loan'},
                        {'title': 'Check loan rates',
                        'payload': '/ask_loan_rates'},
                        {'title': 'Learn more details',
                        'payload': '/ask_loan_detail'}]
            dispatcher.utter_button_message(message, buttons=buttons)
            return[]

I the form validate function i did this

def validate_workplace(self, value, dispatcher, tracker, domain):
    if not tracker.get_slot("workplace"):
        if next(tracker.get_latest_entity_values("workplace"), None) is not None:
            return{"workplace":next(tracker.get_latest_entity_values("workplace"), None)}

        else:
            dispatcher.utter_message("Sorry I cannot recognize this company")
            raise ActionExecutionRejection(
                        "action_fallback",
                        "handover to the fallback policy"
                    )

    else:
        return{"workplace": tracker.get_slot("workplace")}

I need to accept the company even if its not in the list if the applicant needs. but this buttons are not working when the form is activating. Other buttons are working fine. Is there any other way to raise ActionExecutionRejection? I couldn’t find a way to add buttons to validate slots inside form.