RASA not filling in slots for entities detected by Spacy?

Rasa Open Source
1

Hi; I’m trying to create an appointment bot in RASA where I use Spacy’s entity extractor to detect values like dates and times. For example,

"Book an appointment" => fires book_appointment intent

"Book an appointment for tomorrow" => fires book_appointment intent + fills entity DATE

"Book an appointment for tomorrow at 3 PM" => fires book_appointment intent + fills entity DATE + fills entity TIME

The entities are getting detected just fine, and I’ve specified each entity and their values in my actions.py file as well:

class AppointmentForm(FormAction):

def name(self):
    return "appointment_form"

@staticmethod
def required_slots(tracker: Tracker) -> List[Text]: return [
    "appointment_date", "appointment_time", "client_name", "client_phone"]

def slot_mappings(self):
    return {
        "appointment_date": [self.from_entity(entity="DATE")],
        "appointment_time": [self.from_entity(entity="TIME")],
        "client_name": [self.from_text()],
        "client_phone": [self.from_text()],
    }

def submit(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict[Text, Any],
) -> List[Dict]:
    return []

However, the bot is still asking me for a date (utter_ask_appointment_date) even when an input specifies the entities mentioned above. Here is the bot running with --debug:

Your input ->  book an appointment for tomorrow at 3 PM                                                                                                         
2020-01-28 20:37:18 DEBUG    rasa.core.tracker_store  - Creating a new tracker for id 'default'.
2020-01-28 20:37:18 DEBUG    rasa.core.processor  - Starting a new session for conversation ID 'default'.
2020-01-28 20:37:18 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x00000164DFE72FC8>, <rasa.core.events.ActionExecuted object at 0x00000164DFE72A48>]'.
2020-01-28 20:37:18 DEBUG    rasa.core.processor  - Current slot values:
        appointment_date: None
        appointment_time: None
        client_name: None
        client_phone: None
        requested_slot: None
2020-01-28 20:37:18 DEBUG    rasa.core.processor  - Received user message 'book an appointment for tomorrow at 3 PM' with intent '{'name': 'book_appointment', 'confidence': 0.9226344343217546}' and entities '[{'entity': 'DATE', 'value': 'tomorrow', 'start': 24, 'confidence': None, 'end': 32, 'extractor': 'SpacyEntityExtractor'}, {'entity': 'TIME', 'value': '3 PM', 'start': 36, 'confidence': None, 'end': 40, 'extractor': 'SpacyEntityExtractor'}]'
2020-01-28 20:37:18 DEBUG    rasa.core.processor  - Current slot values:
        appointment_date: None
        appointment_time: None
        client_name: None
        client_phone: None
        requested_slot: None
2020-01-28 20:37:18 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-01-28 20:37:18 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'intent_book_appointment': 1.0, 'entity_DATE': 1.0, 'entity_TIME': 1.0, 'prev_action_listen': 1.0}]
2020-01-28 20:37:18 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-01-28 20:37:19 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-01-28 20:37:19 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_KerasPolicy
2020-01-28 20:37:19 DEBUG    rasa.core.processor  - Predicted next action 'appointment_form' with confidence 0.16.
2020-01-28 20:37:19 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'appointment_form'.
2020-01-28 20:37:21 DEBUG    rasa.core.processor  - Action 'appointment_form' ended with events '[BotUttered('What date?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1580224041.1129706), <rasa.core.events.Form object at 0x00000164E0FE8908>, <rasa.core.events.SlotSet object at 0x00000164E0FE8A88>]'.
2020-01-28 20:37:21 DEBUG    rasa.core.processor  - Current slot values:
        appointment_date: None
        appointment_time: None
        client_name: None
        client_phone: None
        requested_slot: appointment_date
2020-01-28 20:37:21 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'intent_book_appointment': 1.0, 'entity_DATE': 1.0, 'entity_TIME': 1.0, 'prev_action_listen': 1.0}]
2020-01-28 20:37:21 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-01-28 20:37:21 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'book_appointment'.
2020-01-28 20:37:21 DEBUG    rasa.core.policies.form_policy  - There is an active form 'appointment_form'
2020-01-28 20:37:21 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_3_FormPolicy
2020-01-28 20:37:21 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-01-28 20:37:21 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-01-28 20:37:21 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'default'.
What date?
Your input ->

I have no idea why RASA refuses to fill in the slot for appointment_date or appointment_time despite the entities being detected and being specifically told to fill in using said entities.

Please help.

2

@ActuallyAcey can I see what your stories look like? And is this the first message that triggers the form?

3

Sure.

## happy path
* greet
- utter_greet
- action_restart

## happy path 
* book_appointment
- appointment_form
- utter_results
- action_send_appointment_data
- action_restart

Following this tutorial, I tried adding a - form{"name": null} but that only led to my bot going into a frozen state, where the circuit breaker fired after a little while.

4

So you should definitely have the story with form events, like this:

## happy path 
* book_appointment
- appointment_form
- form{"name":"appointment_form"}
- form{"name":null}
- utter_results
- action_send_appointment_data
- action_restart
5

I am facing similar issues. I have hacked into rasa_sdk and am dumping tracker state. You can see the entities are there. I will attach bellow logs from both rasa and action server.

Rasa

Rasa 
Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input ->  Find a flight from Madrid
2020-02-25 15:23:14 DEBUG    rasa.core.tracker_store  - Creating a new tracker for id 'default'.
2020-02-25 15:23:14 DEBUG    rasa.core.processor  - Starting a new session for conversation ID 'default'.
2020-02-25 15:23:14 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x7f21d975ca90>, <rasa.core.events.ActionExecuted object at 0x7f21d975c860>]'.
2020-02-25 15:23:14 DEBUG    rasa.core.processor  - Current slot values:
	destination: None
	origin: None
	requested_slot: None
2020-02-25 15:23:15 WARNING  root  - Could not parse timestamp 8b0ef590c5974e2c8e1bf749f52fc218. Instead current UTC time will be passed to duckling. Error: invalid literal for int() with base 10: '8b0ef590c5974e2c8e1bf749f52fc218'
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Received user message 'Find a flight from Madrid' with intent '{'name': 'request_flight_info', 'confidence': 0.9999974178552901}' and entities '[{'entity': 'GPE', 'value': 'Madrid', 'start': 19, 'confidence': None, 'end': 25, 'extractor': 'SpacyEntityExtractor'}]'
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Current slot values:
	destination: None
	origin: None
	requested_slot: None
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-02-25 15:23:15 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'prev_action_listen': 1.0, 'intent_request_flight_info': 1.0, 'entity_GPE': 1.0}]
2020-02-25 15:23:15 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-02-25 15:23:15 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-02-25 15:23:15 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_KerasPolicy
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Predicted next action 'flight_info_form' with confidence 0.74.
2020-02-25 15:23:15 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'flight_info_form'.
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Action 'flight_info_form' ended with events '[BotUttered('Where are you flying from?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1582644195.3159652), <rasa.core.events.Form object at 0x7f21d970f048>, <rasa.core.events.SlotSet object at 0x7f21fe32c5f8>]'.
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Current slot values:
	destination: None
	origin: None
	requested_slot: origin
2020-02-25 15:23:15 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'prev_action_listen': 1.0, 'intent_request_flight_info': 1.0, 'entity_GPE': 1.0}]
2020-02-25 15:23:15 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-02-25 15:23:15 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'request_flight_info'.
2020-02-25 15:23:15 DEBUG    rasa.core.policies.form_policy  - There is an active form 'flight_info_form'
2020-02-25 15:23:15 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_3_FormPolicy
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-02-25 15:23:15 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-02-25 15:23:15 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'default'.
Where are you flying from?
Action server 
2020-02-25 15:22:44 INFO     rasa_sdk.endpoint  - Starting action endpoint server...
2020-02-25 15:22:44 INFO     rasa_sdk.executor  - Registered function for 'flight_info_form'.
2020-02-25 15:22:44 DEBUG    rasa_sdk.utils  - Using the default number of Sanic workers (1).
2020-02-25 15:23:15 DEBUG    rasa_sdk.executor  - Received request to run 'flight_info_form'
2020-02-25 15:23:15 DEBUG    rasa_sdk.forms  - There is no active form
2020-02-25 15:23:15 DEBUG    rasa_sdk.forms  - Activated the form 'flight_info_form'
{'sender_id': 'default', 'slots': {'destination': None, 'origin': None, 'requested_slot': None}, 'latest_message': {'intent': {'name': 'request_flight_info', 'confidence': 0.9999974178552901}, 'entities': [{'entity': 'GPE', 'value': 'Madrid', 'start': 19, 'confidence': None, 'end': 25, 'extractor': 'SpacyEntityExtractor'}], 'intent_ranking': [{'name': 'request_flight_info', 'confidence': 0.9999974178552901}, {'name': 'iata_input', 'confidence': 2.582144710006946e-06}], 'text': 'Find a flight from Madrid'}, 'latest_event_time': 1582644195.1142032, 'paused': False, 'events': [{'event': 'action', 'timestamp': 1582644194.979026, 'name': 'action_session_start', 'policy': None, 'confidence': None}, {'event': 'session_started', 'timestamp': 1582644194.979034}, {'event': 'action', 'timestamp': 1582644194.9790516, 'name': 'action_listen', 'policy': None, 'confidence': None}, {'event': 'user', 'timestamp': 1582644195.1142032, 'text': 'Find a flight from Madrid', 'parse_data': {'intent': {'name': 'request_flight_info', 'confidence': 0.9999974178552901}, 'entities': [{'entity': 'GPE', 'value': 'Madrid', 'start': 19, 'confidence': None, 'end': 25, 'extractor': 'SpacyEntityExtractor'}], 'intent_ranking': [{'name': 'request_flight_info', 'confidence': 0.9999974178552901}, {'name': 'iata_input', 'confidence': 2.582144710006946e-06}], 'text': 'Find a flight from Madrid'}, 'input_channel': 'cmdline', 'message_id': '8b0ef590c5974e2c8e1bf749f52fc218', 'metadata': {}}], 'latest_input_channel': 'cmdline', 'active_form': {}, 'latest_action_name': 'action_listen'}
2020-02-25 15:23:15 DEBUG    rasa_sdk.forms  - No pre-filled required slots to validate.
2020-02-25 15:23:15 DEBUG    rasa_sdk.forms  - Validating user input '{'intent': {'name': 'request_flight_info', 'confidence': 0.9999974178552901}, 'entities': [{'entity': 'GPE', 'value': 'Madrid', 'start': 19, 'confidence': None, 'end': 25, 'extractor': 'SpacyEntityExtractor'}], 'intent_ranking': [{'name': 'request_flight_info', 'confidence': 0.9999974178552901}, {'name': 'iata_input', 'confidence': 2.582144710006946e-06}], 'text': 'Find a flight from Madrid'}'
2020-02-25 15:23:15 DEBUG    rasa_sdk.forms  - Validating extracted slots: {}
2020-02-25 15:23:15 DEBUG    rasa_sdk.forms  - Request next slot 'origin'
2020-02-25 15:23:15 DEBUG    rasa_sdk.executor  - Finished running 'flight_info_form'
FlightInfoForm 
class FlightInfoForm(FormAction):
  """Example of a custom form action"""

  def name(self) -> Text:
    """Unique identifier of the form"""

    return "flight_info_form"

  @staticmethod
  def required_slots(tracker: Tracker) -> List[Text]:
    """A list of required slots that the form has to fill"""

    return ["origin", "destination"]

  def slot_mappings(self) -> Dict[Text, Union[Dict, List[Dict]]]:
    return {
      "origin": [
        self.from_entity(entity="GPE"), // Spacy entity
        self.from_entity(entity="iata")
      ],
      "destination": [
        self.from_entity(entity="GPE"),  // Spacy entity
        self.from_entity(entity="iata")
      ]
    }

    // submit is not relevant here, I have it
}
Stories 
## ask for flight info
* request_flight_info
  - flight_info_form
  - form{"name": "flight_info_form"}
  - form{"name": null}

## ask for flight info with an origin
* request_flight_info{"origin": "Madrid"}
  - flight_info_form
  - form{"name": "flight_info_form"}
  - form{"name": null}

## ask for flight info with an origin and destination
* request_flight_info{"origin": "New York", "destination": "Moscow"}
  - flight_info_form
  - form{"name": "flight_info_form"}
  - form{"name": null}
6

@eugeniumegherea I used a rather hacky workaround; Spacy puts the entities it detects into its own new entities with the same name. For example, a time entity detected by Spacy (“book an appointment for [9 PM]”) be placed in a new entity called TIME.

As such, I just created slots with the same names (TIME, DATE) and let Rasa’s automatic slot filling do the work for me. I did have to write a bit of custom action logic for converting these raw values (e.g. “Tomorrow”, “3 am”, etc.) Into actual DB-friendly friendly values (like ISO strings) but that was a price I was willing to pay. :stuck_out_tongue:

7

@ActuallyAcey Well yes, this will actually work

But in my case I can have an input like Search for a flight from [London](GPE) to [Paris](GPE) and this somehow needs to be mapped to origin and destination. In this case I really need to map these entities to my slots

8

Hi @eugeniumegherea!

I wanted to point out that you don’t need three different stories to handle the three cases explicitly, this kinda defeats the purpose of having FormActions (think if you had 10 slots to fill, how many stories would you have written?). Instead of doing this, you could write this under intents in your domain.yml:

- request_flight_info:
    use_entities: []

and just write this in your stories.md:

## ask for flight info
* request_flight_info
  - flight_info_form
  - form{"name": "flight_info_form"}
  - form{"name": null}

What all this does is that rasa will not consider the extracted entities when deciding on what next action it should take. Hence it would always trigger the FormAction regardless of the entities.

Secondly, when the FormAction activates, it searches for pre-filled slots (which rasa fills by extracting entities from a user’s message and filling the slots with the same name.) Therefore if you want your custom slots to be filled with custom entities, you should have some nlu training examples like:

## intent: request_flight_info
- Search for a flight from [London](origin) to [Paris](destination)

and use CRFEntityExtractor in your pipeline along with the SpacyEntityExtractor.

Hope that hepls!

9

Thanks for your insights

What you described makes sense in the way of defining stories for form. Howether, the use case I described still does not work.

Let me better explain. I use

- name: "SpacyEntityExtractor" dimensions: ["GPE", "FAC"]

to extract cities and airports. They are matched as GPE and FAC entity names. I do not want to train my NLU to recognize cities because I want to use Spacy for that. NLU works as expected. The problem I think is in the form handling itself on the action server.

Now, Because this is a form, rasa server makes a post to the action server which decides to fill the slots or not. I can confirm (see the action server logs above) that action server does indeed receive all entities (including Spacy ones).

Because I need to fill 2 slots: origin and destination, I want to map GPE entity to origin and if there is another GPE - map it to destination ( thats the purpose of self.from_entity(), right? ). But it is not mapped.

So, what I want is when user says: search a flight from Madrid to London, slot origin and slot destination is filled with [Madrid](GPE) and [London](GPE). And because there are no more slots to fill, the form should submit and the form logic should complete.

What happens in reality is that the auto fill does not work and action server is asking to input origin where are you flying from?

10

@akelad can you take a look please

11

Okay, so let me explain why the auto-fill is not working. When a form activates it looks at the required slots (returned by the required_slots method) and checks if it is already filled or not. In your case the GPE entity is getting extracted by the nlu (before the form activates) and if there were any slots of the name GPE they should be getting filled, but since there is no extracted entity of the name origin, hence the slot origin is not getting auto-filled. After this the form activates and checks if the required slots (i.e. origin and destination) are already filled or not (which they are not due the reason I explained earlier.). Since they are not filled, so the form asks the user to fill it.

For the self.from_entity() part, it will only get mapped if the requested_slot is set to one of the required_slots and the nlu extracts the entity passed to self.from_entity(). This slot mapping happens when the form is active. But the auto-filling happens before the form activates.

Hope you got what I’m trying to say.

12

Thank you for explaining this, I get it now.

Speaking about my case, is there any way to make it work with Spacy entities? I don’t really want to train my NLU to detect cities as origin/destination if Spacy already does this nicely…

One workaround I think of is creating some sort of proxy to change entity name based on some condition, like:

if next_action is flight_info_form then
   if there is an entity named GPE and it is first then
     rename entity name to "origin"
   if there is an entity named GPE and it is second then
     rename entity name to "destination"

In this case, the slot names would be the same and autofill should work

13

About this proxy, you could use a custom action which would simply do what you said. Put this custom action just before the form action in your stories, something like this:

## ask for flight info
* request_flight_info
  - action_fill_extracted_slots
  - flight_info_form
  - form{"name": "flight_info_form"}
  - form{"name": null}

Now in the custom action, extract the values of the entity GPE (look into get_latest_entity_values) and simply return slot set events to set the slot origin and destination with extracted values. This way when the form will activate, it’ll get those slots as filled and will not ask for it, otherwise it’ll ask.

14

This is it I think. This should really work.

I will try it. Thank you very much

1 Like