Categorical Slots - why is the slot being populated with a value not in the list from domain.yml?

Rasa Open Source
1

TLDR; If my AttractionName slot is categorical, why is it being populated with a value outside the list of values provided in the domain?

I’m starting to experiment with Entities and having them influence the conversation. My example is for a theme park (but this is a side jam - I don’t work for the park.)

I have an entity defined with a list of values in domain.yml. I also have a categorical slot that is populated by that entity.

(We’ll watch the “Alice in Wonderland” entity as we go along.)

version: '2.0'
actions:
  - utter_ask_attraction_open_not_recognized
  - utter_ask_attraction_open_yes_generic
  - utter_ask_attraction_open_no_generic
intents:
  - ask_attraction_open:
      use_entities:
        - AttractionName
slots:
  AttractionName:
    influence_conversation: true
    type: categorical
    values:
      - Alice in Wonderland
      - Astro Orbiter
      - Autopia

My NLU file contains synonyms for the categorical entity, as well as training examples:

version: "2.0"
nlu:
- intent: ask_attraction_open
  examples: |
    - is [Alice in Wonderland](AttractionName) open?
    - is [Astro Orbiter](AttractionName) open yet?
    - is [Autopia](AttractionName) working?
    - is [Alice in Wonderland](AttractionName) here?
    - is [Astro Orbiter](AttractionName) built?
    - do you have [Autopia](AttractionName)?
    - have you opened [Alice in Wonderland](AttractionName)
    - have you built [Astro Orbiter](AttractionName)
    - can we go on [Autopia](AttractionName)
    - can we ride [Alice in Wonderland](AttractionName)
    - did you build [Alice in Wonderland](AttractionName)
- synonym: Alice in Wonderland
  examples: |
   - Alice in Wonderland
   - Caterpillars
   - Catterpillars
   - Alices Caterpillars
   - Alice dark ride
   - Alice

In my stories file, I’ve written stories to handle the categorical slot values (Alice in Wonderland, Astro Orbiters, Autopia) and I’ve included a story to handle the other case, which I would expect to see when the entity did not match one of the categorical slot values.

However, when a categorical slot value is “missed” I’m not seeing the story behavior I’d expect.

version: '2.0'
stories:
  - story: Ask Attraction Open - Unrecognized AttractionName
    steps:
      - intent: ask_attraction_open
      - action: utter_ask_attraction_open_not_recognized
  - story: Ask Attraction Open - Unrecognized AttractionName
    steps:
      - intent: ask_attraction_open
      - slot_was_set:
          - AttractionName: __other__
      - action: utter_ask_attraction_open_not_recognized
  - story: Ask Attraction Open - Alice in Wonderland
    steps:
      - entities:
          - AttractionName: Alice in Wonderland
        intent: ask_attraction_open
      - slot_was_set:
          - AttractionName: Alice in Wonderland
      - action: utter_ask_attraction_open_yes_generic
  - story: Ask Attraction Open - Astro Orbiter
    steps:
      - entities:
          - AttractionName: Astro Orbiter
        intent: ask_attraction_open
      - slot_was_set:
          - AttractionName: Astro Orbiter
      - action: utter_ask_attraction_open_refurbishment_generic
  - story: Ask Attraction Open - Autopia
    steps:
      - entities:
          - AttractionName: Autopia
        intent: ask_attraction_open
      - slot_was_set:
          - AttractionName: Autopia
      - action: utter_ask_attraction_open_yes_generic

Responses/Utterances

version: '2.0'
responses:
  utter_ask_attraction_open_not_recognized:
    - text: "I'm sorry, I'm not sure what attraction you're talking about -- Please try asking again or check /RIDES"
  utter_ask_attraction_open_yes_generic:
    - text: "{AttractionName} - This attraction is **open**! Try /warp (attraction name) to go to the queue entrance."
  utter_ask_attraction_open_no_generic:
    - text: "{AttractionName} - Sorry, this attraction has not opened."

The behavior I’m seeing:

So in rasa shell debug, I’ll ask

is alice open?

Alice in Wonderland - This attraction is **open**! Try /warp (attraction name) to go to the queue entrance.

This is correct. It matched on the synonym to a value in the categorical slot. It used that categorical value (not the synonym) in the response utterance.

Now I’ll introduce a typo that does not match a categorical slot value nor entity synonym (not the BEST example, but is clear):

is lalice open?

lalice - This attraction is **open**! Try /warp (attraction name) to go to the queue entrance.

In this case we can see the entity being repeated in the response is “lalice” however that doesn’t match any of the categorical slot values.

Here is the debug output:

Your input ->  is lalice open?                                                                                                                                                                                                                 
2020-12-10 11:53:08 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation 'be3cc0b2b42a4e8da4457733697a4f32'.
2020-12-10 11:53:08 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation 'be3cc0b2b42a4e8da4457733697a4f32'.
2020-12-10 11:53:08 DEBUG    rasa.core.lock_store  - Acquired lock for conversation 'be3cc0b2b42a4e8da4457733697a4f32'.
2020-12-10 11:53:08 DEBUG    rasa.core.tracker_store  - Recreating tracker for id 'be3cc0b2b42a4e8da4457733697a4f32'
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - The latest session for conversation ID 'be3cc0b2b42a4e8da4457733697a4f32' has expired.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Starting a new session for conversation ID 'be3cc0b2b42a4e8da4457733697a4f32'.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.shared.core.events.SessionStarted object at 0x7f8bf553db90>, <rasa.shared.core.events.ActionExecuted object at 0x7f8bf553d710>]'.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Current slot values: 
        AttractionName: None
        attribute: None
        mention: None
        object_type: None
        rule_number: None
2020-12-10 11:53:08 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-12-10 11:53:08 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Received user message 'is lalice open?' with intent '{'id': -547235365317989774, 'name': 'ask_attraction_open', 'confidence': 0.9999890327453613}' and entities '[{'entity': 'AttractionName', 'start': 3, 'end': 9, 'confidence_entity': 0.9842124581336975, 'value': 'lalice', 'extractor': 'DIETClassifier'}]'
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Current slot values: 
        AttractionName: lalice
        attribute: None
        mention: None
        object_type: None
        rule_number: None
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 73 events.
2020-12-10 11:53:08 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 0] user intent: ask_attraction_open | user entities: ('AttractionName',) | previous action name: action_listen | slots: {'AttractionName': (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0)}
2020-12-10 11:53:08 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-12-10 11:53:08 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: ask_attraction_open | user entities: ('AttractionName',) | previous action name: action_listen | slots: {'AttractionName': (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0)}
2020-12-10 11:53:08 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2020-12-10 11:53:08 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_TEDPolicy.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Predicted next action 'utter_ask_attraction_open_yes_generic' with confidence 0.58.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Action 'utter_ask_attraction_open_yes_generic' ended with events '[BotUttered('lalice - This attraction is **open**! Try /warp (attraction name) to go to the queue entrance.', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {"template_name": "utter_ask_attraction_open_yes_generic"}, 1607629988.705365)]'.
2020-12-10 11:53:08 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 0] user intent: ask_attraction_open | user entities: ('AttractionName',) | previous action name: utter_ask_attraction_open_yes_generic | slots: {'AttractionName': (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0)}
2020-12-10 11:53:08 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-12-10 11:53:08 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: ask_attraction_open | user entities: ('AttractionName',) | previous action name: action_listen | slots: {'AttractionName': (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0)}
[state 2] user intent: ask_attraction_open | user entities: ('AttractionName',) | previous action name: utter_ask_attraction_open_yes_generic | slots: {'AttractionName': (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0)}
2020-12-10 11:53:08 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2020-12-10 11:53:08 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_TEDPolicy.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2020-12-10 11:53:08 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-12-10 11:53:08 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'be3cc0b2b42a4e8da4457733697a4f32'.
lalice - This attraction is **open**! Try /warp (attraction name) to go to the queue entrance.
Your input ->

If my AttractionName slot is categorical, why is it being populated with a value outside the list of values provided in the domain?

2

I’m still curious about this issue. Any insights?

3

Hi @Chris98106, which version of Rasa Open Source are you on? It should map your slot values to the __other__ value.

4

Hey @akelad

Here is my rasa --version:

Rasa Version     : 2.1.3
Rasa SDK Version : 2.1.2
Rasa X Version   : 0.33.0rc1
Python Version   : 3.7.7
Operating System : Darwin-20.1.0-x86_64-i386-64bit
Python Path      : /Users/Chris/v-waldorf/bin/python3

What you describe is what I’m expecting too. In the last line of the debug output on my main post you can see the utterance responds with a first word of {AttractionName}.

In that example I made an intentional typo but the slot is populated with the misspelled value lalice and not the value I expect of __other__.

I’ll try upgrading to latest and seeing if there are any patches I’m missing.

5

Hm, given that the TEDPolicy jumped in with a confidence of 0.58, it seems like this might be an issue with your stories. I just double checked, and the slot will get populated with the value that was extracted, but in terms of featurization it does get featurized as __other__.

6

@akelad - Is this the correct story structure to match that “other” case then? This is how I was trying to capture those but they are not falling into this behavior.

  - story: Ask Attraction Open - Unrecognized AttractionName
    steps:
      - intent: ask_attraction_open
      - slot_was_set:
          - AttractionName: __other__
      - action: utter_ask_attraction_open_not_recognized
7

Yeah it should be. But could you actually try adding an entities step to that story as well? Because it seems like the entity is still extracted, and so that should be reflected in your story.

8

The behavior seems to be much-much better after I added the entities step to my “other” story, thanks!

  - story: Ask Attraction Open - Not Recognized with entity and other slot
    steps:
      - entities:
          - AttractionName: Lorem Ipsum
        intent: ask_attraction_open
      - slot_was_set:
          - AttractionName: __other__
      - action: utter_ask_attraction_open_not_recognized
3 Likes