Pre-filling form slots with duckling entities

Hey folks!

I’m still wrapping my head around the core building blocks of Rasa, and have a question about how forms and duckling entities interact.

My goal is to use duckling to recognize and pre-fill a slot with a time entity if provided in the intent, otherwise to fill that slot using a form. What I’m unsure about is how to declare the mapping from the slot name (in my case ‘month’) and the duckling entity time (‘time’).

All of my configuration (stories.md, nlu.md, actions.py, config.yml, domain.yml) is in this gist

This is the output of rasa shell -vv. As you can see, the right intent is triggering, and a time entity is being detected, however the month slot is not getting filled, causing the form to ask for information that the user has already provided.

I explicitly want to fill the month slot with the Duckling time entity - not the entity detected by the CRFEntityExtractor, as I want to be robust to multiple ways to specify time.

Jamess-iMac:rasa jmcgill$ rasa shell -vv
2020-02-22 10:59:13 DEBUG    rasa.model  - Extracted model to '/var/folders/m9/3qs5ykv90fg633r0z3b4534m0000gn/T/tmpq976ih6a'.
2020-02-22 10:59:13 DEBUG    rasa.cli.utils  - Parameter 'endpoints' not set. Using default location 'endpoints.yml' instead.
2020-02-22 10:59:13 DEBUG    rasa.cli.utils  - Parameter 'credentials' not set. Using default location 'credentials.yml' instead.
2020-02-22 10:59:14 DEBUG    rasa.model  - Extracted model to '/var/folders/m9/3qs5ykv90fg633r0z3b4534m0000gn/T/tmps07dqyz8'.
2020-02-22 10:59:14 INFO     root  - Connecting to channel 'cmdline' which was specified by the '--connector' argument. Any other channels will be ignored. To connect to all given channels, omit the '--connector' argument.
2020-02-22 10:59:14 DEBUG    sanic.root  - CORS: Configuring CORS with resources: {'/*': {'origins': [''], 'methods': 'DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT', 'allow_headers': ['.*'], 'expose_headers': None, 'supports_credentials': True, 'max_age': None, 'send_wildcard': False, 'automatic_options': True, 'vary_header': True, 'resources': {'/*': {'origins': ''}}, 'intercept_exceptions': True, 'always_send': True}}
2020-02-22 10:59:14 DEBUG    rasa.core.utils  - Available web server routes:
/webhooks/rest                                     GET                            custom_webhook_CmdlineInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_CmdlineInput.receive
/                                                  GET                            hello
2020-02-22 10:59:14 INFO     root  - Starting Rasa server on http://localhost:5005
2020-02-22 10:59:14 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2020-02-22 10:59:14 INFO     root  - Enabling coroutine debugging. Loop id 4501446680.
2020-02-22 10:59:14 DEBUG    rasa.model  - Extracted model to '/var/folders/m9/3qs5ykv90fg633r0z3b4534m0000gn/T/tmpjdedbsqu'.
2020-02-22 10:59:16 INFO     absl  - Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]
2020-02-22 10:59:17 DEBUG    rasa.core.tracker_store  - Connected to InMemoryTrackerStore.
2020-02-22 10:59:17 DEBUG    rasa.core.lock_store  - Connected to lock store 'InMemoryLockStore'.
2020-02-22 10:59:17 DEBUG    rasa.model  - Extracted model to '/var/folders/m9/3qs5ykv90fg633r0z3b4534m0000gn/T/tmph8hdsekp'.
2020-02-22 10:59:17 DEBUG    pykwalify.compat  - Using yaml library: /Users/jmcgill/.pyenv/versions/3.6.2/lib/python3.6/site-packages/ruamel/yaml/__init__.py
2020-02-22 10:59:17 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
Bot loaded. Type a message and press enter (use '/stop' to exit):
Your input ->  I'd like to make a budget for January
2020-02-22 10:59:21 DEBUG    rasa.core.tracker_store  - Creating a new tracker for id 'default'.
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Starting a new session for conversation ID 'default'.
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x14dfa5d68>, <rasa.core.events.ActionExecuted object at 0x14dfa5d30>]'.
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Current slot values:
	month: None
	requested_slot: None
2020-02-22 10:59:21 WARNING  root  - Could not parse timestamp 142977d52718485dadaf8bc1e8286b11. Instead current UTC time will be passed to duckling. Error: invalid literal for int() with base 10: '142977d52718485dadaf8bc1e8286b11'
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Received user message 'I'd like to make a budget for January' with intent '{'name': 'budget', 'confidence': 1.0}' and entities '[{'start': 30, 'end': 37, 'value': 'January', 'entity': 'time', 'confidence': 0.698574552065624, 'extractor': 'CRFEntityExtractor'}, {'start': 30, 'end': 37, 'text': 'January', 'value': '2021-01-01T00:00:00.000-08:00', 'confidence': 1.0, 'additional_info': {'values': [{'value': '2021-01-01T00:00:00.000-08:00', 'grain': 'month', 'type': 'value'}, {'value': '2022-01-01T00:00:00.000-08:00', 'grain': 'month', 'type': 'value'}, {'value': '2023-01-01T00:00:00.000-08:00', 'grain': 'month', 'type': 'value'}], 'value': '2021-01-01T00:00:00.000-08:00', 'grain': 'month', 'type': 'value'}, 'entity': 'time', 'extractor': 'DucklingHTTPExtractor'}]'
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Current slot values:
	month: None
	requested_slot: None
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-02-22 10:59:21 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-02-22 10:59:21 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'intent_budget': 1.0, 'prev_action_listen': 1.0, 'entity_time': 1.0}]
2020-02-22 10:59:21 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '14'
2020-02-22 10:59:21 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-02-22 10:59:21 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_MemoizationPolicy
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Predicted next action 'budget_form' with confidence 1.00.
2020-02-22 10:59:21 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'budget_form'.
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Action 'budget_form' ended with events '[BotUttered('What month do you want to budget for?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1582387161.66399), <rasa.core.events.Form object at 0x14288be10>, <rasa.core.events.SlotSet object at 0x14e1eae48>]'.
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Current slot values:
	month: None
	requested_slot: month
2020-02-22 10:59:21 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-02-22 10:59:21 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'intent_budget': 1.0, 'prev_action_listen': 1.0, 'entity_time': 1.0}]
2020-02-22 10:59:21 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '14'
2020-02-22 10:59:21 DEBUG    rasa.core.policies.form_policy  - There is an active form 'budget_form'
2020-02-22 10:59:21 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'budget'.
2020-02-22 10:59:21 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_FormPolicy
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-02-22 10:59:21 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-02-22 10:59:21 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'default'.
What month do you want to budget for?
Your input ->

Hi there @jmcgill! In order to use duckling as an entity extractor, you don’t need to (and shouldn’t) annotate your NLU data with that entity information. The annotation is only for trainable extractors (CRFEntityExtractor and MitieEntityExtractor), while since duckling is rule-based, entity extraction will always be the same for the same sentence regardless of data.

So remove that entity from both your NLU data and only duckling will do the extracting :slight_smile:

Your slots and slot mappings look fine for the form, so hopefully that should do it. If it doesn’t, I’d recommend running your action server on debug mode for more information about the pre-filled extraction and what it’s picking up.

It would be great is Duckling semantics could be merged with entities discovered by other entity recognizers, like the DIET model.

Right now, when I write a pipeline, it is not clear if the Duckling annotation propagates to the feature extractors as an input feature, or whether it is simply useful ex-post after an extractor is called and fails to find something that Duckling would. For instance, if I put Duckling before an extractor in the pipeline, does this help DIET learn?

Also, how would I be able to test the precision / recall of the time entity if only Duckling is supposed to extract it and it is not labeled in my NLU data?