Form execution rejected if slot is not extracted

Hi everyone,

Recently we have a problem with our formactions. This problem seems to appear in all forms and we have tried it out on the “good mood” bot of the build-in rasa example. We added a small form here and the problem appears as well.

Problem description: Form execution rejected if slot is not extracted. (TED policy takes over)

Assumed behavior: Form repeats asking for slot.

Used Files: Domain.yml

version: "2.0"

intents:
  - greet
  - goodbye
  - affirm
  - deny
  - mood_great
  - mood_unhappy
  - bot_challenge


entities:
  - name
  - weekday

slots:
  name:
    type: text
    influence_conversation: true
  weekday:
    type: text
    influence_conversation: true

responses:
  utter_greet:
  - text: "Hey! How are you?"

  utter_cheer_up:
  - text: "Here is something to cheer you up:"
    image: "https://i.imgur.com/nGF1K8f.jpg"

  utter_did_that_help:
  - text: "Did that help you?"

  utter_happy:
  - text: "Great, carry on!"

  utter_goodbye:
  - text: "Bye"

  utter_iamabot:
  - text: "I am a bot, powered by Rasa."
  
  utter_ask_test_form_name:
  - text: Whats your name? 
  
  utter_ask_weekday:
  - text: Which weekday is it?

forms:
  test_form:
    required_slots:
      name:
        - type: from_text
      weekday:
        - type: from_entity
          entity: weekday     


session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true

Stories.yml

version: "2.0"

stories:

- story: happy path
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_great
  - action: test_form
  - active_loop: test_form
  - active_loop: null
  - action: utter_happy

- story: sad path 1
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - intent: affirm
  - action: utter_happy

- story: sad path 2
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - intent: deny
  - action: utter_goodbye

Rules.yml

version: "2.0"

rules:

- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye

- rule: Say 'I am a bot' anytime the user challenges
  steps:
  - intent: bot_challenge
  - action: utter_iamabot

- rule: Activate form
  steps:
  - intent: mood_great
  - action: test_form
  - active_loop: test_form

- rule: Submit form
  condition:
  # Condition that form is active.
  - active_loop: test_form
  steps:
  # Form is deactivated
  - action: test_form
  - active_loop: null
  - slot_was_set:
    - requested_slot: null
  # The actions we want to run when the form is submitted.
  - action: utter_happy

nlu.yml

version: "2.0"

nlu:
- intent: greet
  examples: |
    - hey
    - hello
    - hi
    - hello there
    - good morning
    - good evening
    - moin
    - hey there
    - let's go
    - hey dude
    - goodmorning
    - goodevening
    - good afternoon

- intent: goodbye
  examples: |
    - good afternoon
    - cu
    - good by
    - cee you later
    - good night
    - bye
    - goodbye
    - have a nice day
    - see you around
    - bye bye
    - see you later

- intent: affirm
  examples: |
    - yes
    - y
    - indeed
    - of course
    - that sounds good
    - correct

- intent: deny
  examples: |
    - no
    - n
    - never
    - I don't think so
    - don't like that
    - no way
    - not really

- intent: mood_great
  examples: |
    - perfect
    - great
    - amazing
    - feeling like a king
    - wonderful
    - I am feeling very good
    - I am great
    - I am amazing
    - I am going to save the world
    - super stoked
    - extremely good
    - so so perfect
    - so good
    - so perfect
    - at [Saturday](weekday)
    - at [Monday](weekday) 
    - at [Friday](weekday)
    - at [Sunday](weekday) 
    - at [Tuesday](weekday)
    - at [Wednesday](weekday) 
    - at [Thursday](weekday)

- intent: mood_unhappy
  examples: |
    - my day was horrible
    - I am sad
    - I don't feel very well
    - I am disappointed
    - super sad
    - I'm so sad
    - sad
    - very sad
    - unhappy
    - not good
    - not very good
    - extremly sad
    - so saad
    - so sad

- intent: bot_challenge
  examples: |
    - are you a bot?
    - are you a human?
    - am I talking to a bot?
    - am I talking to a human?

config.yml/credential.yml/endpoints.yml stay default

Output: If the entities are given correctly in the dialog, everything goes fine (formaction gets activated, rulepolicy gets us through the form and the form gets deactivated correctly)

If the entities are missing in the user input (for example saying hello instead of giving the weekday) the following output is generated:

Which weekday is it?
Your input ->  hello
rasa.core.processor  - Received user message 'hello' with intent '{'id': 286870438279032377, 'name': 'greet', 'confidence': 0.9999989867210388}' and entities '[]'
rasa.core.policies.rule_policy  - There is no applicable rule.
rasa.core.policies.rule_policy  - Predicted loop 'test_form'.
rasa.core.policies.ensemble  - Made prediction using user intent.
rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
rasa.core.policies.ensemble  - Predicted next action using policy_2_RulePolicy.
rasa.core.processor  - Predicted next action 'test_form' with confidence 1.00.
rasa.core.actions.forms  - Validating user input 'UserUttered(text: hello, intent: greet, use_text_for_featurization: False)'.
rasa.core.actions.forms  - Trying to extract requested slot 'weekday' ...
rasa.core.actions.forms  - Got mapping '{'type': 'from_entity', 'entity': 'weekday'}'
rasa.core.actions.forms  - Failed to extract requested slot 'weekday'
rasa.core.actions.forms  - Validating extracted slots: {}
rasa.core.policies.rule_policy  - There is no applicable rule.
rasa.core.policies.ensemble  - Execution of 'test_form' was rejected. Setting its confidence to 0.0 in all predictions.
rasa.core.policies.ensemble  - Made prediction using user intent.
rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
rasa.core.policies.ensemble  - Predicted next action using policy_1_TEDPolicy.
rasa.core.processor  - Predicted next action 'utter_greet' with confidence 0.98.

Most likely there is a misunderstanding on our side but shouldn’t this input lead to repeated question “Which weekday is it?”

Thank you in advance! Your help is appreciated

@thinkinnet Hi, Do you have repeat intent?

not_intent: repeat

@thinkinnet Try change active_loop: null position

- rule: Submit form
  condition:
  # Condition that form is active.
  - active_loop: test_form
  steps:
  # Form is deactivated
  - action: test_form
  - active_loop: null
  - slot_was_set:
    - requested_slot: null
 
  # The actions we want to run when the form is submitted.
  - action: utter_happy

@thinkinnet For more information about form please see this repo : GitHub - RasaHQ/rasa-form-examples: This repository contains a few simple projects with forms.</ti

I hope this will solve your issue.

Hi @nik202, thanks for your quick response!

First question: This is indeed copied from the original code where there is a repeat intent and we won’t record an answer like “can you please repeat” as a name (this is a voice assistant so this is more likely). But it does not affect the problem I am speaking of here. The name slot is stored correctly,

The problem occurs when we try to get a slot from an entity recognized by the diet classifier or a regex for example. (like the weekday example). The examples in the repo you posted are unfortunately all “from text” so I cannot reproduce the same situation in this examples.

Your 2nd question: I tried to change the order of the active_loop: null and slot_was_set. But the same problem occurs.

Do you have an example like the ones you posted with an other sort of entity perhaps? (like a diet classifier example?)

Hi @nik202, I edited my initial post with your comments. The error still occurs. Especially these lines in the output are strange:

rasa.core.actions.forms  - Trying to extract requested slot 'weekday' ...
rasa.core.actions.forms  - Got mapping '{'type': 'from_entity', 'entity': 'weekday'}'
rasa.core.actions.forms  - Failed to extract requested slot 'weekday'

rasa.core.policies.rule_policy  - There is no applicable rule.
rasa.core.policies.ensemble  - Execution of 'test_form' was rejected. Setting its confidence to 0.0 in all predictions.

Isn’t it a default behavior for forms to ask again the same question if a slot is not filled after user input?

@thinkinnet It’s very strange; please share me rasa --version and all the related files, I will try run this code.

Hi @nik202, that would be very nice. The current using rasa version is:

~/project/test$ rasa --version
Rasa Version      :         2.6.0
Minimum Compatible Version: 2.6.0
Rasa SDK Version  :         2.8.1
Rasa X Version    :         None
Python Version    :         3.8.10
Operating System  :         Linux-5.11.0-27-generic-x86_64-with-glibc2.29
Python Path       :         /home/.../venv/bin/python3

But we face this in 2.7.x as well (I was trying this with 2.6 because I assumed changes in versions)

You can find the files here test.zip (7.0 KB)

As mentioned, the problem is when you are asked for the 2nd slot (weekday) and you give information without the entity that is expected. Then the bot rejects the form instead of asking again.

Thanks for checking this, very appreciated!

@thinkinnet meanwhile can you please update the rasa version to rasa 2.8.1

pip install rasa==2.8.1 and try run. Just in case.

Hi @nik202, I just checked the 2.8.1 version. Same behavior.

@thinkinnet I am checking your code now.

@thinkinnet Are you expecting this use case only? or give me bot user conversation example.

Hi @nik202, thanks for checking!

Your dialog shows the happy path where a weekday is given and correctly extracted.

But when the bot asks “which day is it?” We want to say for example “I don’t know” or anything else that doesn’t give the weekday entity. In this case we expect the bot to repeat his question, but instead he exits the form and predicts the next action via TED policy.

@thinkinnet Right, But in your file there is no intent for “which day” or even training example also, do I am missing something, please tell me.

Hi @nik202, well actually I have to create an intent inform or something like that for the weekday indeed. As for now you find the entity recognition for Diet classifier in the intent mood_great. So for now the intent mood_great is recognized if you give a weekday. Of course this is not the way to do it but that’s not the point of the problem here.

I will give 2 example dialogs:

First one (happy path) same as you did above (works fine):

Your input → hi
Hey! How are you? Your input → great
Whats your name? Your input → mark
Which weekday is it? Your input → monday
Great, carry on! Your input →

Second (no day is given):

Your input → hi
Hey! How are you? Your input → great
Whats your name? Your input → mark
Which weekday is it? Your input → i don’t know
Bye

In the second example we would expect to get the question “which weekday is it” again, until the user gives an answer that includes a weekday. Instead he gives utter_goodby chosen by TED policy (instead of rule policy)

@thinkinnet Mean you want to repeat the same question again and again till the user does not enter the value for that question fine got it. For this use case, you can do this with forms .

Hi @nik202, I know forms and in the example you can see my solution does work with a form. But the problem is the form does not do what it’s suppose to do. It does not repeat the question.

@thinkinnet I will see again your use case and ask some questions if you dnt mind, meanwhile you carry on with your project.

Of course, your help is very appreciated. Thank you!

@thinkinnet I am sorry, I not get time for running your use case, I will try today hope so please. Thanks for your patience :pray:

This scenario worked fine checked :white_check_mark: First one (happy path) same as you did above (works fine):

Your input → hi
Hey! How are you? 
Your input → great
Whats your name?
 Your input → mark
Which weekday is it?
Your input → monday
Great, carry on! Your input →

Now second one

Second (no day is given):

Your input → hi
Hey! How are you?
 Your input → great
Whats your name? 
Your input → mark
Which weekday is it? 
Your input → i don’t know
Bye

You want that bot, again will ask the same question

Your input → i don’t know
Which weekday is it?

Till the time user not enter the input right?

Hi @nik202, I just got a tip that this is expected behavior. In Forms I see an execution rejection is send if a slot is not set. I’m very confused because I assumed the question would be asked again.

Yes