Form deactivates with intent listed in stories but continious with intent listed in rules

When form is active and we send intent that is not expected, form deactivates normally, and Rasa sends right answer to the intent, if this intent listed in story.

But if we send intent listed in rule, form deactivates, rasa send right answer from the rule and then form activates again.

Here is the log with the “intent from rule” situation:

2021-07-28 08:15:56 DEBUG    rasa.core.policies.memoization  - Sender: '1234567890' -> There is no memorised next action
2021-07-28 08:15:56 DEBUG    rasa.core.policies.ensemble  - Sender: '1234567890' -> Execution of 'form_question' was rejected. Setting its confidence to 0.0 in all predictions.
2021-07-28 08:15:56 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2021-07-28 08:15:56 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2021-07-28 08:15:56 DEBUG    rasa.core.policies.ensemble  - Sender: '1234567890' -> Predicted next action using policy_0_RulePolicy.
2021-07-28 08:15:56 DEBUG    rasa.core.processor  - Sender: '1234567890' -> Predicted next action 'utter_no_stories' with confidence 1.00.
2021-07-28 08:15:56 DEBUG    rasa.core.nlg.callback  - Requesting NLG for utter_no_stories from https://.......
2021-07-28 08:15:56 DEBUG    rasa.core.processor  - Sender: '1234567890'-> Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x000001835C7422B0>]'.
2021-07-28 08:15:56 DEBUG    rasa.core.processor  - Sender: '1234567890'-> Action 'utter_no_stories' ended with events '[BotUttered('', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": {"metadata": {"ucid": "4736734673736332urfjhfhf333"}, "next_action": "transfer", "endpoint": "", "latest_message": {"intent": {"name": "balans", "confidence": "1.0"}}, "channel": "ivr", "agent": null}}, {"button": null, "element":null, "utter_action": "utter_no_stories"}, 1627449356.878203)]'.
2021-07-28 08:15:56 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 0] slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 1] user intent: alarm_questions | previous action name: action_listen | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 2] user intent: alarm_questions | previous action name: form_question | active loop: {'name': 'form_question'} | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,),'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 3] user intent: balans | previous action name: action_listen | active loop: {'name': 'form_question'} | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 4] user intent: balans | previous action name: utter_no_stories | active loop: {'name': 'form_question'} | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
2021-07-28 08:15:56 DEBUG    rasa.core.policies.rule_policy  - Predicted loop 'form_question' by overwriting 'action_listen' predicted by general rule.
2021-07-28 08:15:56 DEBUG    rasa.core.policies.ted_policy  - TED predicted 'action_listen' based on user intent.

What can we do to prevent this behaviour?

Hi @annthehuman. Can you please share the relevant parts of your domain, NLU, stories, and rules files?

Hi! Thank you for quick answer.

Here are rules and stories:

алармы_вопросы.yml (2.1 KB) алармы_до_последовательность_вызова.yml (1.9 KB) офисы_контактный_номер.yml (180 Bytes) правило_помощник_сколько_тебе_лет.yml (156 Bytes)

And domain:

domain.yml (5.2 KB)

We use NLU via API, so you could trigger /alarm_questions to start the form, and than /ofisy_kontaktnyj_nomer to trigger story, or /pomoshtnik_skolxko_tebe_let to trigger rule

Hey @annthehuman, I’ll need more info in order to reproduce this. From the log you pasted above, the conversation goes something like this:

intent: alarm_questions
action: form_question
active_loop: form_question
intent: balans
action: utter_no_stories
action: form_question

However, the balans intent isn’t anywhere in your rules or stories :thinking: Given the training data and domain you posted, can you share two test stories that will demonstrate the difference you mentioned? (I.e. one story where the interjection is handled with rules and one where it’s handled based on a training story.) I’d like to train Rasa Core on your data and then reproduce the issue.

Sorry we collected another stories from our huge amount of training data. Here is part of log relevant to provided data

2021-08-04 13:35:12 DEBUG    rasa.core.policies.memoization  - Sender: '1234567890' -> There is no memorised next action
2021-08-04 13:35:12 DEBUG    rasa.core.policies.ensemble  - Sender: '1234567890' -> Execution of 'form_question' was rejected. Setting its confidence to 0.0 in all predictions.
2021-08-04 13:35:12 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2021-08-04 13:35:12 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2021-08-04 13:35:12 DEBUG    rasa.core.policies.ensemble  - Sender: '1234567890' -> Predicted next action using policy_0_RulePolicy.
2021-08-04 13:35:12 DEBUG    rasa.core.processor  - Sender: '1234567890' -> Predicted next action 'utter_ask_question' with confidence 1.00.
2021-08-04 13:35:12 DEBUG    rasa.core.nlg.callback  - Requesting NLG for utter_ask_question from https://
2021-08-04 13:35:12 DEBUG    rasa.core.processor  - Sender: '1234567890'-> Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x000001F3E9B4EEE0>]'.
2021-08-04 13:35:12 DEBUG    rasa.core.processor  - Sender: '1234567890'-> Action 'utter_ask_question' ended with events '[BotUttered('Чем я могу Вам помочь?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": nu
ll, "image": null, "custom": {"metadata": {"ucid": "4736734673736332urfjhfhf333"}, "channel": "ivr", "next_action": "reply", "latest_message": {"intent": {"name": "pomoshtnik_skolxko_tebe_let", "confidence": "1.0"}}}}, {"utter_action":
"utter_ask_question"}, 1628073312.2779453)]'.
2021-08-04 13:35:12 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 0] slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 1] user intent: alarm_questions | previous action name: action_listen | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,), 'question5': (1.0,), 'pretense-alarm-bef
ore': (1.0,)}
[state 2] user intent: alarm_questions | previous action name: form_question | active loop: {'name': 'form_question'} | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'question4': (1.0,),
'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 3] user intent: pomoshtnik_skolxko_tebe_let | previous action name: action_listen | active loop: {'name': 'form_question'} | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'questio
n4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
[state 4] user intent: pomoshtnik_skolxko_tebe_let | previous action name: utter_ask_question | active loop: {'name': 'form_question'} | slots: {'alarm-before': (1.0,), 'question1': (1.0,), 'question2': (1.0,), 'question3': (1.0,), 'qu
estion4': (1.0,), 'question5': (1.0,), 'pretense-alarm-before': (1.0,)}
2021-08-04 13:35:12 DEBUG    rasa.core.policies.rule_policy  - Predicted loop 'form_question' by overwriting 'action_listen' predicted by general rule.
2021-08-04 13:35:12 DEBUG    rasa.core.policies.ted_policy  - TED predicted 'action_listen' based on user intent.

Now conversation is:

intent: alarm_questions
action: form_question
active_loop: form_question
intent: pomoshtnik_skolxko_tebe_let
action: utter_ask_question
action: form_question

Hey @annthehuman, sorry for the delay. I now tried to reproduce the discrepancy again. However, in my case, whether the unexpected intent is handled by a story or by a rule, the form re-activates after the intent is responded to. So, it remains a mystery to me why you’re seeing the discrepancy… I’ll attach my example and maybe you can tell me what you’re doing differently. I use Rasa Open Source 2.8.3.

Config:

language: en
pipeline:
policies:
  - name: MemoizationPolicy
  - name: RulePolicy

Domain:

version: "2.0"
actions:
- utter_alarm_before
- utter_ask_question
intents:
- pomoshtnik_skolxko_tebe_let:
    use_entities: []
- ofisy_kontaktnyj_nomer:
    use_entities: []
- alarm_questions:
    use_entities: []
- yes:
    use_entities: []
slots:
  question1:
    type: text
    influence_conversation: true
    initial_value: "q1"
  answer1:
    type: text
    influence_conversation: true
forms:
  form_question:
    required_slots:
        answer1:
        - type: from_intent
          value: yes
          intent: yes

Training data:

version: "2.0"
stories:
  - story: офисы_контактный номер
    steps:
      - intent: ofisy_kontaktnyj_nomer
      - action: utter_ask_question
rules:
  - rule: pomoshtnik_skolxko_tebe_let
    steps:
      - intent: pomoshtnik_skolxko_tebe_let
      - action: utter_ask_question
  - rule: form_question
    steps:
    - intent: alarm_questions
    - slot_was_set:
      - question1: set
    - action: form_question
    - active_loop: form_question
  - rule: arbitrary rule just to continue the form...
    steps:
      - active_loop: form_question
      - intent: alarm_questions
      - action: utter_alarm_before

Test stories

version: "2.0"
stories:
- story: rules
- steps:
  - intent: alarm_questions
  - action: form_question
  - active_loop: form_question
  - intent: pomoshtnik_skolxko_tebe_let
  - action: utter_ask_question
  - action: form_question
  - active_loop: form_question
  - intent: alarm_questions
  - action: utter_alarm_before
- story: stories
- steps:
  - intent: alarm_questions
  - action: form_question
  - active_loop: form_question
  - intent: ofisy_kontaktnyj_nomer
  - action: utter_ask_question
  - action: form_question
  - active_loop: form_question
  - intent: alarm_questions
  - action: utter_alarm_before

Note that both of the test stories succeed in my case, i.e. the bot re-activates the form whether the unexpected intent is handled by a rule (pomoshtnik_skolxko_tebe_let) or by a story (ofisy_kontaktnyj_nomer). And this seems to be the intended behaviour (even though it’s not so clear from the docs) – within a loop, rules and stories should be treated as handling unhappy paths and, once these rules/stories finish handling the interjections, the form re-activates again.

Thank you for your answer!

As for documentation we thought what to re-activate the form we need to write “unhappy-path” story, and in all other cases with non-expected intent it deactivates with the help of ActionExecutionRejection.

We have nearly 400 inents, and only one of them is expected in form. What is the best way to deactivate form in our case if coming intent is not expected in form?

@annthehuman now this sounds like a very interesting use case. I asked around to see if it can be solved easily. However, our current solution kind of assumes that even unexpected intents can occur in a form and that the form should continue as long as some rule or story can deal with the intent (in the intent:chitchat->action:utter_chitchat fashion). Out of curiosity, what is the thinking behind wanting to deactivate the form as a default (instead of continuing it)? Can you give an example?

We call forms at the beginning of each dialog if there is some kind of accident. We notify the user about the crash and ask him to answer a few questions if the crash is related to his appeal. If he wants to address some other problem, we need to exit the form, serve his question and wait for the next one, that is, not to return to the form.

We use custom validation, can somehow ActionExecutionRejected help us?

Also I added to our original data ( алармы_вопросы.yml (2.1 KB) алармы_до_последовательность_вызова.yml (1.9 KB) офисы_контактный_номер.yml (180 Bytes) правило_помощник_сколько_тебе_лет.yml (156 Bytes) domain.yml (5.2 KB)) this rule

  - rule: arbitrary rule just to continue the form...
    steps:
      - active_loop: form_question
      - intent: alarm_questions
      - action: utter_alarm_before

and tested it all on test stories:

version: "2.0"
stories:
- story: rules
  steps:
  - intent: alarm_questions
  - action: form_question
  - active_loop: form_question
  - intent: pomoshtnik_skolxko_tebe_let
  - action: utter_ask_question
  - action: form_question
  - active_loop: form_question
  - intent: alarm_questions
  - action: utter_alarm_before
- story: stories
  steps:
  - intent: alarm_questions
  - action: form_question
  - active_loop: form_question
  - intent: ofisy_kontaktnyj_nomer
  - action: utter_ask_question
  - action: form_question
  - active_loop: form_question
  - intent: alarm_questions
  - action: utter_alarm_before

and failed test stories are:

version: "2.0"
stories:
- story: stories (tests\test_stories.yml)
  steps:
  - intent: alarm_questions
  - action: form_question
  - active_loop: form_question
  - intent: ofisy_kontaktnyj_nomer
  - action: utter_ask_question
  - action: form_question  # predicted: action_listen
  - active_loop: form_question
  - intent: alarm_questions
  - action: utter_alarm_before

Seems like Rasa exited form or didn’t provide form question after intent ofisy_kontaktnyj_nomer

Thanks for elaborating on the use case, @annthehuman. Now it makes sense to me.

Regarding ActionExecutionRejected, I think it’s definitely worth a try. If your custom validation runs just after the interjecting intent, it could deactivate the form by returning the event as mentioned here. An alternative could be getting rid of the form and writing the logic as stories.

As for the failed test story you mentioned in your latest message: I think it has nothing to do with the added rule. Maybe you can try out the ActionExecutionRejected approach and see if it works?