Rules are only recognized within the first turns

Claude · March 31, 2022, 1:12pm

I have built a social robot with some stories, custom actions and also some rules. But the rules are not reliably recognised. Especially when a few dialogue steps have already been completed. They work well at the very beginning of the dialogue.

policies:
  - name: RulePolicy
  - name: MemoizationPolicy
    max_history: 5
  - name: TEDPolicy
    max_history: 10
    epochs: 50
    constrain_similarities: true

- rule: ask skills
  steps:
  - intent: ask_skills
  - action: utter_answer_skills

debug:
[state 13] user intent: ask_skills | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 1.0), 'asked_well_being': (1.0, 1.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-03-31 13:48:59 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.

Thank you very much!

stephens · April 1, 2022, 6:36pm

Can you post a link to your repo? Can you show a more complete log of the policy prediction. Should show all of the policy predictions and the results.

Claude · April 6, 2022, 1:16pm

Hello Greg, unfortunately, the repo is not currently public. I have now reduced the storries and rules to just one each to narrow down the error. Now even the simple story is not being executed. There is something fundamentally wrong. Below is the complete log file. I don’t know if you can help here? Thank you very much!

2022-04-06 15:01:49,553 [DEBUG]  CORS: Configuring CORS with resources: {'/*': {'origins': [''], 'methods': 'DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT', 'allow_headers': ['.*'], 'expose_headers': 'filename', 'supports_credentials': True, 'max_age': None, 'send_wildcard': False, 'automatic_options': True, 'vary_header': True, 'resources': {'/*': {'origins': ''}}, 'intercept_exceptions': True, 'always_send': True}}
2022-04-06 15:01:49,555 [DEBUG]  Available web server routes: 
/webhooks/rest                                     GET                            rasa.core.run.custom_webhook_CmdlineInput.health
/webhooks/rest/webhook                             POST                           rasa.core.run.custom_webhook_CmdlineInput.receive
/                                                  GET                            rasa.core.run.hello
2022-04-06 15:01:49,555 [INFO ]  Starting Rasa server on http://0.0.0.0:5005
2022-04-06 15:01:49,556 [DEBUG]  Using the default number of Sanic workers (1).
2022-04-06 15:01:49,946 [DEBUG]  Starting new HTTPS connection (1): api.segment.io:443
2022-04-06 15:01:50,672 [DEBUG]  https://api.segment.io:443 "POST /v1/track HTTP/1.1" 200 21
2022-04-06 15:01:50,695 [DEBUG]  Connected to InMemoryTrackerStore.
2022-04-06 15:01:50,695 [DEBUG]  Connected to lock store 'InMemoryLockStore'.
2022-04-06 15:01:50,695 [DEBUG]  Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2022-04-06 15:01:50,696 [INFO ]  Loading model models/20220406-145531-yellow-alfalfa.tar.gz...
2022-04-06 15:01:51,075 [DEBUG]  Extracted model to '/tmp/tmpwy652beu'.
2022-04-06 15:01:51,077 [DEBUG]  Node 'nlu_message_converter' loading 'NLUMessageConverter.load' and kwargs: '{}'.
2022-04-06 15:01:51,078 [DEBUG]  Node 'provide_SpacyNLP0' loading 'SpacyNLP.load' and kwargs: '{}'.
2022-04-06 15:01:51,078 [INFO ]  Trying to load SpaCy model with name 'de_core_news_md'.
2022-04-06 15:01:53,006 [DEBUG]  Node 'run_SpacyNLP0' loading 'SpacyNLP.load' and kwargs: '{}'.
2022-04-06 15:01:53,006 [INFO ]  Trying to load SpaCy model with name 'de_core_news_md'.
2022-04-06 15:01:54,494 [DEBUG]  Node 'run_SpacyTokenizer1' loading 'SpacyTokenizer.load' and kwargs: '{}'.
2022-04-06 15:01:54,494 [DEBUG]  Node 'run_SpacyFeaturizer2' loading 'SpacyFeaturizer.load' and kwargs: '{}'.
2022-04-06 15:01:54,494 [DEBUG]  Node 'run_RegexFeaturizer3' loading 'RegexFeaturizer.load' and kwargs: '{}'.
2022-04-06 15:01:54,494 [DEBUG]  Resource 'train_RegexFeaturizer3' was requested for reading.
2022-04-06 15:01:54,495 [DEBUG]  Node 'run_LexicalSyntacticFeaturizer4' loading 'LexicalSyntacticFeaturizer.load' and kwargs: '{}'.
2022-04-06 15:01:54,495 [DEBUG]  Resource 'train_LexicalSyntacticFeaturizer4' was requested for reading.
2022-04-06 15:01:54,496 [DEBUG]  Node 'run_SpacyEntityExtractor5' loading 'SpacyEntityExtractor.load' and kwargs: '{}'.
2022-04-06 15:01:54,496 [DEBUG]  Node 'run_CountVectorsFeaturizer6' loading 'CountVectorsFeaturizer.load' and kwargs: '{}'.
2022-04-06 15:01:54,496 [DEBUG]  Resource 'train_CountVectorsFeaturizer6' was requested for reading.
2022-04-06 15:01:54,497 [DEBUG]  Node 'run_CountVectorsFeaturizer7' loading 'CountVectorsFeaturizer.load' and kwargs: '{}'.
2022-04-06 15:01:54,497 [DEBUG]  Resource 'train_CountVectorsFeaturizer7' was requested for reading.
2022-04-06 15:01:54,499 [DEBUG]  Node 'run_DIETClassifier8' loading 'DIETClassifier.load' and kwargs: '{}'.
2022-04-06 15:01:54,500 [DEBUG]  Resource 'train_DIETClassifier8' was requested for reading.
2022-04-06 15:01:54,501 [DEBUG]  Loading the model from /tmp/tmp1flq9vq9/train_DIETClassifier8/DIETClassifier.tf_model with finetune_mode=False...
2022-04-06 15:01:54,561 [DEBUG]  Following metrics will be logged during training: 
2022-04-06 15:01:54,561 [DEBUG]    t_loss (total loss)
2022-04-06 15:01:54,561 [DEBUG]    i_acc (intent acc)
2022-04-06 15:01:54,561 [DEBUG]    i_loss (intent loss)
2022-04-06 15:01:54,561 [DEBUG]    e_f1 (entity f1)
2022-04-06 15:01:54,561 [DEBUG]    e_loss (entity loss)
2022-04-06 15:02:14,117 [DEBUG]  Finished loading the model.
2022-04-06 15:02:14,118 [DEBUG]  Node 'run_EntitySynonymMapper9' loading 'EntitySynonymMapper.load' and kwargs: '{}'.
2022-04-06 15:02:14,118 [DEBUG]  Resource 'train_EntitySynonymMapper9' was requested for reading.
2022-04-06 15:02:14,118 [DEBUG]  Failed to load ABCMeta from model storage. Resource 'train_EntitySynonymMapper9' doesn't exist.
2022-04-06 15:02:14,118 [DEBUG]  Node 'run_ResponseSelector10' loading 'ResponseSelector.load' and kwargs: '{}'.
2022-04-06 15:02:14,118 [DEBUG]  Resource 'train_ResponseSelector10' was requested for reading.
2022-04-06 15:02:14,118 [DEBUG]  Failed to load ABCMeta from model storage. Resource 'train_ResponseSelector10' doesn't exist.
2022-04-06 15:02:14,119 [DEBUG]  Resource 'train_ResponseSelector10' was requested for reading.
2022-04-06 15:02:14,119 [DEBUG]  Failed to load ResponseSelector from model storage. Resource 'train_ResponseSelector10' doesn't exist.
2022-04-06 15:02:14,119 [DEBUG]  Node 'run_FallbackClassifier11' loading 'FallbackClassifier.load' and kwargs: '{}'.
2022-04-06 15:02:14,119 [DEBUG]  Node 'run_RegexMessageHandler' loading 'RegexMessageHandler.load' and kwargs: '{}'.
2022-04-06 15:02:14,119 [DEBUG]  Node 'domain_provider' loading 'DomainProvider.load' and kwargs: '{}'.
2022-04-06 15:02:14,119 [DEBUG]  Resource 'domain_provider' was requested for reading.
2022-04-06 15:02:14,521 [DEBUG]  Node 'run_RulePolicy0' loading 'RulePolicy.load' and kwargs: '{}'.
2022-04-06 15:02:14,521 [DEBUG]  Resource 'train_RulePolicy0' was requested for reading.
2022-04-06 15:02:14,522 [DEBUG]  Node 'run_MemoizationPolicy1' loading 'MemoizationPolicy.load' and kwargs: '{}'.
2022-04-06 15:02:14,522 [DEBUG]  Resource 'train_MemoizationPolicy1' was requested for reading.
2022-04-06 15:02:14,522 [DEBUG]  Node 'run_TEDPolicy2' loading 'TEDPolicy.load' and kwargs: '{}'.
2022-04-06 15:02:14,522 [DEBUG]  Resource 'train_TEDPolicy2' was requested for reading.
2022-04-06 15:02:14,523 [DEBUG]  Loading the model from /tmp/tmp1flq9vq9/train_TEDPolicy2/ted_policy.tf_model with finetune_mode=False...
2022-04-06 15:02:22,677 [DEBUG]  Finished loading the model.
2022-04-06 15:02:22,677 [DEBUG]  Node 'rule_only_data_provider' loading 'RuleOnlyDataProvider.load' and kwargs: '{}'.
2022-04-06 15:02:22,677 [DEBUG]  Resource 'train_RulePolicy0' was requested for reading.
2022-04-06 15:02:22,677 [DEBUG]  Node 'select_prediction' loading 'DefaultPolicyPredictionEnsemble.load' and kwargs: '{}'.
2022-04-06 15:02:22,681 [INFO ]  Rasa server is up and running.
2022-04-06 15:02:22,682 [INFO ]  Enabling coroutine debugging. Loop id 96284384.
2022-04-06 15:02:36,608 [DEBUG]  Issuing ticket for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:36,609 [DEBUG]  Acquiring lock for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:36,609 [DEBUG]  Acquired lock for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:36,609 [DEBUG]  Could not find tracker for conversation ID 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:36,609 [DEBUG]  Starting a new session for conversation ID 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:36,610 [DEBUG]  Policy prediction ended with events '[]'.
2022-04-06 15:02:36,610 [DEBUG]  Action 'action_session_start' ended with events '[<rasa.shared.core.events.SessionStarted object at 0x7f59a458e130>, ActionExecuted(action: action_listen, policy: None, confidence: None)]'.
2022-04-06 15:02:36,610 [DEBUG]  Current slot values: 
	name: 
	origin: None
	news_topic: None
	role: None
	greeted: False
	asked_well_being: False
	game_state: inactive
	from_munchen: None
	jokes_remaining: True
	personel_question1: None
	personel_question2: None
	requested_slot: None
	session_started_metadata: None
2022-04-06 15:02:36,610 [DEBUG]  Running graph with inputs: {'__message__': [<rasa.core.channels.channel.UserMessage object at 0x7f59a458e5e0>]}, targets: ['run_RegexMessageHandler'] and ExecutionContext(model_id='6a568f2ff65e4ae6baad71857b4a1eaf', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-04-06 15:02:36,610 [DEBUG]  Node 'nlu_message_converter' running 'NLUMessageConverter.convert_user_message'.
2022-04-06 15:02:36,611 [DEBUG]  Node 'provide_SpacyNLP0' running 'SpacyNLP.provide'.
2022-04-06 15:02:36,611 [DEBUG]  Node 'run_SpacyNLP0' running 'SpacyNLP.process'.
2022-04-06 15:02:36,615 [DEBUG]  Node 'run_SpacyTokenizer1' running 'SpacyTokenizer.process'.
2022-04-06 15:02:36,615 [DEBUG]  Node 'run_SpacyFeaturizer2' running 'SpacyFeaturizer.process'.
2022-04-06 15:02:36,615 [DEBUG]  Node 'run_RegexFeaturizer3' running 'RegexFeaturizer.process'.
2022-04-06 15:02:36,615 [DEBUG]  Node 'run_LexicalSyntacticFeaturizer4' running 'LexicalSyntacticFeaturizer.process'.
2022-04-06 15:02:36,616 [DEBUG]  Node 'run_SpacyEntityExtractor5' running 'SpacyEntityExtractor.process'.
2022-04-06 15:02:36,619 [DEBUG]  Node 'run_CountVectorsFeaturizer6' running 'CountVectorsFeaturizer.process'.
2022-04-06 15:02:36,620 [DEBUG]  Node 'run_CountVectorsFeaturizer7' running 'CountVectorsFeaturizer.process'.
2022-04-06 15:02:36,620 [DEBUG]  Node 'run_DIETClassifier8' running 'DIETClassifier.process'.
2022-04-06 15:02:36,847 [DEBUG]  Node 'run_EntitySynonymMapper9' running 'EntitySynonymMapper.process'.
2022-04-06 15:02:36,847 [DEBUG]  Node 'run_ResponseSelector10' running 'ResponseSelector.process'.
2022-04-06 15:02:36,848 [DEBUG]  There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2022-04-06 15:02:36,848 [DEBUG]  Adding following selector key to message property: default
2022-04-06 15:02:36,848 [DEBUG]  Node 'run_FallbackClassifier11' running 'FallbackClassifier.process'.
2022-04-06 15:02:36,848 [DEBUG]  Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-04-06 15:02:36,848 [DEBUG]  Node 'run_RegexMessageHandler' running 'RegexMessageHandler.process'.
2022-04-06 15:02:36,848 [DEBUG]  Received user message 'Hallo' with intent '{'name': 'greet', 'confidence': 0.9986380934715271}' and entities '[]'
2022-04-06 15:02:36,849 [DEBUG]  Logged UserUtterance - tracker now has 4 events.
2022-04-06 15:02:36,849 [DEBUG]  Validating extracted slots: 
2022-04-06 15:02:36,849 [DEBUG]  Default action 'action_extract_slots' was executed, resulting in 0 events: 
2022-04-06 15:02:36,850 [DEBUG]  Running graph with inputs: {'__tracker__': <rasa.shared.core.trackers.DialogueStateTracker object at 0x7f59a458e610>}, targets: ['select_prediction'] and ExecutionContext(model_id='6a568f2ff65e4ae6baad71857b4a1eaf', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-04-06 15:02:36,850 [DEBUG]  Node 'rule_only_data_provider' running 'RuleOnlyDataProvider.provide'.
2022-04-06 15:02:36,850 [DEBUG]  Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-04-06 15:02:36,850 [DEBUG]  Node 'run_MemoizationPolicy1' running 'MemoizationPolicy.predict_action_probabilities'.
2022-04-06 15:02:36,851 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user intent: greet | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:36,851 [DEBUG]  There is a memorised next action 'action_greet'
2022-04-06 15:02:36,851 [DEBUG]  Node 'run_RulePolicy0' running 'RulePolicy.predict_action_probabilities'.
2022-04-06 15:02:36,851 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user text: Hallo | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:36,851 [DEBUG]  There is no applicable rule.
2022-04-06 15:02:36,852 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user intent: greet | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:36,852 [DEBUG]  There is no applicable rule.
2022-04-06 15:02:36,852 [DEBUG]  Node 'run_TEDPolicy2' running 'TEDPolicy.predict_action_probabilities'.
2022-04-06 15:02:36,855 [DEBUG]  TED predicted 'utter_response_wellbeing_good' based on user intent.
2022-04-06 15:02:36,855 [DEBUG]  Node 'select_prediction' running 'DefaultPolicyPredictionEnsemble.combine_predictions_from_kwargs'.
2022-04-06 15:02:36,855 [DEBUG]  Made prediction using user intent.
2022-04-06 15:02:36,856 [DEBUG]  Added `DefinePrevUserUtteredFeaturization(False)` event.
2022-04-06 15:02:36,856 [DEBUG]  Predicted next action using MemoizationPolicy.
2022-04-06 15:02:36,856 [DEBUG]  Predicted next action 'action_greet' with confidence 1.00.
2022-04-06 15:02:36,856 [DEBUG]  Calling action endpoint to run action 'action_greet'.
2022-04-06 15:02:36,863 [DEBUG]  Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x7f59a45893a0>]'.
2022-04-06 15:02:36,863 [DEBUG]  Action 'action_greet' ended with events '[BotUttered('\mrk=17\\mrk=13\\mrk=50\\mrk=15\\pau=300\ \mrk=1\ Hi, \mrk=14\\mrk=10\Ich bin NÃ¤ivel! Und wie heiÃŸt Du\mrk=12\?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1649250156.8635461), <rasa.shared.core.events.SlotSet object at 0x7f59a4589dc0>]'.
2022-04-06 15:02:36,864 [DEBUG]  Current slot values: 
	name: 
	origin: None
	news_topic: None
	role: None
	greeted: True
	asked_well_being: False
	game_state: inactive
	from_munchen: None
	jokes_remaining: True
	personel_question1: None
	personel_question2: None
	requested_slot: None
	session_started_metadata: None
2022-04-06 15:02:36,864 [DEBUG]  Running graph with inputs: {'__tracker__': <rasa.shared.core.trackers.DialogueStateTracker object at 0x7f59a458e610>}, targets: ['select_prediction'] and ExecutionContext(model_id='6a568f2ff65e4ae6baad71857b4a1eaf', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-04-06 15:02:36,864 [DEBUG]  Node 'rule_only_data_provider' running 'RuleOnlyDataProvider.provide'.
2022-04-06 15:02:36,864 [DEBUG]  Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-04-06 15:02:36,864 [DEBUG]  Node 'run_MemoizationPolicy1' running 'MemoizationPolicy.predict_action_probabilities'.
2022-04-06 15:02:36,865 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user intent: greet | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 2] user intent: greet | previous action name: action_greet | slots: {'name': (1.0,), 'greeted': (1.0, 1.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:36,865 [DEBUG]  There is no memorised next action
2022-04-06 15:02:36,865 [DEBUG]  Node 'run_RulePolicy0' running 'RulePolicy.predict_action_probabilities'.
2022-04-06 15:02:36,865 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user intent: greet | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 2] user intent: greet | previous action name: action_greet | slots: {'name': (1.0,), 'greeted': (1.0, 1.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:36,865 [DEBUG]  There is no applicable rule.
2022-04-06 15:02:36,866 [DEBUG]  Node 'run_TEDPolicy2' running 'TEDPolicy.predict_action_probabilities'.
2022-04-06 15:02:36,869 [DEBUG]  TED predicted 'utter_response_wellbeing_good' based on user intent.
2022-04-06 15:02:36,869 [DEBUG]  Node 'select_prediction' running 'DefaultPolicyPredictionEnsemble.combine_predictions_from_kwargs'.
2022-04-06 15:02:36,869 [DEBUG]  Predicted next action using RulePolicy.
2022-04-06 15:02:36,869 [DEBUG]  Predicted next action 'action_default_fallback' with confidence 0.30.
2022-04-06 15:02:36,870 [DEBUG]  Policy prediction ended with events '[]'.
2022-04-06 15:02:36,870 [DEBUG]  Action 'action_default_fallback' ended with events '[<rasa.shared.core.events.UserUtteranceReverted object at 0x7f59a4591a60>]'.
2022-04-06 15:02:36,870 [DEBUG]  Current slot values: 
	name: 
	origin: None
	news_topic: None
	role: None
	greeted: False
	asked_well_being: False
	game_state: inactive
	from_munchen: None
	jokes_remaining: True
	personel_question1: None
	personel_question2: None
	requested_slot: None
	session_started_metadata: None
2022-04-06 15:02:36,870 [DEBUG]  Predicted next action 'action_listen' with confidence 1.00.
2022-04-06 15:02:36,870 [DEBUG]  Policy prediction ended with events '[]'.
2022-04-06 15:02:36,870 [DEBUG]  Action 'action_listen' ended with events '[]'.
2022-04-06 15:02:36,870 [DEBUG]  Deleted lock for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:43,110 [DEBUG]  Issuing ticket for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:43,111 [DEBUG]  Acquiring lock for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:43,111 [DEBUG]  Acquired lock for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:43,112 [DEBUG]  Recreating tracker for id 'b40af955780547aca3d8594cccd3a7d4'
2022-04-06 15:02:43,115 [DEBUG]  Running graph with inputs: {'__message__': [<rasa.core.channels.channel.UserMessage object at 0x7f59a53c4b80>]}, targets: ['run_RegexMessageHandler'] and ExecutionContext(model_id='6a568f2ff65e4ae6baad71857b4a1eaf', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-04-06 15:02:43,116 [DEBUG]  Node 'nlu_message_converter' running 'NLUMessageConverter.convert_user_message'.
2022-04-06 15:02:43,116 [DEBUG]  Node 'provide_SpacyNLP0' running 'SpacyNLP.provide'.
2022-04-06 15:02:43,117 [DEBUG]  Node 'run_SpacyNLP0' running 'SpacyNLP.process'.
2022-04-06 15:02:43,129 [DEBUG]  Node 'run_SpacyTokenizer1' running 'SpacyTokenizer.process'.
2022-04-06 15:02:43,129 [DEBUG]  Node 'run_SpacyFeaturizer2' running 'SpacyFeaturizer.process'.
2022-04-06 15:02:43,129 [DEBUG]  Node 'run_RegexFeaturizer3' running 'RegexFeaturizer.process'.
2022-04-06 15:02:43,130 [DEBUG]  Node 'run_LexicalSyntacticFeaturizer4' running 'LexicalSyntacticFeaturizer.process'.
2022-04-06 15:02:43,130 [DEBUG]  Node 'run_SpacyEntityExtractor5' running 'SpacyEntityExtractor.process'.
2022-04-06 15:02:43,136 [DEBUG]  Node 'run_CountVectorsFeaturizer6' running 'CountVectorsFeaturizer.process'.
2022-04-06 15:02:43,137 [DEBUG]  Node 'run_CountVectorsFeaturizer7' running 'CountVectorsFeaturizer.process'.
2022-04-06 15:02:43,138 [DEBUG]  Node 'run_DIETClassifier8' running 'DIETClassifier.process'.
2022-04-06 15:02:43,144 [DEBUG]  Node 'run_EntitySynonymMapper9' running 'EntitySynonymMapper.process'.
2022-04-06 15:02:43,145 [DEBUG]  Node 'run_ResponseSelector10' running 'ResponseSelector.process'.
2022-04-06 15:02:43,145 [DEBUG]  There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2022-04-06 15:02:43,145 [DEBUG]  Adding following selector key to message property: default
2022-04-06 15:02:43,145 [DEBUG]  Node 'run_FallbackClassifier11' running 'FallbackClassifier.process'.
2022-04-06 15:02:43,145 [DEBUG]  Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-04-06 15:02:43,145 [DEBUG]  Node 'run_RegexMessageHandler' running 'RegexMessageHandler.process'.
2022-04-06 15:02:43,146 [DEBUG]  Received user message 'Thomas' with intent '{'name': 'inform_name', 'confidence': 0.9986963868141174}' and entities '[{'entity': 'name', 'start': 0, 'end': 6, 'confidence_entity': 0.9884451627731323, 'value': 'Thomas', 'extractor': 'DIETClassifier'}]'
2022-04-06 15:02:43,146 [DEBUG]  Current slot values: 
	name: 
	origin: None
	news_topic: None
	role: None
	greeted: False
	asked_well_being: False
	game_state: inactive
	from_munchen: None
	jokes_remaining: True
	personel_question1: None
	personel_question2: None
	requested_slot: None
	session_started_metadata: None
2022-04-06 15:02:43,146 [DEBUG]  Logged UserUtterance - tracker now has 12 events.
2022-04-06 15:02:43,147 [DEBUG]  Validating extracted slots: name
2022-04-06 15:02:43,147 [DEBUG]  Default action 'action_extract_slots' was executed, resulting in 1 events: SlotSet(key: name, value: Thomas)
2022-04-06 15:02:43,147 [DEBUG]  Running graph with inputs: {'__tracker__': <rasa.shared.core.trackers.DialogueStateTracker object at 0x7f59a53c4a90>}, targets: ['select_prediction'] and ExecutionContext(model_id='6a568f2ff65e4ae6baad71857b4a1eaf', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-04-06 15:02:43,147 [DEBUG]  Node 'rule_only_data_provider' running 'RuleOnlyDataProvider.provide'.
2022-04-06 15:02:43,147 [DEBUG]  Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-04-06 15:02:43,147 [DEBUG]  Node 'run_MemoizationPolicy1' running 'MemoizationPolicy.predict_action_probabilities'.
2022-04-06 15:02:43,148 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user intent: inform_name | user entities: ('name',) | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:43,148 [DEBUG]  There is no memorised next action
2022-04-06 15:02:43,148 [DEBUG]  Node 'run_RulePolicy0' running 'RulePolicy.predict_action_probabilities'.
2022-04-06 15:02:43,148 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user text: Thomas | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:43,148 [DEBUG]  There is no applicable rule.
2022-04-06 15:02:43,149 [DEBUG]  Current tracker state:
[state 0] slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
[state 1] user intent: inform_name | user entities: ('name',) | previous action name: action_listen | slots: {'name': (1.0,), 'greeted': (1.0, 0.0), 'asked_well_being': (1.0, 0.0), 'game_state': (1.0, 0.0, 0.0, 0.0, 0.0, 0.0), 'jokes_remaining': (1.0, 1.0)}
2022-04-06 15:02:43,149 [DEBUG]  There is no applicable rule.
2022-04-06 15:02:43,149 [DEBUG]  Node 'run_TEDPolicy2' running 'TEDPolicy.predict_action_probabilities'.
2022-04-06 15:02:43,153 [DEBUG]  TED predicted 'action_check_name' based on user intent.
2022-04-06 15:02:43,153 [DEBUG]  Node 'select_prediction' running 'DefaultPolicyPredictionEnsemble.combine_predictions_from_kwargs'.
2022-04-06 15:02:43,153 [DEBUG]  Made prediction using user intent.
2022-04-06 15:02:43,153 [DEBUG]  Added `DefinePrevUserUtteredFeaturization(False)` event.
2022-04-06 15:02:43,154 [DEBUG]  Predicted next action using RulePolicy.
2022-04-06 15:02:43,154 [DEBUG]  Predicted next action 'action_default_fallback' with confidence 0.30.
2022-04-06 15:02:43,154 [DEBUG]  Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x7f59a5455520>]'.
2022-04-06 15:02:43,154 [DEBUG]  Action 'action_default_fallback' ended with events '[<rasa.shared.core.events.UserUtteranceReverted object at 0x7f59a45892e0>]'.
2022-04-06 15:02:43,154 [DEBUG]  Current slot values: 
	name: 
	origin: None
	news_topic: None
	role: None
	greeted: False
	asked_well_being: False
	game_state: inactive
	from_munchen: None
	jokes_remaining: True
	personel_question1: None
	personel_question2: None
	requested_slot: None
	session_started_metadata: None
2022-04-06 15:02:43,154 [DEBUG]  Predicted next action 'action_listen' with confidence 1.00.
2022-04-06 15:02:43,155 [DEBUG]  Policy prediction ended with events '[]'.
2022-04-06 15:02:43,155 [DEBUG]  Action 'action_listen' ended with events '[]'.
2022-04-06 15:02:43,155 [DEBUG]  Deleted lock for conversation 'b40af955780547aca3d8594cccd3a7d4'.
2022-04-06 15:02:47,425 [INFO ]  Killing Sanic server now.
type or paste code here

andres_pineiro · May 5, 2022, 12:23pm

Hi @Claude, are you using slots with influence_conversation: true? If so, are you setting the slots from a custom action?

Claude · May 5, 2022, 2:47pm

Hi @andres_pineiro, thank you for your response. I have tried various things and now seem to have found the problem. In fact, I have several slots with ‘influence_conversation: true’ and which are set via a custom action. But apparently only one of these slots has caused problems. Namely a categorical slot that had an ‘initial_value’ assigned to it. When I removed the ‘initial_value’ and defined it via a custom action, the rules worked again. Is this a bug or does it make sense? Are the rules trained with the initial_values of the slots?

andres_pineiro · May 6, 2022, 8:59am

Hi @Claude, I came to exactly the same conclusion.

My problem was very similar: my rules stopped working the moment I set a slot that influenced the conversation. Until then everything works fine.

In this post Felicia commented that if you use slots that influence the conversation, it is necessary to define rules with conditions for each of the possible cases. However, this solution does not seem very scalable, since as the number of slots grows, or as you use non-boolean slots, handling the rules with these conditions becomes impossible.

As you comment, omitting the initial value seems to make the problem go away. I don’t know if this is a bug or if it follows a logical behavior, but I think this part is underdocumented in rasa.

Thanks for the reply. I’m going to ask Felicia about it and I will update the post if I find more information.

jule · May 6, 2022, 12:07pm

Hi Claude, hi @andres_pineiro,

Are you using rules and stories or just rules for your bot?

Claude · May 6, 2022, 1:13pm

Hi @jule , i am using both stories and rules.

andres_pineiro · May 9, 2022, 5:59am

Hi @jule I’m also using both.

jule · May 17, 2022, 1:43pm

Hi, I did some research. Hope this two links can help you

github.com/RasaHQ/rasa

Rule policies will not fire if unrelated featurized slots are set

opened 10:05AM - 09 Aug 21 UTC

closed 09:50AM - 10 Sep 21 UTC

TyDunn

type:bug

area:rasa-oss

area:rasa-oss/ml/policies cse-issues

_Product feedback submission from @rgstephens_ **Who is the feedback from?** … Multiple customers **What is the feedback?** Rule policies will not fire if unrelated featurized slots are set. This behavior is counterintuitive and not documented. **What problem are they trying to solve?** A simple rule like this should work if a featurized slot is set: ```yml - rule: greet steps: - intent: greet - action: utter_greet ``` [Repo with more details](https://github.com/rgstephens/rasa-rule-slot-issue) [Slack thread](https://rasa-hq.slack.com/archives/C0148JNFXLG/p1628271530006800) **Recommendation** - Assign two researcher to this issue **Possible Inelegant Workaround** - Add a rule for every possible slot state **Possible Solutions** - Make the state of slot featurization explicit in the rules (e.g. none or any) **Definition of Done** In order of urgency: - [ ] See whether suggested workaround will resolve immediate customer problem - [ ] Understand customer context: who does this affect? - [ ] Identify consequences: what would change if this suggested behavior change were implemented? - [ ] Identify possible alternative next steps to be refined and scoped later, likely to include more explicit definition of rule behavior Instead of the above steps, @samsucik carried out an investigation to pin-point the bug and, while doing so, proposed a simple fix (implemented in #9475), which has been verified to fix the issue on all examples provided by @rgstephens. The discussion inspired [related improvements to our docs](https://github.com/RasaHQ/rasa/issues/9521) and [another bug report](https://github.com/RasaHQ/rasa/issues/9483). @samsucik to coordinate who is doing what on this issue @kedz assigned reviewer

github.com/RasaHQ/rasa

Featurised slots with `initial_value` get wrongly embedded in general rules

opened 10:43AM - 31 Aug 21 UTC

samsucik

area:rasa-oss

priority:high area:rasa-oss/ml/policies effort:research/4

### Rasa Open Source version 2.8.3 ### Rasa SDK version _No response_ ### …Rasa X version _No response_ ### Python version 3.8 ### What operating system are you using? OSX ### What happened? We encountered this issue a long time ago and thought we fixed it in #8161\. But looks like we didn't. If a featurised slot has an `initial_value` set, the slot gets included in rules that don't mention this slot explicitly. This is incorrect as those rules are then not applicable in states where the slot has a value different from the initial one. **Findings from initial investigation** _See bellow for an example setup to reproduce this issue._ In #8161, we introduced the `omit_unset_slots` flag, which, when training `RulePolicy`, [is used](https://github.com/RasaHQ/rasa/blob/0d97d427fd342b3f0fab4eaabc3d5169249dbf61/rasa/core/policies/rule_policy.py#L734) to exclude slots that are unset (i.e. set to their initial values) from general rules (i.e. from rules that don't mention those slots and should be applicable regardless of those slots' values). Unfortunately, the intended pattern with `omit_unset_slots` needs further work (and testing). What is going wrong is: 1. During data generation from rules, the generator [removes duplicated trackers](https://github.com/RasaHQ/rasa/blob/0d97d427fd342b3f0fab4eaabc3d5169249dbf61/rasa/shared/core/generator.py#L394) – deduplication is done by [creating tracker states (and caching them)](https://github.com/RasaHQ/rasa/blob/0d97d427fd342b3f0fab4eaabc3d5169249dbf61/rasa/shared/core/generator.py#L108) and subsequently hashing them. In this step, `omit_unset_slots` is switched off. 2. Subsequently, `RulePolicy` trains – [creates its rule lookup](https://github.com/RasaHQ/rasa/blob/0d97d427fd342b3f0fab4eaabc3d5169249dbf61/rasa/core/policies/rule_policy.py#L794), with `omit_unset_slots` switched on. In doing this, trackers need to be turned into states. And, since we have the states cached from the previous step, we [re-use them](https://github.com/RasaHQ/rasa/blob/0d97d427fd342b3f0fab4eaabc3d5169249dbf61/rasa/shared/core/generator.py#L106). And this is where things break – the previously created states incorrectly contain initial values of featurised slots because they were created with `omit_unset_slots=False`. I see a few options for possible fixes: * see if it makes sense to use `omit_unset_slots=True` also during tracker deduplication * don't use the caching * use caching but check that the cached states were created with the same value of `omit_unset_slots` * ... **Example setup to reproduce the issue** Command: <pre>rasa train --augmentation 0 rasa test</pre> The default config. Training data: <pre>version: "2.0" rules: * rule: goodbye steps: * intent: goodbye * action: utter_goodbye nlu: * intent: greet examples: | * hey * hello * hi * hello there * good morning * good evening * moin * hey there * let's go * hey dude * goodmorning * goodevening * good afternoon * intent: goodbye examples: | * cu * good by * cee you later * good night * bye * goodbye * have a nice day * see you around * bye bye * see you later</pre> Test stories: <pre>version: "2.0" stories: * story: goodbye - num_appts slot steps: * slot_was_set: * num_appts: multiple * intent: goodbye * action: utter_goodbye</pre> Domain: <pre>version: "2.0" intents: * greet * goodbye slots: num_appts: type: categorical initial_value: single values: * none * single responses: utter_goodbye: * text: "Bye"</pre> **Definition of done** * <span class="error">[x]</span> Verify that the cause of the issue is as outlined above * <span class="error">[x]</span> Write tests that catch the bug (turns out the tests in #8161 weren't sufficient) * [ ] Decide which way to fix it (involve CSE and Product Management) * [ ] Create followup issue: "Implement the fix, adding more tests if necessary"

andres_pineiro · May 18, 2022, 7:11am

Hi @jule, thank you for your research. Both links help indeed. We now confirm that it is a bug and it is still present on Rasa 3.x

Topic		Replies	Views
Why are my rules not being applied? Rasa Open Source	10	1761	September 10, 2021
Stories are not working if there is Rule Policy in the Config file Rasa Open Source	10	1151	September 19, 2021
Problem with rules Rasa Open Source	4	568	February 10, 2021
Why Rasa/policies are inconsistent? I need some clarity on the below questions Rasa Open Source	1	184	February 13, 2023
Trouble with 'There is no memorized next action' message Rasa Open Source	10	5894	March 19, 2021

Rules are only recognized within the first turns

Related topics