Right intent but wrong response with MemoizationPolicy

joacko · March 20, 2020, 5:48pm

Hi! I started having a problem after I upgraded Rasa from 0.13 to 1.8, with python 3.7 After I train a chatbot, the intents are correctly identified but the utter yhat is triggered is wrong.

Here are the files I Use to train. the domain and the stories are generated programmatically and the agent is trained by code, not using the cli (because my service creates multiple chatbots): domain.yml (753 Bytes) nlu_config.yml (874 Bytes) stories.md (270 Bytes)

And here is the log when I send ‘/greet’:

  >    [2020-03-20 14:36:39,447: DEBUG/ForkPoolWorker-1] Received user message '/greet' with intent '{'name': 'greet', 'confidence': 1.0}' and entities '[]'
   > [2020-03-20 14:36:39,448: DEBUG/ForkPoolWorker-1] Logged UserUtterance - tracker now has 18 events.
   > [2020-03-20 14:36:39,448: DEBUG/ForkPoolWorker-1] Current tracker state [None, None, None, {}, {'prev_action_listen': 1.0, 'intent_greet': 1.0}]
   > [2020-03-20 14:36:39,448: DEBUG/ForkPoolWorker-1] There is a memorised next action '13'
   > 2020-03-20 14:36:39.469156: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
   > [2020-03-20 14:36:39,471: INFO/ForkPoolWorker-1] Redis value for key company_opp_step/joacko12 got => b'0'
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] Predicted next action using policy_0_MemoizationPolicy
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] Predicted next action 'utter_6Ae5FyMuh' with confidence 1.00.
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] Action 'utter_6Ae5FyMuh' ended with events '[BotUttered('chau', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1584725799.471813)]'.
   > [2020-03-20 14:36:39,472: DEBUG/ForkPoolWorker-1] Current tracker state [None, None, {}, {'prev_action_listen': 1.0, 'intent_greet': 1.0}, {'prev_utter_6Ae5FyMuh': 1.0, 'intent_greet': 1.0}]
   > [2020-03-20 14:36:39,472: DEBUG/ForkPoolWorker-1] There is no memorised next action
   > 2020-03-20 14:36:39.485910: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] There is no mapped action for the predicted intent, 'greet'.
   > [2020-03-20 14:36:39,487: INFO/ForkPoolWorker-1] Redis value for key company_opp_step/joacko12 got => b'0'
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] Predicted next action using policy_3_NFallbackPolicy
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] Predicted next action 'action_fallback' with confidence 0.30.
   > [2020-03-20 14:36:39,488: DEBUG/ForkPoolWorker-1] Calling action endpoint to run action 'action_fallback'.
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Action 'action_fallback' ended with events '[BotUttered('Perdon, no entendi :( Podrias repetirlo?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1584725799.5003626), <rasa.core.events.UserUtteranceReverted object at 0x7f3e80484290>]'.
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Current slot values: 
   > 	answer_AYOvUwPR: None
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Predicted next action 'action_listen' with confidence 1.00.
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Action 'action_listen' ended with events '[]'.
   > [2020-03-20 14:36:39,501: DEBUG/ForkPoolWorker-1] Deleted lock for conversation 'company_opp_step/joacko12'.
   > [2020-03-20 14:36:39,502: INFO/ForkPoolWorker-1] response: [{'recipient_id': 'company_opp_step/joacko12', 'text': 'chau'}, {'recipient_id': 'company_opp_step/joacko12', 'text': 'Perdon, no entendi :( Podrias repetirlo?'}]
   > [2020-03-20 14:36:39,502: INFO/ForkPoolWorker-1] Message: chau
   > Perdon, no entendi :( Podrias repetirlo?

Here is the code used to create the chatbots:


domain = Domain.from_yaml(chatbot_data['domain'])
tracker_store = MongoTrackerStore(domain, ........)
policies = core_config.load('nlu_config.yml')
agent = Agent(domain, policies=policies, interpreter=interpreter, action_endpoint=endpoint, tracker_store=tracker_store)
tmp_file, path = tempfile.mkstemp()
with open(tmp_file, 'w') as stories_file:
    stories_file.write(chatbot_data['stories'])
    stories_file.seek(0)
    training_data = asyncio.run(agent.load_data(path, augmentation_factor=0))
    stories_file.close()
LOG.info('training core agent')
agent.train(training_data, validation_split=0.1)

But, If I train an agent with the cli and the same files, It predicts right:

2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Received user message '/greet' with intent '{'name': 'greet', 'confidence': 1.0}' and entities '[]'
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'intent_greet': 1.0, 'prev_action_listen': 1.0}]
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '15'
2020-03-20 11:55:27 INFO     n_fallback  - Redis value for key bcad3cecceb34aa0b3b3ba67766d8c9d got => None
2020-03-20 11:55:27 DEBUG    n_fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-03-20 11:55:27 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_MemoizationPolicy
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Predicted next action 'utter_AYOvUwPR' with confidence 1.00.
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Action 'utter_AYOvUwPR' ended with events '[BotUttered('Mensaje inicial del chatbot', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1584716127.1188364)]'.
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, {}, {'intent_greet': 1.0, 'prev_action_listen': 1.0}, {'prev_utter_AYOvUwPR': 1.0, 'intent_greet': 1.0}]
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '0'
2020-03-20 11:55:27 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'greet'.
2020-03-20 11:55:27 INFO     n_fallback  - Redis value for key bcad3cecceb34aa0b3b3ba67766d8c9d got => None
2020-03-20 11:55:27 DEBUG    n_fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-03-20 11:55:27 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_MemoizationPolicy
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-03-20 11:55:27 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'bcad3cecceb34aa0b3b3ba67766d8c9d'.
Mensaje inicial del chatbot
Your input ->

Any ideas about what I’m doing wrong? Thanks in advance!!

saurabh-m523 · March 20, 2020, 7:32pm

Seems like an issue with the stories, the actions are not properly indented. Try proper indenting and train again.

joacko · March 24, 2020, 2:29pm

Hi! Thanks for responding!

I tried a different indentation with 4 spaces before the - as in rasa/stories.md at 50be45f62816733562c2933f8eb9d27c0090b0c0 · RasaHQ/rasa · GitHub and also tried with 3, 2 and 1 spaces, and the result was always the same

My stories are generated programmatically, so the output of the generator is something like this: ## story_0\n* greet\n - utter_AYOvUwPR\n* inform\n - action_choice\n - slot{"answer_AYOvUwPR":"izquierda"}\n - utter_6Mttj3QF\n - action_tag\n\n\n\n## story_1\n* greet\n - utter_AYOvUwPR\n* inform\n - action_choice\n - slot{"answer_AYOvUwPR":"derecha"}\n - utter_6Ae5FyMuh\n - action_tag\n (with 2 spaces indentation in this example)

joacko · March 25, 2020, 7:35pm

Please if someone could give me a hint about where to look or what to change, it would mean a lot to me I’ve been struggling with that for over a week… Thanks!

joacko · March 27, 2020, 1:06pm

I resolved the Issue loading the domain with Domain.load instead of from_yaml. It seems that the from_yaml method leads to a missorder in the action_names, so the index for the labeling in memoization training is not correct.

kurtsenol · October 4, 2020, 9:22am

I experience the same issue and struggling more than a week, but couldn’t understand your solution. Can you please explain your solution? In which file did you use Domain.load? Can you share your Github?

Topic		Replies	Views
Rasa bot is not predicting the correct intents Rasa Open Source	11	739	September 23, 2020
Utter_default triggered despite nlu intent recognition is really high Rasa Open Source	18	767	September 14, 2023
Getting same answer from bot for user texts that are not in intents Rasa Open Source	1	548	October 30, 2019
Rasa producing wrong (random) actions with correctly identified intents Rasa Open Source	4	1005	February 25, 2020
Incorrect RESPONSE Rasa Open Source	2	650	December 10, 2019

Right intent but wrong response with MemoizationPolicy

Related Topics