Right intent but wrong response with MemoizationPolicy

joacko · March 20, 2020, 5:48pm

Hi! I started having a problem after I upgraded Rasa from 0.13 to 1.8, with python 3.7 After I train a chatbot, the intents are correctly identified but the utter yhat is triggered is wrong.

Here are the files I Use to train. the domain and the stories are generated programmatically and the agent is trained by code, not using the cli (because my service creates multiple chatbots): domain.yml (753 Bytes) nlu_config.yml (874 Bytes) stories.md (270 Bytes)

And here is the log when I send ‘/greet’:

  >    [2020-03-20 14:36:39,447: DEBUG/ForkPoolWorker-1] Received user message '/greet' with intent '{'name': 'greet', 'confidence': 1.0}' and entities '[]'
   > [2020-03-20 14:36:39,448: DEBUG/ForkPoolWorker-1] Logged UserUtterance - tracker now has 18 events.
   > [2020-03-20 14:36:39,448: DEBUG/ForkPoolWorker-1] Current tracker state [None, None, None, {}, {'prev_action_listen': 1.0, 'intent_greet': 1.0}]
   > [2020-03-20 14:36:39,448: DEBUG/ForkPoolWorker-1] There is a memorised next action '13'
   > 2020-03-20 14:36:39.469156: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
   > [2020-03-20 14:36:39,471: INFO/ForkPoolWorker-1] Redis value for key company_opp_step/joacko12 got => b'0'
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] Predicted next action using policy_0_MemoizationPolicy
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] Predicted next action 'utter_6Ae5FyMuh' with confidence 1.00.
   > [2020-03-20 14:36:39,471: DEBUG/ForkPoolWorker-1] Action 'utter_6Ae5FyMuh' ended with events '[BotUttered('chau', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1584725799.471813)]'.
   > [2020-03-20 14:36:39,472: DEBUG/ForkPoolWorker-1] Current tracker state [None, None, {}, {'prev_action_listen': 1.0, 'intent_greet': 1.0}, {'prev_utter_6Ae5FyMuh': 1.0, 'intent_greet': 1.0}]
   > [2020-03-20 14:36:39,472: DEBUG/ForkPoolWorker-1] There is no memorised next action
   > 2020-03-20 14:36:39.485910: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] There is no mapped action for the predicted intent, 'greet'.
   > [2020-03-20 14:36:39,487: INFO/ForkPoolWorker-1] Redis value for key company_opp_step/joacko12 got => b'0'
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] Predicted next action using policy_3_NFallbackPolicy
   > [2020-03-20 14:36:39,487: DEBUG/ForkPoolWorker-1] Predicted next action 'action_fallback' with confidence 0.30.
   > [2020-03-20 14:36:39,488: DEBUG/ForkPoolWorker-1] Calling action endpoint to run action 'action_fallback'.
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Action 'action_fallback' ended with events '[BotUttered('Perdon, no entendi :( Podrias repetirlo?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1584725799.5003626), <rasa.core.events.UserUtteranceReverted object at 0x7f3e80484290>]'.
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Current slot values: 
   > 	answer_AYOvUwPR: None
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Predicted next action 'action_listen' with confidence 1.00.
   > [2020-03-20 14:36:39,500: DEBUG/ForkPoolWorker-1] Action 'action_listen' ended with events '[]'.
   > [2020-03-20 14:36:39,501: DEBUG/ForkPoolWorker-1] Deleted lock for conversation 'company_opp_step/joacko12'.
   > [2020-03-20 14:36:39,502: INFO/ForkPoolWorker-1] response: [{'recipient_id': 'company_opp_step/joacko12', 'text': 'chau'}, {'recipient_id': 'company_opp_step/joacko12', 'text': 'Perdon, no entendi :( Podrias repetirlo?'}]
   > [2020-03-20 14:36:39,502: INFO/ForkPoolWorker-1] Message: chau
   > Perdon, no entendi :( Podrias repetirlo?

Here is the code used to create the chatbots:


domain = Domain.from_yaml(chatbot_data['domain'])
tracker_store = MongoTrackerStore(domain, ........)
policies = core_config.load('nlu_config.yml')
agent = Agent(domain, policies=policies, interpreter=interpreter, action_endpoint=endpoint, tracker_store=tracker_store)
tmp_file, path = tempfile.mkstemp()
with open(tmp_file, 'w') as stories_file:
    stories_file.write(chatbot_data['stories'])
    stories_file.seek(0)
    training_data = asyncio.run(agent.load_data(path, augmentation_factor=0))
    stories_file.close()
LOG.info('training core agent')
agent.train(training_data, validation_split=0.1)

But, If I train an agent with the cli and the same files, It predicts right:

2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Received user message '/greet' with intent '{'name': 'greet', 'confidence': 1.0}' and entities '[]'
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, None, {}, {'intent_greet': 1.0, 'prev_action_listen': 1.0}]
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '15'
2020-03-20 11:55:27 INFO     n_fallback  - Redis value for key bcad3cecceb34aa0b3b3ba67766d8c9d got => None
2020-03-20 11:55:27 DEBUG    n_fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-03-20 11:55:27 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_MemoizationPolicy
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Predicted next action 'utter_AYOvUwPR' with confidence 1.00.
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Action 'utter_AYOvUwPR' ended with events '[BotUttered('Mensaje inicial del chatbot', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {}, 1584716127.1188364)]'.
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - Current tracker state [None, None, {}, {'intent_greet': 1.0, 'prev_action_listen': 1.0}, {'prev_utter_AYOvUwPR': 1.0, 'intent_greet': 1.0}]
2020-03-20 11:55:27 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '0'
2020-03-20 11:55:27 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'greet'.
2020-03-20 11:55:27 INFO     n_fallback  - Redis value for key bcad3cecceb34aa0b3b3ba67766d8c9d got => None
2020-03-20 11:55:27 DEBUG    n_fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.3).
2020-03-20 11:55:27 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_MemoizationPolicy
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-03-20 11:55:27 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2020-03-20 11:55:27 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'bcad3cecceb34aa0b3b3ba67766d8c9d'.
Mensaje inicial del chatbot
Your input ->

Any ideas about what I’m doing wrong? Thanks in advance!!

saurabh-m523 · March 20, 2020, 7:32pm

Seems like an issue with the stories, the actions are not properly indented. Try proper indenting and train again.

joacko · March 24, 2020, 2:29pm

Hi! Thanks for responding!

I tried a different indentation with 4 spaces before the - as in rasa/stories.md at 50be45f62816733562c2933f8eb9d27c0090b0c0 · RasaHQ/rasa · GitHub and also tried with 3, 2 and 1 spaces, and the result was always the same

My stories are generated programmatically, so the output of the generator is something like this: ## story_0\n* greet\n - utter_AYOvUwPR\n* inform\n - action_choice\n - slot{"answer_AYOvUwPR":"izquierda"}\n - utter_6Mttj3QF\n - action_tag\n\n\n\n## story_1\n* greet\n - utter_AYOvUwPR\n* inform\n - action_choice\n - slot{"answer_AYOvUwPR":"derecha"}\n - utter_6Ae5FyMuh\n - action_tag\n (with 2 spaces indentation in this example)

joacko · March 25, 2020, 7:35pm

Please if someone could give me a hint about where to look or what to change, it would mean a lot to me I’ve been struggling with that for over a week… Thanks!

joacko · March 27, 2020, 1:06pm

I resolved the Issue loading the domain with Domain.load instead of from_yaml. It seems that the from_yaml method leads to a missorder in the action_names, so the index for the labeling in memoization training is not correct.

kurtsenol · October 4, 2020, 9:22am

I experience the same issue and struggling more than a week, but couldn’t understand your solution. Can you please explain your solution? In which file did you use Domain.load? Can you share your Github?