Hello everyone,
I have some questions. I am trying to build an agent to handle very similar conversations. I am pasting below two stories:
- story: Agent receives positive sentiment and does a short chitchat with the patient
steps:
- intent: greet
entities:
- sentiment: pos
- slot_was_set:
- sentiment: pos
- action: utter_greet
- action: utter_how_do_you_feel_today
- intent: mood_happy
entities:
- sentiment: pos
- slot_was_set:
- sentiment: pos
- action: action_utter_nice_to_hear_that
- intent: inform
entities:
- sentiment: pos
- slot_was_set:
- sentiment: pos
- action: utter_nice_to_hear_that
- action: utter_inform_doctor
- intent: affirm
entities:
- sentiment: pos
- slot_was_set:
- sentiment: pos
- action: utter_confirm_doctor
- intent: thank_you
entities:
- sentiment: neu
- slot_was_set:
- sentiment: neu
- action: utter_you_are_welcome
- story: Agent handles negative sentiment and does a short chitchat with the patient.
steps:
- intent: greet
entities:
- sentiment: neu
- slot_was_set:
- sentiment: neu
- action: utter_greet
- action: utter_how_do_you_feel_today
- intent: mood_happy
entities:
- sentiment: pos
- slot_was_set:
- sentiment: pos
- action: action_utter_nice_to_hear_that
- intent: affirm
entities:
- sentiment: neu
- slot_was_set:
- sentiment: neu
- action: utter_please_go_on
- intent: inform
entities:
- sentiment: neg
- slot_was_set:
- sentiment: neg
- action: utter_handle_negative_sentiment
- intent: inform
- action: utter_inform_doctor
- intent: affirm
entities:
- sentiment: neu
- slot_was_set:
- sentiment: neu
- action: utter_confirm_doctor
- intent: thank_you
entities:
- sentiment: neu
- slot_was_set:
- sentiment: neu
- action: utter_you_are_welcome
and here is my config:
# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
# # If you'd like to customize it, uncomment and adjust the pipeline.
# # See https://rasa.com/docs/rasa/tuning-your-model for more information.
- name: sentiment.SentimentAnalyzer
- name: WhitespaceTokenizer
- name: RegexFeaturizer
case_sensitive: False
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
use_lemma: False
- name: CountVectorsFeaturizer
analyzer: char_wb
min_ngram: 1
max_ngram: 4
- name: DIETClassifier
epochs: 200
constrain_similarities: true
- name: EntitySynonymMapper
- name: FallbackClassifier
threshold: 0.25
# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
# # No configuration for policies was provided. The following default policies were used to train your model.
# # If you'd like to customize them, uncomment and adjust the policies.
# # See https://rasa.com/docs/rasa/policies for more information.
- name: MemoizationPolicy
max_history: 8
- name: TEDPolicy
max_history: 6
epochs: 150
constrain_similarities: true
- name: UnexpecTEDIntentPolicy
max_history: 6
epochs: 100
- name: RulePolicy
# Confidence threshold for the `core_fallback_action_name` to apply.
# The action will apply if no other action was predicted with
# a confidence >= core_fallback_threshold
core_fallback_threshold: 0.2
core_fallback_action_name: "action_default_fallback"
enable_fallback_prediction: True
The agent is not able to follow these stories even though they are training examples. If I have understood correctly the TED
policy does exactly this, follow the training examples, or not? I don’t know how to solve this issue. I have tried different epochs and different configurations.
Any ideas or suggestions?
Extra information might help:
Rasa Version : 2.8.3
Minimum Compatible Version: 2.8.0
Rasa SDK Version : 2.8.1
Rasa X Version : None
Python Version : 3.8.11
UPDATE I tried to make even simpler my stories in a different branch:
- story: Agent receives positive sentiment, user agrees to inform doctors
steps:
- intent: greet
- action: utter_greet
- intent: inform
entities:
- sentiment
- slot_was_set:
- sentiment: pos
- action: utter_inform_doctor
- intent: affirm
- action: utter_confirm_doctor
- story: Agent receives positive sentiment, user doesn't agree to inform doctors.
steps:
- intent: greet
- action: utter_greet
- intent: inform
entities:
- sentiment
- slot_was_set:
- sentiment: pos
- action: utter_inform_doctor
- intent: deny
- action: utter_handle_user_deny_to_inform_doctors
- intent: deny
- action: utter_no_problem
When I run rasa shell
after intent:greet
the agent predicts the wrong answer action: utter_inform_doctor
. If I comment out this utterance, train again then the agent doesn’t predict utter_greet
instead it does nothing.
UPDATE
I am not sure why but the pipeline seems to have better accuracy in predicting the next action if it is a custom action versus a simple utterance. What is the explanation for this?