Rasa fails on simple story

Rasa Open Source
1

Hello everyone,

I have some questions. I am trying to build an agent to handle very similar conversations. I am pasting below two stories:

- story: Agent receives positive sentiment and does a short chitchat with the patient
    steps:
      - intent: greet
        entities:
          - sentiment: pos
      - slot_was_set:
          - sentiment: pos
      - action: utter_greet
      - action: utter_how_do_you_feel_today
      - intent: mood_happy
        entities:
          - sentiment: pos
      - slot_was_set:
          - sentiment: pos
      - action: action_utter_nice_to_hear_that
      - intent: inform
        entities:
          - sentiment: pos
      - slot_was_set:
          - sentiment: pos
      - action: utter_nice_to_hear_that
      - action: utter_inform_doctor
      - intent: affirm
        entities:
          - sentiment: pos
      - slot_was_set:
          - sentiment: pos
      - action: utter_confirm_doctor
      - intent: thank_you
        entities:
          - sentiment: neu
      - slot_was_set:
          - sentiment: neu
      - action: utter_you_are_welcome

- story: Agent handles negative sentiment and does a short chitchat with the patient.
    steps:
      - intent: greet
        entities:
          - sentiment: neu
      - slot_was_set:
          - sentiment: neu
      - action: utter_greet
      - action: utter_how_do_you_feel_today
      - intent: mood_happy
        entities:
          - sentiment: pos
      - slot_was_set:
          - sentiment: pos
      - action: action_utter_nice_to_hear_that
      - intent: affirm
        entities:
          - sentiment: neu
      - slot_was_set:
          - sentiment: neu
      - action: utter_please_go_on
      - intent: inform
        entities:
          - sentiment: neg
      - slot_was_set:
          - sentiment: neg
      - action: utter_handle_negative_sentiment
      - intent: inform
      - action: utter_inform_doctor
      - intent: affirm
        entities:
          - sentiment: neu
      - slot_was_set:
          - sentiment: neu
      - action: utter_confirm_doctor
      - intent: thank_you
        entities:
          - sentiment: neu
      - slot_was_set:
          - sentiment: neu
      - action: utter_you_are_welcome

and here is my config:

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
  # # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
  # # If you'd like to customize it, uncomment and adjust the pipeline.
  # # See https://rasa.com/docs/rasa/tuning-your-model for more information.
  - name: sentiment.SentimentAnalyzer
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
    case_sensitive: False
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
    use_lemma: False
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 200
    constrain_similarities: true
  - name: EntitySynonymMapper
  - name: FallbackClassifier
    threshold: 0.25

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  # # No configuration for policies was provided. The following default policies were used to train your model.
  # # If you'd like to customize them, uncomment and adjust the policies.
  # # See https://rasa.com/docs/rasa/policies for more information.
  - name: MemoizationPolicy
    max_history: 8
  - name: TEDPolicy
    max_history: 6
    epochs: 150
    constrain_similarities: true
  - name: UnexpecTEDIntentPolicy
    max_history: 6
    epochs: 100
  - name: RulePolicy
    # Confidence threshold for the `core_fallback_action_name` to apply.
    # The action will apply if no other action was predicted with
    # a confidence >= core_fallback_threshold
    core_fallback_threshold: 0.2
    core_fallback_action_name: "action_default_fallback"
    enable_fallback_prediction: True

The agent is not able to follow these stories even though they are training examples. If I have understood correctly the TED policy does exactly this, follow the training examples, or not? I don’t know how to solve this issue. I have tried different epochs and different configurations.

Any ideas or suggestions?

Extra information might help:

Rasa Version      :         2.8.3
Minimum Compatible Version: 2.8.0
Rasa SDK Version  :         2.8.1
Rasa X Version    :         None
Python Version    :         3.8.11

UPDATE I tried to make even simpler my stories in a different branch:

  - story: Agent receives positive sentiment, user agrees to inform doctors
    steps:
      - intent: greet
      - action: utter_greet
      - intent: inform
        entities:
          - sentiment
      - slot_was_set:
          - sentiment: pos
      - action: utter_inform_doctor
      - intent: affirm
      - action: utter_confirm_doctor

  - story: Agent receives positive sentiment, user doesn't agree to inform doctors.
    steps:
      - intent: greet
      - action: utter_greet
      - intent: inform
        entities:
          - sentiment
      - slot_was_set:
          - sentiment: pos
      - action: utter_inform_doctor
      - intent: deny
      - action: utter_handle_user_deny_to_inform_doctors
      - intent: deny
      - action: utter_no_problem

When I run rasa shell after intent:greet the agent predicts the wrong answer action: utter_inform_doctor. If I comment out this utterance, train again then the agent doesn’t predict utter_greet instead it does nothing.

UPDATE

I am not sure why but the pipeline seems to have better accuracy in predicting the next action if it is a custom action versus a simple utterance. What is the explanation for this?

2

@nik202 ah sorry I am bringing you in but I need help on this and I can’t find a way to solve it. When you have some time, please have a look at the above.

Many thanks for this. :pray:

3

Hi,
I probably can’t help you at all since I am a newbie :smirk: - but I am interested to see what causes you this trouble. So, just to clarify and better understand your problem:
The stories you posted are in your stories.yml-file (data\stories\stories.yml)? Do you use any rules (data\rules.yml)?
Ciao :wave:

4

Hey,

Exactly.

Yes, I am also using rules.

5

Just in case anyone sees this. It seems that for the stories.yml file in the branch version if I remove from stories this part:

it doesn’t skip the

and works fine.

But, in the main branch that stories, are exactly the same it works just fine.