Problem with stories and checkpoints

Rasa Open Source
1

So… I’ve stumbled upon a problem when creating a story using checkpoints. For reason I cannot see - tests for validating my stories (and story itself) are not working properly.

I have stories like this:

- story: handle user don't want to make appointment after previous action
  steps:
    - checkpoint: check_if_want_to_make_appointment
    - or:
      - intent: deny
      - intent: stop
    - action: utter_thank_you

- story: handle user want to make appointment after previous action
  steps:
    - checkpoint: check_if_want_to_make_appointment
    - intent: affirm
    - checkpoint: start_make_appointment

- story: handle user don't want to continue after previous action
  steps:
    - checkpoint: check_if_want_to_continue
    - or:
      - intent: deny
      - intent: stop
    - action: utter_thank_you

- story: handle user want to continue after previous action
  steps:
    - checkpoint: check_if_want_to_continue
    - intent: affirm
    - action: action_listen

- story: make appointment (MA) and stop at data verification checkpoint
  steps:
    - or:
      - intent: make_appointment
      - intent: greet+make_appointment
      - intent: greet+make_appointment+inform
    - action: action_reset_make_appointment_slots
    - checkpoint: start_make_appointment
    - action: make_appointment_form
    - active_loop: make_appointment_form
    - active_loop: null
    - action: utter_make_appointment_slots_values
    - action: utter_ask_data_correctness
    - checkpoint: check_data_correctness

- story: make appointment (MA), provide additional info on ask_data_correctness and stop at data verification checkpoint
  steps:
    - or:
      - intent: make_appointment
      - intent: greet+make_appointment
      - intent: greet+make_appointment+inform
    - action: action_reset_make_appointment_slots
    - action: make_appointment_form
    - active_loop: make_appointment_form
    - active_loop: null
    - action: utter_make_appointment_slots_values
    - action: utter_ask_data_correctness
    - intent: inform
    - action: utter_make_appointment_slots_values
    - action: utter_ask_data_correctness
    - checkpoint: check_data_correctness

- story: 1a - handle user deny MA data correctness
  steps:
    - checkpoint: check_data_correctness
    - or:
      - intent: deny
      - intent: stop
    - action: action_reset_make_appointment_slots
    - action: make_appointment_form
    - active_loop: make_appointment_form
    - active_loop: null
    - action: utter_make_appointment_slots_values
    - action: utter_ask_data_correctness
    - checkpoint: check_data_correctness

- story: 1b - handle user affirm MA data correctness and stop at date confirmation
  steps:
    - checkpoint: check_data_correctness
    - intent: affirm
    - action: action_submit_make_appointment_form
    - action: utter_ask_if_date_suitable
    - checkpoint: check_date_confirmation

- story: 2a - handle user negates appointment date
  steps:
    - checkpoint: check_date_confirmation
    - or:
      - intent: deny
      - intent: stop
    - action: action_submit_make_appointment_form
    - action: utter_ask_if_date_suitable
    - checkpoint: check_date_confirmation

- story: 2b - handle user confirms appointment date
  steps:
    - checkpoint: check_date_confirmation
    - intent: affirm
    - action: action_make_appointment
    - action: utter_ask_continue
    - checkpoint: check_if_want_to_continue

- story: ask specialist (AS) and stop at data verification checkpoint
  steps:
    - or:
        - intent: ask_specialist
        - intent: greet+ask_specialist
        - intent: greet+ask_specialist+inform
    - action: ask_specialist_form
    - active_loop: ask_specialist_form
    - active_loop: null
    - action: utter_ask_specialist_slots_values
    - action: utter_ask_data_correctness
    - checkpoint: check_ask_specialist_data_correctness

- story: 1a - handle user deny AS data correctness
  steps:
    - checkpoint: check_ask_specialist_data_correctness
    - or:
      - intent: deny
      - intent: stop
    - action: action_reset_ask_specialist_slots
    - action: ask_specialist_form
    - active_loop: ask_specialist_form
    - active_loop: null
    - action: utter_ask_specialist_slots_values
    - action: utter_ask_data_correctness
    - checkpoint: check_ask_specialist_data_correctness

- story: 1b - handle user confirms AS data correctness, specialist in DB
  steps:
    - checkpoint: check_ask_specialist_data_correctness
    - intent: affirm
    - action: action_submit_ask_specialist_form
    - action: action_search_for_specialist
    - checkpoint: check_specialist_exist

- story: 2a - handle specialist exist
  steps:
    - checkpoint: check_specialist_exist
      slot_was_set:
      - specialist_exist: True
    - action: utter_ask_if_make_appointment
    - checkpoint: check_if_want_to_make_appointment

- story: 2b - handle specialist doesn't exist
  steps:
    - checkpoint: check_specialist_exist
      slot_was_set:
      - specialist_exist: False
    - checkpoint: check_if_want_to_continue
...

Rules:

# make appointment rules
- rule: activate make appointment form
  steps:
    - or:
      - intent: make_appointment
      - intent: greet+make_appointment
      - intent: greet+make_appointment+inform
    - action: action_reset_make_appointment_slots
    - action: make_appointment_form
    - active_loop: make_appointment_form

# ask specialist rules
- rule: activate ask specialist form
  steps:
    - or:
      - intent: ask_specialist
      - intent: greet+ask_specialist
      - intent: greet+ask_specialist+inform
    - action: ask_specialist_form
    - active_loop: ask_specialist_form

And test like this (from the results to see what’s wrong):

- story: ask a specialist, confirm data correctness and make appointment (/tmp/tmpd_d9tjk5/d6a25c7d55084b51b98fbd567b931f39_test_conversations.yml)
  steps:
  - intent: greet
  - action: utter_greet
  - intent: ask_specialist
  - action: ask_specialist_form
  - active_loop: ask_specialist_form
  - active_loop: null
  - action: utter_ask_specialist_slots_values  # predicted: utter_ask_if_date_suitable
  - action: utter_ask_data_correctness  # predicted: action_listen
  - intent: affirm
  - action: action_submit_ask_specialist_form  # predicted: action_submit_ask_price_form
  - action: action_search_for_specialist  # predicted: utter_make_appointment_slots_values
  - action: utter_ask_if_make_appointment  # predicted: action_listen
  - intent: affirm
  - action: make_appointment_form
  - active_loop: make_appointment_form
  - active_loop: null
  - action: utter_make_appointment_slots_values
  - action: utter_ask_data_correctness
  - intent: affirm
  - action: action_submit_make_appointment_form
  - action: utter_ask_if_date_suitable
  - intent: affirm
  - action: action_make_appointment
  - action: utter_ask_continue
  - intent: deny
  - action: utter_thank_you

I have no idea, how it could possibly predict utter_ask_if_date_suitable after ask_specialist_form. I have rules for activating forms and I had rules for submitting it, but I had removed submission rules, because I needed to validate slot correctness before submission ifself (and that was considered to be a state machine).

What is interesting is that removing the last checkpoint from stories in the 2nd “divide” level fixed everything.

- story: 2a - handle specialist exist
  steps:
    - checkpoint: check_specialist_exist
      slot_was_set:
      - specialist_exist: True
    - action: utter_ask_if_make_appointment
    - checkpoint: check_if_want_to_make_appointment

- story: 2b - handle specialist doesn't exist
  steps:
    - checkpoint: check_specialist_exist
      slot_was_set:
      - specialist_exist: False
    - checkpoint: check_if_want_to_continue

So having story like below (and remove stories shown above) works just fine both in test and in rasa shell:

- story: 1b - handle user confirms AS data correctness, specialist in DB
  steps:
    - checkpoint: check_ask_specialist_data_correctness
    - intent: affirm
    - action: action_submit_ask_specialist_form
    - action: action_search_for_specialist
      slot_was_set:
         - specialist_exist: True
    - action: utter_ask_if_make_appointment
    - checkpoint: check_if_want_to_make_appointment

Problem with this approach is that I can’t create 3rd variant, where specalist_exist: False, because a lot of conflicts are shown with rasa data validate then.

I’ve spent a long time on this problem and didn’t find any solution and source of the problem. Any help would be very appreciated.

2

Hi! First off – I’d advise you cut down on the number of checkpoints you’re using. Checkpoints should be used sparingly because they slow down training, and some of your story snippets could probably be rules, for example all of these:

“handle user don’t want to continue after previous action”:

- rule:
  steps:
  - action: utter_ask_continue
  - or:
      - intent: deny
      - intent: stop
  - action: utter_thank_you

“handle user do want to continue after previous action”

“handle user (don’t) want to make appointment after previous action”

“handle user confirms/negates appointment date”

“handle specialist (doesn’t) exist”

I’m not sure why you’re seeing the behaviour you’re seeing, in part because I don’t have all of your stories, and I can’t try this out for myself. I think maybe you should try getting rid of some of these checkpoints and writing rules instead, and if the problem persists we can take another look. What do you think?

3

I honestly thought about it, but… I had some problems with rules in my previous ideas because rules predictions were conflicting with stories. Problem was, that conflicts were mainly because rules wanted to predict “action listen” as its last step, but stories were supposed to continue, so… I’ve limited rules to very basic things (greet, goodbye, etc.) and introduced checkpoints, which almost works :stuck_out_tongue:

I will try to utilize rules in place of checkpoints and see if it will work.

4

Oh, you can prevent this by adding wait_for_user_input: false to the end of your rule (see here)