How to properly create end-to-end stories with forms for automatic testing?

Rasa Open Source
1

Hi there,

I am trying to create end-to-end stories that we will use to evaluate our models whenever we update it. That way, we can test if everything we’ve done in the past is still working.

In order to do so, I did:

  • I chatted with the bot through Facebook Messenger without deviating from the happy path
  • I downloaded the end-to-end story of this conversation using the /conversations/{{sender_id}}/story endpoint. The file is saved under the data_test folder
  • I ran rasa test --stories data_test/happy_interested.md --e2e

My problem is that the test actually fails although I am using the exact same model as the one used to in the conversation to generate the test story in the first place. I am not sure I understand the output here (full output below):

* explain: why do you need this?
    - action_explain
    - action_listen   <!-- predicted: form_apartment_interest -->
    - action_listen
    - form_apartment_interest

Is that expected?

Also, I am not sure I fully understand the failed_story.md output in general. Shouldn’t it give something similar to the actual input story. In the output (see below), the inputs from the user within a form are not here (basically, all the affirms).

How should I properly create my end-to-end story here?

Thanks a lot for your help! Nicolas

Here is the end-to-end story (I removed some useless parts):

## 2373134909417533
* greeting: Hello
    - utter_greet
[...]
    - action_display_apt_selector
* select_apartment: /select_apartment[{"slug": "XYZ"}](slug:XYZ)
    - slot{"slug": "XYZ"}
    - action_get_apt_data
[...]
    - slot{"selection_location": "Somewhere"}
    - utter_apartment_basic_information
    - form_apartment_interest
    - form{"name": "form_apartment_interest"}
    - slot{"requested_slot": "apt_interested"}
* form: affirm: /affirm
    - form: form_apartment_interest
    - slot{"apt_interested": true}
    - slot{"requested_slot": "apt_move_in_date_fits"}
* explain: why do you need this?
    - action_explain
    - action_listen
    - form_apartment_interest
    - slot{"requested_slot": "apt_move_in_date_fits"}
* form: affirm: yes it does
    - form: form_apartment_interest
    - slot{"apt_move_in_date_fits": true}
    - slot{"requested_slot": "apt_lease_length_fits"}
* form: affirm: indeed
    - form: form_apartment_interest
    - slot{"apt_lease_length_fits": true}
    - slot{"preferences_lease_length": 12}
    - slot{"requested_slot": "has_pet"}

and here is the failed_story.md file:

## 2373134909417533
* greeting: Hello
    - utter_greet
[...]
    - action_display_apt_selector
* select_apartment: /select_apartment[{"slug": "EVKEI"}](slug:EVKEI)
    - slot{"slug": "EVKEI"}
    - slot{"slug": "EVKEI"}
    - action_get_apt_data
[...]
    - slot{"selection_location": "Somewhere"}
    - utter_apartment_basic_information
    - form_apartment_interest
    - form{"name": "form_apartment_interest"}
    - slot{"requested_slot": "apt_interested"}
    - slot{"apt_interested": true}
    - slot{"requested_slot": "apt_move_in_date_fits"}
* explain: why do you need this?
    - action_explain
    - action_listen   <!-- predicted: form_apartment_interest -->
    - action_listen
    - form_apartment_interest
    - slot{"requested_slot": "apt_move_in_date_fits"}
    - slot{"apt_move_in_date_fits": true}
    - slot{"requested_slot": "apt_lease_length_fits"}
    - slot{"apt_lease_length_fits": true}
    - slot{"preferences_lease_length": 12}
    - slot{"requested_slot": "has_pet"}
3 Likes