Does rasa end-to-end test mechanism have a limit of instructions? (circuit breaker tripped)

duketemon · May 20, 2020, 2:31pm

Hello! I’m working on the questionnaire (in italian). When I’m using the bot in a production (telegram as frontend) everything working fine, but when I trying to use end-to-end tests it fails. I checked all values it’s correct.

failed_stories.md

## Shortest path
* start: /start
    - utter_compliance
* agree_compliance: Accetto
    - slot{"age": 22}
    - slot{"gender": "m"}
    - slot{"school_degree": "laurea specialistica"}
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - slot{"physical_activity_per_day": "10 o più"}
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"physical_activity_per_week": "150 o più"}
    - physical_activity_per_week_form
    - form{"name": "physical_activity_per_week_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"red_meat": "meno di 2"}
    - red_meat_form
    - form{"name": "red_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"cured_meat": "meno di 1"}
    - cured_meat_form
    - form{"name": "cured_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"fish_products": "2 o più"}
    - fish_products_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "fish_products_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"fruit_portions_per_day": "3 o più"}
    - fruit_portions_per_day_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "fruit_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"vegetable_portions_per_day": "2 o più"}
    - vegetable_portions_per_day_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "vegetable_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"sweets": "meno di 2"}
    - sweets_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "sweets_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"milk_products": "1 o più"}
    - milk_products_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "milk_products_form"}
    - form{"name": null}
    - action_delay_response   <!-- predicted: circuit breaker tripped -->
    - slot{"olive_oil": "si"}
    - olive_oil_form   <!-- predicted: circuit breaker tripped -->
    - form{"name": "olive_oil_form"}
    - form{"name": null}
    - action_finish_questionare   <!-- predicted: circuit breaker tripped -->
    - utter_thanks_for_answers   <!-- predicted: circuit breaker tripped -->
    - action_listen   <!-- predicted: circuit breaker tripped -->

I’m using:

ubuntu 20.04
docker version 18.09.9
docker compose version 1.25.5. Rasa backend container + mongoDB container for logs

How I run the test:

sudo docker exec -ti $(sudo docker ps -f "name=rasa-backend" -q) rasa test core -s tests/test.md --e2e

stories.md

## standard workflow
* start
    - utter_compliance
* agree_compliance
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_week_form
    - form{"name": "physical_activity_per_week_form"}
    - form{"name": null}
    - action_delay_response
    - red_meat_form
    - form{"name": "red_meat_form"}
    - form{"name": null}
    - action_delay_response
    - cured_meat_form
    - form{"name": "cured_meat_form"}
    - form{"name": null}
    - action_delay_response
    - fish_products_form
    - form{"name": "fish_products_form"}
    - form{"name": null}
    - action_delay_response
    - fruit_portions_per_day_form
    - form{"name": "fruit_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - vegetable_portions_per_day_form
    - form{"name": "vegetable_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - sweets_form
    - form{"name": "sweets_form"}
    - form{"name": null}
    - action_delay_response
    - milk_products_form
    - form{"name": "milk_products_form"}
    - form{"name": null}
    - action_delay_response
    - olive_oil_form
    - form{"name": "olive_oil_form"}
    - form{"name": null}
    - action_finish_questionare
    - utter_thanks_for_answers

## non standard workflow
* start
    - utter_compliance
* not_now
    - utter_accept_compliance_to_start

test.md

## Accept rules later
* start: /start
    - utter_compliance
* not_now: Non ora
    - utter_accept_compliance_to_start

## Shortest path
* start: /start
    - utter_compliance
* agree_compliance: Accetto
    - slot{"age": 22}
    - slot{"gender": "m"}
    - slot{"school_degree": "laurea specialistica"}
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - slot{"physical_activity_per_day": "10 o più"}
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"physical_activity_per_week": "150 o più"}
    - physical_activity_per_week_form
    - form{"name": "physical_activity_per_week_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"red_meat": "meno di 2"}
    - red_meat_form
    - form{"name": "red_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"cured_meat": "meno di 1"}
    - cured_meat_form
    - form{"name": "cured_meat_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"fish_products": "2 o più"}
    - fish_products_form
    - form{"name": "fish_products_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"fruit_portions_per_day": "3 o più"}
    - fruit_portions_per_day_form
    - form{"name": "fruit_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"vegetable_portions_per_day": "2 o più"}
    - vegetable_portions_per_day_form
    - form{"name": "vegetable_portions_per_day_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"sweets": "meno di 2"}
    - sweets_form
    - form{"name": "sweets_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"milk_products": "1 o più"}
    - milk_products_form
    - form{"name": "milk_products_form"}
    - form{"name": null}
    - action_delay_response
    - slot{"olive_oil": "si"}
    - olive_oil_form
    - form{"name": "olive_oil_form"}
    - form{"name": null}
    - action_finish_questionare
    - utter_thanks_for_answers

requirements.txt

tensorflow==2.1.0
tensorflow-text==2.1.1
tensorflow-addons==0.9.1
rasa_sdk==1.9.0
rasa==1.9.7

What I tried:

Shuffle an order of forms (each form contains group of questions). The same error.
Instead of

* agree_compliance: Accetto
    - slot{"age": 22}
...
  - personal_information_form

use

* agree_compliance: Accetto
    - personal_information_form
...
* form: [22](age)

The same error.

Remove a few forms (in stories.md and tests.md). The same error.
Use only two forms

## standard workflow
* start
    - utter_compliance
* agree_compliance
    - personal_information_form
    - form{"name": "personal_information_form"}
    - form{"name": null}
    - action_delay_response
    - physical_activity_per_day_form
    - form{"name": "physical_activity_per_day_form"}
    - form{"name": null}

This one is working fine. It seems like an end-to-end has limit of the number instructions. Probably the rasa in not the best choice for questionnaire, but any suggestions are welcomed. The whole questionnaire can be splitted on the group of questions with indentical logic, that’s why I used forms for it.

Ghostvv · May 25, 2020, 1:00pm

I think form: prefix here is not supported in e2e tests:

Forms are specifically designed for questionnaire. I think is there is technical problem how form input is handled in e2e test stories

ricwo · May 27, 2020, 7:05am

@duketemon we’ll look into this - it could be that forms are not fully supported in e2e tests

Ghostvv · May 27, 2020, 9:02am

the test stories similar like normal training stories should contain intents that are handled by the form, that’s why the last story is working fine

duketemon · June 1, 2020, 11:45am

Thanks everybody. The problem have been solved

Topic		Replies	Views
Same code behaving differently on different servers Rasa Open Source testing	1	272	July 11, 2023
Rasa refuses to work properly, for further flow developments Rasa Open Source testing	3	580	July 12, 2023
Circuit Break Tripped Rasa Open Source	1	1041	May 4, 2020
RASA threhold reached - tripping issue Rasa Open Source	2	594	June 22, 2021
What is the standard practice for Rasa Testing? Rasa Open Source	4	469	September 24, 2020

Does rasa end-to-end test mechanism have a limit of instructions? (circuit breaker tripped)

Related topics