Forcing intent selection and entities recognition during e2e evaluation

I am trying to use rasa test --stories --e2e to perform an end to end testing of a model I have built but it is unclear to me how you force intent selection and the entities extracted when writing the test stories. For instance, assume I have a story like this:

## end-to-end story 1
* greet: hello
 - utter_greet
* find_restaurant: /find_restaurant{"chinese":1}
 - action_set_restaurant
* goodbye
 - utter_goodbye

usually you can force the intent selection during a conversation with the forward slash, followed by the entities you wish to be recognized in json format. This strategy however does not seem to help when building a test story. Anyone can help?

the example above seems to be the right format, what version are you using?

I’m getting the same problem on rasa 1.3.0.

## example 1
* start: /start
 - utter_start_message
Traceback (most recent call last):
  File "/home/ant/.anaconda3/envs/rasa/bin/rasa", line 10, in <module>
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/", line 76, in main
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/cli/", line 153, in test
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/cli/", line 88, in test_core
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/", line 108, in test_core
    rasa.core.test(stories, _agent, out_directory=output, **kwargs)
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/core/", line 482, in test
    completed_trackers, agent, fail_on_prediction_errors, e2e
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/core/", line 429, in collect_story_predictions
    in_training_data_fraction = _in_training_data_fraction(action_list)
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/core/", line 382, in _in_training_data_fraction
    return len(in_training_data) / len(action_list)
ZeroDivisionError: division by zero

Hi @ghost, thanks for bringing this up. It looks like this is a bug in rasa - we’ll be looking into it. I’ve created a bug report here: e2e evaluation raises ZeroDivisionError · Issue #4520 · RasaHQ/rasa · GitHub

Hi @ghost does it work without e2e?

Are there any updates for this issue? I’m having something similar happen on Rasa 1.2.3, but without the ZeroDivisionError

Here is the story from my

## end-to-end story greet
* greet: /greet{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}
    - action_greet

The resulting

## end-to-end story greet
    * greet: /greet{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}  <!-- predicted: greet: /greet[{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}](channelUserId:c8542593f0394358ae24535c624822b4)[{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}](guid:2c8e1e7c-92bd-4507-a8fa-206d4d5d0366)[{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}](csiFound:true) -->
        - slot{"channelUserId": "c8542593f0394358ae24535c624822b4"}
        - slot{"csiFound": "true"}
        - slot{"guid": "2c8e1e7c-92bd-4507-a8fa-206d4d5d0366"}
        - action_greet

Just using “/greet” as the user input works, so it appears to be an issue with the entitites.