Forcing intent selection and entities recognition during e2e evaluation

Rasa Open Source
1

I am trying to use rasa test --stories e2e_stories.md --e2e to perform an end to end testing of a model I have built but it is unclear to me how you force intent selection and the entities extracted when writing the test stories. For instance, assume I have a story like this:

## end-to-end story 1
* greet: hello
 - utter_greet
* find_restaurant: /find_restaurant{"chinese":1}
 - action_set_restaurant
* goodbye
 - utter_goodbye

usually you can force the intent selection during a conversation with the forward slash, followed by the entities you wish to be recognized in json format. This strategy however does not seem to help when building a test story. Anyone can help?

2

the example above seems to be the right format, what version are you using?

3

I’m getting the same problem on rasa 1.3.0.

## example 1
* start: /start
 - utter_start_message

Traceback (most recent call last):
  File "/home/ant/.anaconda3/envs/rasa/bin/rasa", line 10, in <module>
    sys.exit(main())
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/__main__.py", line 76, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/cli/test.py", line 153, in test
    test_core(args)
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/cli/test.py", line 88, in test_core
    kwargs=vars(args),
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/test.py", line 108, in test_core
    rasa.core.test(stories, _agent, out_directory=output, **kwargs)
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/core/test.py", line 482, in test
    completed_trackers, agent, fail_on_prediction_errors, e2e
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/core/test.py", line 429, in collect_story_predictions
    in_training_data_fraction = _in_training_data_fraction(action_list)
  File "/home/ant/.anaconda3/envs/rasa/lib/python3.6/site-packages/rasa/core/test.py", line 382, in _in_training_data_fraction
    return len(in_training_data) / len(action_list)
ZeroDivisionError: division by zero
4

Hi @ghost, thanks for bringing this up. It looks like this is a bug in rasa - we’ll be looking into it. I’ve created a bug report here: e2e evaluation raises ZeroDivisionError · Issue #4520 · RasaHQ/rasa · GitHub

5

Hi @ghost does it work without e2e?

6

Are there any updates for this issue? I’m having something similar happen on Rasa 1.2.3, but without the ZeroDivisionError

Here is the story from my e2e_stories.md:

## end-to-end story greet
* greet: /greet{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}
    - action_greet

The resulting failed_stories.md:

## end-to-end story greet
    * greet: /greet{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}  <!-- predicted: greet: /greet[{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}](channelUserId:c8542593f0394358ae24535c624822b4)[{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}](guid:2c8e1e7c-92bd-4507-a8fa-206d4d5d0366)[{"channelUserId":"c8542593f0394358ae24535c624822b4", "guid":"2c8e1e7c-92bd-4507-a8fa-206d4d5d0366", "csiFound":"true"}](csiFound:true) -->
        - slot{"channelUserId": "c8542593f0394358ae24535c624822b4"}
        - slot{"csiFound": "true"}
        - slot{"guid": "2c8e1e7c-92bd-4507-a8fa-206d4d5d0366"}
        - action_greet

Just using “/greet” as the user input works, so it appears to be an issue with the entitites.