I’m trying to test a few end to end stories as described here ( Evaluating Models ) and I am getting some errors that don’t seem to follow any pattern I can find. Some stories succeed, some fail, and there is little difference between them. This is the stack trace that’s getting printed when I see the error: 2019-07-09 10:56:21 INFO rasa.nlu.test - Confusion matrix, without normalization: [[1 0] [1 0]] Traceback (most recent call last): File "/Users/a1451195/anaconda3/bin/rasa", line 10, in <module> sys.exit(main()) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/__main__.py", line 70, in main cmdline_arguments.func(cmdline_arguments) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/cli/test.py", line 147, in test test_core(args) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/cli/test.py", line 85, in test_core kwargs=vars(args), File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/test.py", line 102, in test_core rasa.core.test(stories, _agent, out_directory=output, **kwargs) File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/test.py", line 514, in test log_failed_stories(story_evaluation.failed_stories, out_directory) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/test.py", line 470, in log_failed_stories f.write(failure.export_stories()) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/trackers.py", line 434, in export_stories return story.as_story_string(flat=True, e2e=e2e) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/structures.py", line 376, in as_story_string story_content += step.as_story_string(flat, e2e) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/structures.py", line 183, in as_story_string result += self._user_string(s, e2e) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/structures.py", line 137, in _user_string return "* {}{}\n".format(prefix, story_step_element.as_story_string(e2e)) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/test.py", line 171, in as_story_string correct_message = md_format_message(self.text, self.intent, self.entities) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/events/__init__.py", line 58, in md_format_message "entities": deserialised_entities, File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/nlu/training_data/formats/markdown.py", line 293, in _generate_message_md entities = sorted(message.get("entities", []), key=lambda k: k["start"]) File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/nlu/training_data/formats/markdown.py", line 293, in <lambda> entities = sorted(message.get("entities", []), key=lambda k: k["start"]) KeyError: 'start' H…

Hi Tyler, looks like an issue converting the story. Can you please share a minimal reproducible example? E.g. config, training story, testing story?

sure thing @amn41 . config.yml language: en pipeline: supervised_embeddings policies: - name: MemoizationPolicy - name: KerasPolicy - name: MappingPolicy nlu.md ## intent:TechSupportIssue - [change](Issue) my [default](SubIssue) [browser](Product) - My [computer](Product) is too [slow](Issue) - My [printer](Product) [won't print](Issue) - how to [fix](Issue) my [router](Product) ## synonym:internet - browser stories.md ## quick path * TechSupportIssue - utter_tech_support_issue ## Generated Story -4339791933908113526 * TechSupportIssue{"Product": "phone", "issue": "won't turn on"} - slot{"Product": "phone"} - utter_tech_support_issue e2e_stories.md ## e to e story 1 * TechSupportIssue: My [internet](Product) was working [yesterday](date) but then I unplugged it and now it is [slow](Issue) - utter_tech_support_issue ** One thing to note is that if I add the specific phrase from the e2e_stories file into the NLU this will pass. Also, this is just a small subset of the NLU files. This was created as a direct port from Dialogflow. Thanks for any help you can provide.

thanks @jabberlope ! this looks like a bug. Could you please copy the above into a github issue?

Errors using `rasa test`

Rasa Open Source

jabberlope (Tyler Burton) July 15, 2019, 9:27pm 5

@amn41 Done.

Here: Errors using `rasa test` · Issue #4017 · RasaHQ/rasa · GitHub

Topic		Replies	Views
Invalid Rasa End-To-End Test Evaluation Rasa Open Source	7	917	December 26, 2022
Tests not finishing Rasa Open Source	8	1751	January 20, 2021
Rasa test Error Rasa Open Source	1	407	July 24, 2021
Rasa test vs. rasa test core Rasa Open Source testing	3	1285	November 7, 2021
'rasa test' complains test stories generated by Rasa X based on real conversation [Deprecated] Rasa X Community Edition	0	253	September 24, 2021

Errors using `rasa test`

Related topics