Errors using `rasa test`

I’m trying to test a few end to end stories as described here (Evaluating Models) and I am getting some errors that don’t seem to follow any pattern I can find.

Some stories succeed, some fail, and there is little difference between them.

This is the stack trace that’s getting printed when I see the error:

2019-07-09 10:56:21 INFO     rasa.nlu.test  - Confusion matrix, without normalization:
[[1 0]
 [1 0]]
Traceback (most recent call last):
  File "/Users/a1451195/anaconda3/bin/rasa", line 10, in <module>
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/", line 70, in main
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/cli/", line 147, in test
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/cli/", line 85, in test_core
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/", line 102, in test_core
    rasa.core.test(stories, _agent, out_directory=output, **kwargs)
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/", line 514, in test
    log_failed_stories(story_evaluation.failed_stories, out_directory)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/", line 470, in log_failed_stories
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/", line 434, in export_stories
    return story.as_story_string(flat=True, e2e=e2e)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/", line 376, in as_story_string
    story_content += step.as_story_string(flat, e2e)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/", line 183, in as_story_string
    result += self._user_string(s, e2e)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/", line 137, in _user_string
    return "* {}{}\n".format(prefix, story_step_element.as_story_string(e2e))
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/", line 171, in as_story_string
    correct_message = md_format_message(self.text, self.intent, self.entities)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/events/", line 58, in md_format_message
    "entities": deserialised_entities,
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/nlu/training_data/formats/", line 293, in _generate_message_md
    entities = sorted(message.get("entities", []), key=lambda k: k["start"])
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/nlu/training_data/formats/", line 293, in <lambda>
    entities = sorted(message.get("entities", []), key=lambda k: k["start"])
KeyError: 'start'

Hoping someone has seen this or can point me to what the issue may be.

Thanks in advance.

Hi Tyler, looks like an issue converting the story. Can you please share a minimal reproducible example? E.g. config, training story, testing story?

sure thing @amn41.


language: en

pipeline: supervised_embeddings

- name: MemoizationPolicy
- name: KerasPolicy
- name: MappingPolicy

## intent:TechSupportIssue

- [change](Issue) my [default](SubIssue) [browser](Product)

- My [computer](Product) is too [slow](Issue)

- My [printer](Product) [won't print](Issue)

- how to [fix](Issue) my [router](Product)

## synonym:internet

- browser

## quick path
* TechSupportIssue
    - utter_tech_support_issue

## Generated Story -4339791933908113526
* TechSupportIssue{"Product": "phone", "issue": "won't turn on"}
    - slot{"Product": "phone"}
    - utter_tech_support_issue

## e to e story 1
* TechSupportIssue: My [internet](Product) was working [yesterday](date) but then I unplugged it and now it is [slow](Issue)
    - utter_tech_support_issue

** One thing to note is that if I add the specific phrase from the e2e_stories file into the NLU this will pass.

Also, this is just a small subset of the NLU files. This was created as a direct port from Dialogflow.

Thanks for any help you can provide.

thanks @jabberlope ! this looks like a bug. Could you please copy the above into a github issue?

@amn41 Done.

Here: Errors using `rasa test` · Issue #4017 · RasaHQ/rasa · GitHub

1 Like