Errors using `rasa test`

Rasa Open Source
1

I’m trying to test a few end to end stories as described here (Evaluating Models) and I am getting some errors that don’t seem to follow any pattern I can find.

Some stories succeed, some fail, and there is little difference between them.

This is the stack trace that’s getting printed when I see the error:

2019-07-09 10:56:21 INFO     rasa.nlu.test  - Confusion matrix, without normalization:
[[1 0]
 [1 0]]
Traceback (most recent call last):
  File "/Users/a1451195/anaconda3/bin/rasa", line 10, in <module>
    sys.exit(main())
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/__main__.py", line 70, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/cli/test.py", line 147, in test
    test_core(args)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/cli/test.py", line 85, in test_core
    kwargs=vars(args),
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/test.py", line 102, in test_core
    rasa.core.test(stories, _agent, out_directory=output, **kwargs)
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/test.py", line 514, in test
    log_failed_stories(story_evaluation.failed_stories, out_directory)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/test.py", line 470, in log_failed_stories
    f.write(failure.export_stories())
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/trackers.py", line 434, in export_stories
    return story.as_story_string(flat=True, e2e=e2e)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/structures.py", line 376, in as_story_string
    story_content += step.as_story_string(flat, e2e)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/structures.py", line 183, in as_story_string
    result += self._user_string(s, e2e)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/training/structures.py", line 137, in _user_string
    return "* {}{}\n".format(prefix, story_step_element.as_story_string(e2e))
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/test.py", line 171, in as_story_string
    correct_message = md_format_message(self.text, self.intent, self.entities)
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/core/events/__init__.py", line 58, in md_format_message
    "entities": deserialised_entities,
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/nlu/training_data/formats/markdown.py", line 293, in _generate_message_md
    entities = sorted(message.get("entities", []), key=lambda k: k["start"])
  File "/Users/a1451195/anaconda3/lib/python3.7/site-packages/rasa/nlu/training_data/formats/markdown.py", line 293, in <lambda>
    entities = sorted(message.get("entities", []), key=lambda k: k["start"])
KeyError: 'start'

Hoping someone has seen this or can point me to what the issue may be.

Thanks in advance.

2

Hi Tyler, looks like an issue converting the story. Can you please share a minimal reproducible example? E.g. config, training story, testing story?

3

sure thing @amn41.

config.yml

language: en

pipeline: supervised_embeddings

policies:
- name: MemoizationPolicy
- name: KerasPolicy
- name: MappingPolicy

nlu.md

## intent:TechSupportIssue

- [change](Issue) my [default](SubIssue) [browser](Product)

- My [computer](Product) is too [slow](Issue)

- My [printer](Product) [won't print](Issue)

- how to [fix](Issue) my [router](Product)

## synonym:internet

- browser

stories.md

## quick path
* TechSupportIssue
    - utter_tech_support_issue

## Generated Story -4339791933908113526
* TechSupportIssue{"Product": "phone", "issue": "won't turn on"}
    - slot{"Product": "phone"}
    - utter_tech_support_issue

e2e_stories.md

## e to e story 1
* TechSupportIssue: My [internet](Product) was working [yesterday](date) but then I unplugged it and now it is [slow](Issue)
    - utter_tech_support_issue

** One thing to note is that if I add the specific phrase from the e2e_stories file into the NLU this will pass.

Also, this is just a small subset of the NLU files. This was created as a direct port from Dialogflow.

Thanks for any help you can provide.

4

thanks @jabberlope ! this looks like a bug. Could you please copy the above into a github issue?

5

@amn41 Done.

Here: Errors using `rasa test` · Issue #4017 · RasaHQ/rasa · GitHub

1 Like