# Failed stories

**URL:** <https://forum.rasa.com/t/failed-stories/46003>\
**Category:** Rasa Open Source\
**Created:** [July 22, 2021, 1:14am UTC](https://forum.rasa.com/t/failed-stories/46003 "2021-07-22T01:14:15Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 22, 2021, 1:14am UTC](https://forum.rasa.com/t/failed-stories/46003/1 "2021-07-22T01:14:15Z")

</div>

I really hope someone answers my question this time.

Why do I see some stories reported as failed stories in `failed_test_stories.yml` file when they actually didn’t fail. Testing those same intents with `rasa shell` showed they **did not** fail. For example;

```auto
- story: show the covid testing centers (./tests/test_stories.yml)
  steps:
  - intent: testing_centers
    entities:
    - disease: covid
  - action: action_show_testing_centers

```

usually failed intents come with information on what went wrong. e.g:

```auto
- intent: sendoff # predicted: nlu_fallback: bye

```

can someone please explain?

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 23, 2021, 4:06pm UTC](https://forum.rasa.com/t/failed-stories/46003/2 "2021-07-23T16:06:16Z")

</div>

hi @laboratory in a nutshell, rasa predicted a different intent or action than how you expected the conversation to flow - a failed story means that the steps in the test conversation you designed didn’t match the steps predicted by rasa. Also, one thing to bear in mind is that the custom actions won’t execute during testing, so best to check if you have a test conversation flow relying on the execution of a custom action. This [section in the docs](https://rasa.com/docs/rasa/testing-your-assistant#writing-test-stories) could be helpful too in case you haven’t checked it yet.

Are you able to share your stories and rules files with me as well as the `test_stories.yml` file to help you further?

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 23, 2021, 4:27pm UTC](https://forum.rasa.com/t/failed-stories/46003/3 "2021-07-23T16:27:40Z")

</div>

Thanks, @anca for your response.

1. Your opening statement mentioned,

> “rasa predicted a different intent or action than how you expected the conversation to flow”

I understand this and that’s why I think that particular intent or story is additionally tagged with Rasa’s prediction like so `# predicted: nlu_fallback: bye`

1. 

> a failed story means that the steps in the test conversation you designed didn’t match the steps predicted by rasa.

Are you referring to the step as a whole or each intent and action in a step (like point 1 above)?

1. please find both stories and rules files

[test\_stories.yml](https://forum.rasa.com/uploads/short-url/5gdrtOmatd8OM7c3WiLL0CuWkWQ.yml) (5.0 KB) [rules.yml](https://forum.rasa.com/uploads/short-url/ghcmZPsIjzy6S2QnZY42lHsJ4kb.yml) (862 Bytes) [stories.yml](https://forum.rasa.com/uploads/short-url/i6lmv36v6IGzUQDg3Jj4US7hs1U.yml) (2.2 KB)

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 23, 2021, 4:47pm UTC](https://forum.rasa.com/t/failed-stories/46003/7 "2021-07-23T16:47:13Z")

</div>

> Are you referring to the step as a whole or each intent and action in a step (like point 1 above)?

@laboratory By step I meant intent or action step.

I had a quick look at the files you provided - most of the stories in `stories.yml` seem to be rules actually, you should design a story that is like a conversation (with beginning, middle, end queries) similar to how you expect an end user to interact with the assistant. So one step would be to restructure `stories.yml` and move most of these small existing stories to `rules.yml`. Another advice I have is to break down some of your test stories, because some seem to be 2 or more stories in the test story.

For example:

```auto
- story: check and display lockdown information
  steps:
  - user: |
      are we under lockdown?
    intent: lockdown_areas
  - action: action_check_lockdown
  - user: |
      is [musanze]{"entity":"location"} currently experiencing lockdown?
    intent: lockdown_areas
  - action: action_check_lockdown

```

Should be actually 2 story examples.

Hope this helps - I can check again on Mon how you’re progressing 🚀

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 23, 2021, 4:59pm UTC](https://forum.rasa.com/t/failed-stories/46003/10 "2021-07-23T16:59:05Z")

</div>

@anca .Yes this works well. I will look into story restructuring (probably with `rasa interactive`)

> one thing to bear in mind is that the custom actions won’t execute during testing

looking at my files has this affected testing in a way or the way some of the (passed) test cases still showing up in `failes_stories.yml` (the reason for this post)?

> I can check again on Mon how you’re progressing 🚀

Please do. I will aprreciate

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 26, 2021, 11:16am UTC](https://forum.rasa.com/t/failed-stories/46003/16 "2021-07-26T11:16:58Z")

</div>

> has this affected testing in a way or the way some of the (passed) test cases still showing up in `failes_stories.yml`

It is possible if the outcome of your custom action influences the direction of the conversation, for example setting a `SlotSet` event etc.

@laboratory have you managed to progress otherwise with your test stories?

Additionally, the point raised by @nik202 with regards to making sure your nlu examples should be double-checked and improved is also valid - rasa might not be able to generalise the user message in your test stories and predict the intent you expected.

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 26, 2021, 12:17pm UTC](https://forum.rasa.com/t/failed-stories/46003/17 "2021-07-26T12:17:34Z")

</div>

> have you managed to progress otherwise with your test stories?

@anca I am afraid I haven’t made progress with my test stories. The only thing I did was splitting the ones that were 2 or more into individual stories (as you recommended).

> you should design a story that is like a conversation (with beginning, middle, end queries) similar to how you expect an end user to interact with the assistant.

As regards my `stories.yml`, I tried designing like an actual conversation (with the help of rasa interactive) however, it messed up my bot performance seriousyl so I reverted to the rule-like format as you noticed initially.

> It is possible if the outcome of your custom action influences the direction of the conversation, for example setting a `SlotSet` event etc.

For now, I haven’t used slots in my development. The only way I think custom actions might have caused this is are the if…else clauses (if I am right). if this entity, do this and dispatch this, else do that and dispatch that. My worry is it also happens for intents that don’t require custom actions at all.

> rasa might not be able to generalise the user message in your test stories and predict the intent you expected.

I believe these are for cases where the model predicted **nlu\_fallbacks**. But the point of this post is around those intents that weren’t predicted as fallbacks but still showing up as failed tests.

Please see my `failed_test stories.yml` file. [failed\_test\_stories.yml](https://forum.rasa.com/uploads/short-url/nRxbSIQNASx4a44wSdUGXs5qwO0.yml) (2.1 KB) You would observe that only intent `vaccination_requirement` predicted fallback but others showed up as well.

Thank you

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 27, 2021, 10:03am UTC](https://forum.rasa.com/t/failed-stories/46003/18 "2021-07-27T10:03:41Z")

</div>

@anca wondering if you saw the above message.

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 27, 2021, 2:20pm UTC](https://forum.rasa.com/t/failed-stories/46003/19 "2021-07-27T14:20:05Z")

</div>

@laboratory are you able to send me a link to your assistant repository? That would enable me to check the latest versions of your files because it’s a bit difficult to interpret `failed_test_stories.yml` without looking at the last version of the rest of the files.

> it messed up my bot performance seriousyl so I reverted to the rule-like format as you noticed initially

Please provide more details as what exactly happened, the debug log for errors etc.

I still strongly advise to move into `rules.yml` any sets of dialogue steps that should always behave like that (a certain action that should always follow a certain intent etc).

Your test stories should ideally test happy conversation paths as well as unhappy paths, as in the documentation link I originally shared in my first reply.

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 27, 2021, 3:13pm UTC](https://forum.rasa.com/t/failed-stories/46003/20 "2021-07-27T15:13:37Z")

</div>

> are you able to send me a link to your assistant repository?

@anca That would be great. But I’m afraid I can’t share a link here since it would be accessed by the public. It’s a company project. Any suggestion on how to get a link across is highly welcomed.

> Please provide more details as what exactly happened, the debug log for errors etc.

What I meant was that different and unrelated actions/response were shown for intents when I type in a question. I assume I don’t yet have a good knowledge or grasp of good story writing

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 27, 2021, 4:45pm UTC](https://forum.rasa.com/t/failed-stories/46003/21 "2021-07-27T16:45:31Z")

</div>

> I’m afraid I can’t share a link

No worries, I completely understand.

Since it’s a bit tricky then, I suggest running [`rasa x`](https://rasa.com/docs/rasa/command-line-interface/#rasa-x) to generate stories - that could be helpful. You could even go further with [sharing your bot](https://rasa.com/docs/rasa-x/installation-and-setup/deploy#share-your-bot-deployment) with guest testers that could help you generate more stories.

> different and unrelated actions/response were shown for intents when I type in a question.

Did this happen even though you’ve transferred those original short story snippets to `rules.yml`?

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 27, 2021, 5:09pm UTC](https://forum.rasa.com/t/failed-stories/46003/22 "2021-07-27T17:09:34Z")

</div>

@anca Currently I have left them still in `stories.yml`… I know they may be rule-like and maybe acting like rules. I was wondering if there’s a difference between having them in `stories.yml` or in `rules.yml`. Didn’t think there was any. So I thought, when I eventually generate (proper) stories, I’ll move them.

Is this assumption correct? What could be the downsides (if any)?

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 28, 2021, 8:27am UTC](https://forum.rasa.com/t/failed-stories/46003/23 "2021-07-28T08:27:11Z")

</div>

@laboratory not moving those snippets into `rules.yml` is highly likely the reason why your bot became unstable.

I’ll reference the [documentation](https://rasa.com/docs/rasa/training-data-format#conversation-training-data) on the difference between stories and rules, because they should be used together for best results:

```auto
Stories are used to train a machine learning model to identify patterns in conversations and generalize to unseen conversation paths. 
Rules describe small pieces of conversations that should always follow the same path and are used to train the RulePolicy.

```

You could also follow this [advice in the docs](https://rasa.com/docs/rasa/writing-stories#when-to-write-stories-vs-rules) on how to write conversation training data.

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 28, 2021, 10:16am UTC](https://forum.rasa.com/t/failed-stories/46003/24 "2021-07-28T10:16:45Z")

</div>

Thanks @anca

I studied the rasa docs all night and I have a better understanding on your reply above and it makes much sense.

I think the only thing I am battling with is story conflict/resolution (or conflict between a particular story and rule). I am wondering if there’s a resource on that.

I wrote stories for lockdown information based on the different conversation path I think as follows:

```auto
stories:
- story: show if a location is under lockdown or not
  steps:
  - intent: lockdown_areas
  - action: action_check_lockdown

- story: location + ask location + specify location
  steps:
  - intent: lockdown_areas
  - action: action_check_lockdown
  - action: utter_ask_location_lockdown
  - intent: specify_location_lockdown
  - action: action_check_lockdown

- story: location + location not found + specify location
  steps:
  - intent: lockdown_areas
  - action: action_check_lockdown
  - action: utter_location_not_found
  - intent: specify_location_lockdown
  - action: action_check_lockdown

```

However, when I run `rasa data validate`, I get the below warning:

```auto
Story structure conflict after action 'action_check_lockdown':
  utter_ask_location_lockdown predicted in 'location + ask location + specify location'
  utter_location_not_found predicted in 'location + location not found + specify location'
  action_listen predicted in 'show if a location is under lockdown or not'

```

When I removed story 3 completely and removed `utter_ask_location_lockdown` in story 2. The error went.

Can you help me understand this?

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 28, 2021, 11:00am UTC](https://forum.rasa.com/t/failed-stories/46003/25 "2021-07-28T11:00:08Z")

</div>

@anca Also, while the conversation in this thread has been useful, I still meet my initial challenge. Stories still shows up in `failed_test_stories.yml` and showing no comment on what was predicted by the model instead. Meaning the stories were correct (upon confirmation with `rasa shell` and shouldn’t be there.

I am wondering what it takes to solve this.

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 28, 2021, 11:08am UTC](https://forum.rasa.com/t/failed-stories/46003/26 "2021-07-28T11:08:54Z")

</div>

@laboratory what does `action_check_lockdown` do? Look-up a location entity in a database and set a slot (let’s say location?) I think based on a raw understanding of your stories, you could try a few options:

- use [checkpoints](https://rasa.com/docs/rasa/training-data-format/#checkpoints) and [`slot_was_set`](https://rasa.com/docs/rasa/training-data-format#slots) step after your custom action:

```auto
stories:
- story: show if a location is under lockdown or not
  steps:
  - intent: lockdown_areas
  - action: action_check_lockdown
  - checkpoint: lockdown_checkpoint

- story: location + ask location + specify location
  steps:
  - checkpoint: lockdown_checkpoint
  - slot_was_set:
        - location: test_location
  - action: utter_ask_location_lockdown
  - intent: specify_location_lockdown
  - action: action_check_lockdown

- story: location + location not found + specify location
  steps:
  - checkpoint: lockdown_checkpoint
  - slot_was_set: 
      - location: null
  - action: utter_location_not_found
  - intent: specify_location_lockdown
  - action: action_check_lockdown

```

- you could also have your custom action dispatch different messages and this way you can remove the 2nd action `utter_` and probably you’d have one single story in this case:

```auto
- story: handle lockdown check path
  steps:
  - intent: lockdown_areas
  - action: action_check_lockdown
  - intent: specify_location_lockdown
  - action: action_check_lockdown

```

It’s a matter of experimenting, I’m a bit uncertain myself if the first option would work only because I don’t know the code in your custom action.

---

<div class="post-metadata">

**Author:** ![anca](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/anca/32/14540_2.png) [@anca](https://forum.rasa.com/u/anca)\
**Post date:** [July 28, 2021, 11:10am UTC](https://forum.rasa.com/t/failed-stories/46003/27 "2021-07-28T11:10:49Z")

</div>

> what it takes to solve this.

This entire thread on designing stories and rules should help; your test stories are failing because rasa predicts different actions for intents and fixing your conversation training data would definitely help this.

---

<div class="post-metadata">

**Author:** ![ermarkar](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ermarkar/32/13716_2.png) [@ermarkar](https://forum.rasa.com/u/ermarkar)\
**Post date:** [July 28, 2021, 11:39am UTC](https://forum.rasa.com/t/failed-stories/46003/28 "2021-07-28T11:39:40Z")

</div>

hi @anca , this is dfferent question, but can you please answer this [Failed to load the component rasa](http://forum.rasa.com/t/failed-to-load-the-component-rasa/46240)

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 28, 2021, 1:15pm UTC](https://forum.rasa.com/t/failed-stories/46003/29 "2021-07-28T13:15:11Z")

</div>

> you could also have your custom action dispatch different messages and this way you can remove the 2nd action `utter_` and probably you’d have one single story in this case:

@anca Yes this works. My custom action dispatches different messages based of if an entity **location** was found in a user message or not. For now, I haven’t implemented slots in this project

> [@anca](#):
>
> your test stories are failing because rasa predicts different actions for intents and fixing your conversation training data would definitely help this.

I don’t know how else to explain that the tests are not failing. **Maybe I don’t understand how some of these things work under the hood** but the some of the tests in `failed_test_stories.yml` aren’t failing when I test my assistant (with `rasa shell`). Moreover, my knowledge is that filed stories comes with an extra comment as to what was predicted for example ( `# predicted: nlu_fallback: bye` or `# predicted: action_default_fallback`. But yea, I will keep studying and working on this

Speaking of `action_default_fallback`. Please do you know of suggested way to improve the model to solve this error? Testing the same path I wrote in my stories, (for most), the last action returns action\_default\_fallback. I have tried playing with the no of epochs, no revolve still.

Looking at the logs for one of the stories, I see

```auto
2021-07-29 02:10:53 DEBUG rasa.core.policies.ted_policy - TED predicted 'utter_not_sure' based on user intent.

```

(which is the action I have specified).

looking further, I also see:

```auto
2021-07-29 02:10:53 DEBUG rasa.core.processor - Predicted next action 'action_default_fallback' with confidence 0.30.

```

Wondering why the latter was chosen.

**EDIT:**** I have specified entity type for my story (as below) and I have it working now:**

```auto
- intent: specify_location_lockdown
    entities:
    - location: "test_location"

```

Additionally, I must add that you have been very helpful. Just so I don’t to forget to mention.

---

<div class="post-metadata">

**Author:** ![laboratory](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@laboratory](https://forum.rasa.com/u/laboratory)\
**Post date:** [July 29, 2021, 12:06am UTC](https://forum.rasa.com/t/failed-stories/46003/30 "2021-07-29T00:06:10Z")

</div>

Hi @anca wondering if you got my reply above.

[Next page](https://forum.rasa.com/t/failed-stories/46003.md?page=2)