Failed stories

Rasa Open Source
31

@laboratory I just tried replicating your issue and I didn’t get any failed test stories. This is what I did:

  • use rasa init to create bot template
  • add 2 new intents (check_lockdown; specify_location) and two entities (location, postcode)
  • add a similar custom action to what yours does (action_check_lockdown):
class ActionCheckLockdown(Action):

    def name(self) -> Text:
        return "action_check_lockdown"

    def run(self, dispatcher: CollectingDispatcher,
            tracker: Tracker,
            domain: Dict[Text, Any]) -> List[Dict[Text, Any]]:

        entity = next(tracker.get_latest_entity_values("location"), None)
        if entity:
            dispatcher.utter_message(text="Yes, I can check this area for you, please specify exact address.")
        else:
            dispatcher.utter_message(text="Please specify exact address.")

        return []
  • add a story in stories.yml; nothing added to rules.yml 
- story: handle lockdown check
  steps:
    - intent: check_lockdown
    - action: action_check_lockdown
    - intent: specify_location
    - action: action_check_lockdown
  • write two additional test stories
- story: lockdown + location found
  steps:
    - user: |
        I would like to check lockdown rules in [Islington]{"entity": "location"}.
      intent: check_lockdown
    - action: action_check_lockdown
    - user: |
        This is my postcode: [SW1W 9AX]{"entity": "postcode"}.
      intent: specify_location
    - action: action_check_lockdown

- story: lockdown + location not found
  steps:
    - user: |
        I would like to check lockdown rules in my area.
      intent: check_lockdown
    - action: action_check_lockdown
    - user: |
        This is my postcode: [SW1W 9AX]{"entity": "postcode"}.
      intent: specify_location
    - action: action_check_lockdown

I will aim to follow-up to your question on Fallbacks in the next few hours, that’s an entirely different topic.

Please also note we aim to provide a response usually within a 3-day time window, so I would ask for a bit of patience.

32

Speaking of action_default_fallback. Please do you know of suggested way to improve the model to solve this error?

You can tweak the confidence threshold at which action_default_fallback gets predicted by following the instructions here. If this threshold is higher than other predicted actions’ confidences, then action_default_fallback will be predicted and executed.

33

@anca Thanks a lot. I editid my last post but didn’t save it. But here it is:

Basically, I didn’t specify the entities in the story.

Thanks a lot

1 Like
34

Maybe we could go back to the initial question as I am facing the same issue. Latest test results are 3 of 30 test stories are marked as correct, 27 can be found in failed_test_stories, but only 1 story has been marked with the # hashtag.

What might be a good explanation for that?

BR ANdre

35

Yes please explain what @AndreD mentioned. failed_test_stories is only useful if it tells you what failed so what is meant by a non-empty failed_test_stoires with no # indicating the issues?

36

@anca Can you explain this?