TED policy predict right action with low confidence

Rasa Open Source
1

Hi!

My problem is: when I add new stories to my stories stack, TED predicts right next actions but with confidence lower whan 0.4. And it causes action_default_fallback (I established core_fallback_threshold: 0.4)

In examples:

  1. When I train model with stories number.yml, pass.yml, service.yml, specification.yml prediction works great
  2. When I train model with all of the previous stories and debt.yml in addition, TED predicts next action for intent ask_for_specification right, but with low confidence:
    2021-06-04 16:34:17 DEBUG rasa.core.policies.ted_policy - TED predicted 'utter_ask_for_specification_old_customer' based on user intent with confidence 0.29507872462272644

pass.yml (2.6 KB) debt.yml (17.0 KB) number.yml (2.0 KB) service.yml (1.3 KB) specification.yml (942 Bytes)
domain.yml (4.0 KB) config.yml (563 Bytes)

I use separated NLU and Action Server and connect to them by API. Here I provide small version of Action Server, just to illustrate stories befaviour. actions.py (6.9 KB)

Why these debt.yml stories affects the other sories? And maybe I need to use some specific configurations for TED Policy to train?

2

Hello!

I see you set the number of epochs to train the TEDPolicy to 12. Could you try with 100 and see if that improves it’s performance?

Could you also share the output of rasa --version?

3

rasa --version is

Rasa Version      :         2.6.2
Minimum Compatible Version: 2.6.0
Rasa SDK Version  :         2.6.0
Rasa X Version    :         None
Python Version    :         3.6.8
Operating System  :         Windows-10-10.0.17763-SP0

I tried training with 100 epochs, and now TED predicts wrong next action in some cases.

For example, for intent: ask_for_specification and slots

  - old-customer: True
  - segment: customer

it predicts utter_ask_for_specification_old_business with confidence 0.47366… instead of utter_ask_for_specification_old_customer.

And several more cases like that, TED confuses between segment: customer and segment: business

4

Hello @annthehuman

This looks like your stories might be in conflict with each other. What do you get when you run rasa data validate and rasa data validate stories --max-history X? (Here X is the max_history that you’ve set for TED.)

5

Hi!

I tried rasa data validate and rasa data validate stories (without max_history, cause I didnt specified it in config), and I found contradicting stories. Now I fixed them, and new debt.yml file is debt.yml (17.1 KB)

But the problem stayed,TED still fails to predict right new action, or predicts it with low confidence.

6

Hi @annthehuman

I just trained and tested with your data on Rasa 2.6.2 (I replaced the debt.yml with the new one you provided). TED gets all actions right upon testing on my machine, but the circuit breaker trips. This happens when you let rasa predict more than 10 actions in a row (no user messages in between). You can fix this by setting the environment variable:

export MAX_NUMBER_OF_PREDICTIONS=20

before running rasa test core.