# TED policy predict right action with low confidence

**URL:** <https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280>\
**Category:** Rasa Open Source\
**Created:** [June 4, 2021, 2:09pm UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280 "2021-06-04T14:09:47Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![annthehuman](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/annthehuman/32/15303_2.png) [@annthehuman](https://forum.rasa.com/u/annthehuman)\
**Post date:** [June 4, 2021, 2:09pm UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280/1 "2021-06-04T14:09:47Z")

</div>

Hi!

My problem is: when I add new stories to my stories stack, TED predicts right next actions but with confidence lower whan 0.4. And it causes action\_default\_fallback (I established core\_fallback\_threshold: 0.4)

In examples:

1. When I train model with stories number.yml, pass.yml, service.yml, specification.yml prediction works great
2. When I train model with all of the previous stories and debt.yml in addition, TED predicts next action for intent ask\_for\_specification right, but with low confidence:  
`2021-06-04 16:34:17 DEBUG rasa.core.policies.ted_policy - TED predicted 'utter_ask_for_specification_old_customer' based on user intent with confidence 0.29507872462272644`

[pass.yml](https://forum.rasa.com/uploads/short-url/j7ARZnlnAwMOcrumbYNfPi1pqtG.yml) (2.6 KB) [debt.yml](https://forum.rasa.com/uploads/short-url/vgmxlBWFN1aHJpadakTClESxdkj.yml) (17.0 KB) [number.yml](https://forum.rasa.com/uploads/short-url/mK8qWk8zimBUuPqXI61E11J0UIq.yml) (2.0 KB) [service.yml](https://forum.rasa.com/uploads/short-url/bKLDQqkWYkZUyhFjg0KdAcJio9o.yml) (1.3 KB) [specification.yml](https://forum.rasa.com/uploads/short-url/n4sqi4QLsLOgpK3qYwldNGAO8wP.yml) (942 Bytes)  
[domain.yml](https://forum.rasa.com/uploads/short-url/qzsAjtZmZ5n2lbYP6Gnq7OvjF8U.yml) (4.0 KB) [config.yml](https://forum.rasa.com/uploads/short-url/yWRTVkhgOjNmdTdDhPOfgYJXpKi.yml) (563 Bytes)

I use separated NLU and Action Server and connect to them by API. Here I provide small version of Action Server, just to illustrate stories befaviour. [actions.py](https://forum.rasa.com/uploads/short-url/eZCDJNfNXShVIJwOdPiUmLfps9X.py) (6.9 KB)

Why these debt.yml stories affects the other sories? And maybe I need to use some specific configurations for TED Policy to train?

---

<div class="post-metadata">

**Author:** ![hsm207](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/hsm207/32/2647_2.png) [@hsm207](https://forum.rasa.com/u/hsm207)\
**Post date:** [June 7, 2021, 7:06am UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280/2 "2021-06-07T07:06:26Z")

</div>

Hello!

I see you set the number of epochs to train the TEDPolicy to 12. Could you try with 100 and see if that improves it’s performance?

Could you also share the output of `rasa --version`?

---

<div class="post-metadata">

**Author:** ![annthehuman](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/annthehuman/32/15303_2.png) [@annthehuman](https://forum.rasa.com/u/annthehuman)\
**Post date:** [June 7, 2021, 11:44am UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280/3 "2021-06-07T11:44:16Z")

</div>

rasa --version is

```
Rasa Version : 2.6.2
Minimum Compatible Version: 2.6.0
Rasa SDK Version : 2.6.0
Rasa X Version : None
Python Version : 3.6.8
Operating System : Windows-10-10.0.17763-SP0

```

I tried training with 100 epochs, and now TED predicts wrong next action in some cases.

For example, for `intent: ask_for_specification` and slots

```
  - old-customer: True
  - segment: customer

```

it predicts `utter_ask_for_specification_old_business` with confidence 0.47366… instead of `utter_ask_for_specification_old_customer`.

And several more cases like that, TED confuses between `segment: customer` and `segment: business`

---

<div class="post-metadata">

**Author:** ![j.mosig](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/j.mosig/32/3588_2.png) [@j.mosig](https://forum.rasa.com/u/j.mosig)\
**Post date:** [June 8, 2021, 5:44am UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280/4 "2021-06-08T05:44:22Z")

</div>

Hello @annthehuman

This looks like your stories might be in conflict with each other. What do you get when you run `rasa data validate` and `rasa data validate stories --max-history X`? (Here `X` is the `max_history` that you’ve set for TED.)

---

<div class="post-metadata">

**Author:** ![annthehuman](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/annthehuman/32/15303_2.png) [@annthehuman](https://forum.rasa.com/u/annthehuman)\
**Post date:** [June 8, 2021, 11:36am UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280/5 "2021-06-08T11:36:37Z")

</div>

Hi!

I tried `rasa data validate` and `rasa data validate stories` (without max\_history, cause I didnt specified it in config), and I found contradicting stories. Now I fixed them, and new debt.yml file is [debt.yml](https://forum.rasa.com/uploads/short-url/rLAxtruYblURwqeMIhZOMr3nT1I.yml) (17.1 KB)

But the problem stayed,TED still fails to predict right new action, or predicts it with low confidence.

---

<div class="post-metadata">

**Author:** ![j.mosig](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/j.mosig/32/3588_2.png) [@j.mosig](https://forum.rasa.com/u/j.mosig)\
**Post date:** [June 10, 2021, 5:18pm UTC](https://forum.rasa.com/t/ted-policy-predict-right-action-with-low-confidence/44280/6 "2021-06-10T17:18:17Z")

</div>

Hi @annthehuman

I just trained and tested with your data on Rasa 2.6.2 (I replaced the debt.yml with the new one you provided). TED gets all actions right upon testing on my machine, but the circuit breaker trips. This happens when you let rasa predict more than 10 actions in a row (no user messages in between). You can fix this by setting the environment variable:

```auto
export MAX_NUMBER_OF_PREDICTIONS=20

```

before running `rasa test core`.
