TEDPolicy error after running rasa test but story test success

Hello everyone!

I am running the stories tests (rasa test) that I have written and despite the fact that the output shows me that everything has passed OK (or so I think), I see that the TEDPolicy_errors.json file shows two texts that do not predict entity, as an example, this would be one of them (the others are related to this same user input)

If the language is a problem, I have no problem changing it to English, I put it like this (Spanish) since for now the development is in Spanish, but I plan to change it in the future.

     "text": "quiero hacer reserva para 4 comensales",
     "entities": [
         "start": 26,
         "end": 27,
         "value": "4",
         "entity": "comensales"
     "predicted_entities": []

The story test is:

   - story: Reserva mesa deny path 1
       - user: |
           quiero hacer reserva para [4](comensales) comensales
         intent: hacer_reserva_comensales
           - comensales: "4"
       - slot_was_set:
           - comensales: "4"
       - action: ValidateComensales
       - slot_was_set:
           - comensales: "4"
       - slot_was_set:
           - mesa_disponible: true
       - action: utter_ask_reservar_mesa
       - user: |
         intent: deny
       - action: ReservarMesa
       - slot_was_set:
           - comensales: null
       - slot_was_set:
           - mesa_reservada: null
       - slot_was_set:
           - mesa_disponible: null

The intent (nlu.yml file): (I know I can put this a bit better but right now I’m just looking for basic functionality)

 - intent: hacer_reserva_comensales
     examples: |
       - quiero hacer reserva para [1](comensales) persona
       - quiero hacer reserva para [2](comensales) personas
       - quiero hacer reserva para [3](comensales) personas
       - quiero hacer reserva para [4](comensales) personas
       - quiero hacer reserva para [5](comensales) personas
       - reserva para [1](comensales) persona
       - reserva para [2](comensales) personas
       - reserva para [3](comensales) personas
       - reserva para [4](comensales) personas
       - reserva para [5](comensales) personas
       - reserva mesa [1](comensales) persona
       - reserva mesa [2](comensales) personas
       - reserva mesa [3](comensales) personas
       - reserva mesa [4](comensales) personas
       - reserva mesa [5](comensales) personas

The domain.yml file:

   - comensales
     type: text
     influence_conversation: true
       - type: from_entity
         entity: comensales

The story test passes, but I don’t quite understand the problem, or at least, what I would understand is that the value “4” is not being recognized with the entity “comensales” even though the story itself is functional (both rasa shell and rasa interactive show correct values for this slot/entity)

2022-07-08 13:43:21 INFO     rasa.core.test  - Finished collecting predictions.
2022-07-08 13:43:21 INFO     rasa.core.test  - Evaluation Results on CONVERSATION level:
2022-07-08 13:43:21 INFO     rasa.core.test  - 	Correct:          7 / 7
2022-07-08 13:43:21 INFO     rasa.core.test  - 	Accuracy:         1.000
2022-07-08 13:43:22 INFO     rasa.core.test  - Evaluation Results on ACTION level:
2022-07-08 13:43:22 INFO     rasa.core.test  - 	Correct:          29 / 29
2022-07-08 13:43:22 INFO     rasa.core.test  - 	F1-Score:         1.000
2022-07-08 13:43:22 INFO     rasa.core.test  - 	Precision:        1.000
2022-07-08 13:43:22 INFO     rasa.core.test  - 	Accuracy:         1.000
2022-07-08 13:43:22 INFO     rasa.core.test  - 	In-data fraction: 0
  • DIETClassifier_confusion_matrix.png has only the diagonal line
  • DIETClassifier_histogram.png has no wrong bars
  • DIETClassifier_report.json has not “confused_with” match and all “precision” values are 1.0
  • failed_test_stories.yml is empty (all good)
  • intent_confusion_matrix.png has only the diagonal line
  • intent_histogram.png has no wrong bars
  • intent_report.json has not “confused_with” match and all “precision” values are 1.0
  • stories_with_warnings.yml is empty (all good)
  • story_confusion_matrix.png has only the diagonal line
  • story_report.json has all “precision” values to 1.0

And then, the error files:

TEDPolicy_errors.json (454 Bytes) TEDPolicy_report.json (469 Bytes)

Here comes the question: am I writing the user input correctly in the test? Looking at the documentation, I see these two input examples (adapted to my entity):

  • Testing Your Assistant
  • Example 1:
    • [4]{“entity”: “comensales”} (same result as 4
  • Example 2:
    • [4][{\“entity\”: \“comensales\”}] (this directly gives the following error running the command)

InvalidEntityFormatException: Incorrect training data format (‘{\“entity\”: \“comensales\”}’). More info at Training Data Format line 1 column 2 (char 1)

I can’t put a link to the project on GitHub because until I defend the development it has to be private (university stuff), but if you require something specific I could share it

Regards, Raúl.

1 Like

Bumping this thread, can someone assist?

I have exactly the same issue, with upgrading to RASA 3, now all the stories that have entity examples in the sentences all fall into ted errors.

It’s true, I didn’t put the version, I also use rasa 3, more exactly rasa 3.1.1

# rasa --version
Rasa Version      :         3.1.1
Minimum Compatible Version: 3.0.0
Rasa SDK Version  :         3.1.1
Rasa X Version    :         None
Python Version    :         3.7.7
Operating System  :         Linux-5.17.15-76051715-generic-x86_64-with-debian-bookworm-sid

Hello @Juste

I ping you since you sent me the notification of possible spam in a preventive way with its consequent resolution of not being spam, it would be good if someone could take a look at this and it will not be forgotten, I still have this problem and I think that other users too

Regards, Raul.

I am also having the same Issue. Did you get any solution.

hello @Monica1 , I keep trying to find a solution due to the lack of response, if I find it I will publish it here.

In the worst-case scenario, you can implement the tests yourself. I know it wouldn’t make much sense, but you can do inference using a trained Rasa model and then create an evaluator and compute these metrics on your own.

Hi, I’m also facing this problem. Did you find a solution @rauldpm ?

Hi @pmj , sorry for the late reply, I didn’t see the notification.

No, unfortunately I did not find a solution and I stopped working on this when I finished my TFG, I would recommend you open another post in case they pay more attention to you than to me, good luck!