Why does rasa_nlu.evaluate tests each examples two times?

Rasa Open Source
1

Hello, I’m trying to understand why when I evaluate my model, the rasa command go through my test fold two times, doubling the time it takes to complete.

Here’s the command launched :

python3 -m rasa_nlu.evaluate -d data/fr_init_data_set_test_2.json -m models/bert_feature_test_on_FR/nlu --report reports/bert_feature_test_on_FR --histogram reports/bert_feature_test_on_FR/hist.png --confmat reports/bert_feature_test_on_FR/confmat.png --errors reports/bert_feature_test_on_FR/errors.json --debug

When I take a look at my errors logs, I can see that each sentences from my test fold of 11 sentences appears two times, as if the evaluation was launch two times, one after the other.

My packages versions :

rasa-core==0.13.8
rasa-core-sdk==0.12.2
rasa-nlu==0.14.6

The logs doesn’t show anything suspicious between the last sentence of the first run on the test fold, and the first sentence of the second run on the test fold.

Is it supposed to happen ? Or do you have an idea of what may be going on ? I don’t think it’s what’s supposed to happen when we evaluate a model, as I doesn’t see the point of running the same prediction two time on the same examples without modifying the model in the meantime.

Thanks !

3

Could you try this with the latest Rasa version and see if the issue persists? Also, just to confirm, you’re seeing duplicate sentences in the errors.json file right?

4

I’ve tried to upgrade rasa_nlu to its latest version (`rasa-nlu==0.15.1 ̀), but I have to retrain the model, and I get this error at the end of the training :

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/aliednikova/medibot/venv/lib/python3.5/site-packages/rasa_nlu/train.py", line 173, in <module>
    num_threads=cmdline_args.num_threads)
  File "/home/aliednikova/medibot/venv/lib/python3.5/site-packages/rasa_nlu/train.py", line 149, in train
    fixed_model_name)
  File "/home/aliednikova/medibot/venv/lib/python3.5/site-packages/rasa_nlu/model.py", line 237, in persist
    update = component.persist(file_name, dir_name)
TypeError: persist() takes 2 positional arguments but 3 were given

I’m afraid I won’t be able to upgrade easily rasa_nlu.

And there is no duplicates, in all reports files (confmat.png, errors.json, intent_report.json and hist.png), just in the training logs.

5

This seems to be related: Duplicate output in log file · Issue #4113 · RasaHQ/rasa · GitHub

6

@DavidForlen The method signature for the latest version accepts only a folder name. The filename is metadata.json, so you can remove that argument, and it should be fixed.