Evaluate NLU models with different pretrained models

According to the docs it is possible to evaluate different pipeline configurations and compare them afterwards. Currently, I guess we can only change the rasa components and make an evaluation.

However sometimes it´s neccessary to use different pretrained models for the different configuration files.

For example the first configuration should use the German_MD Model and for the second yml file should use the German BERT Model.

We can only use one spaCy model at a time and we have to link the model before using. So if I want to run an evaluation with

rasa test nlu --config ./test_config/config.yml ./test_config/config3.yml --runs 1 --percentages 25 70

I get the error message:

2020-03-24 08:42:03 WARNING rasa.nlu.test - Training model 'config3' failed. Error: Model 'de-trf-bertbasecased-lgte' is not a linked spaCy model. Please download and/or link a spaCy model, e.g. by running:

Is there any workaround I can use to evaluate different NLU pipelines with different pretrained models? If not, are you planing to provide any functions to support this?

I think it´s very important to compare the performance of the bot using different pretrained models.

Hi @lindig,

the problem you posted wasn’t caused by Rasa. Currently it is possible to evaluate different configs. I suggest using e.g. this command:

rasa test nlu -u data/nlu.md -c config1.yml config2.yml config3.yml --cross-validation -f 5 -r 3 --percentages 25 70

Besides, the error means that you did not install de-trf-bertbasecased-lgte. Did you perform a pip install dist/de_trf_bertbasecased_lgte-1.0.0.tar.gz ?

Kind regards

Hi @JulianGerhard ,

I made a typo in the configuration file, German BERT model is installed as de-trf-bertbasecased-lg. Sorry for that!