According to the docs it is possible to evaluate different pipeline configurations and compare them afterwards. Currently, I guess we can only change the rasa components and make an evaluation.
However sometimes it´s neccessary to use different pretrained models for the different configuration files.
For example the first configuration should use the German_MD Model and for the second yml file should use the German BERT Model.
We can only use one spaCy model at a time and we have to link the model before using. So if I want to run an evaluation with
rasa test nlu --config ./test_config/config.yml ./test_config/config3.yml --runs 1 --percentages 25 70
I get the error message:
2020-03-24 08:42:03 WARNING rasa.nlu.test - Training model 'config3' failed. Error: Model 'de-trf-bertbasecased-lgte' is not a linked spaCy model. Please download and/or link a spaCy model, e.g. by running:
Is there any workaround I can use to evaluate different NLU pipelines with different pretrained models? If not, are you planing to provide any functions to support this?
I think it´s very important to compare the performance of the bot using different pretrained models.