NLU test for the two same config files give me different results

endreb · November 3, 2021, 2:28pm

Hi There,

I have the exact same config files. config2.yml and config3.yml. And I run the following command: rasa test nlu --nlu data/nlu.yml --config config2.yml config3.yml

How come I get different results for the two different config files?

My config files: language: en

pipeline:

name: WhitespaceTokenizer
name: LexicalSyntacticFeaturizer
name: CountVectorsFeaturizer
name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4
name: DIETClassifier epochs: 100 constrain_similarities: True model_confidence: linear_norm
name: EntitySynonymMapper

policies:

name: MemoizationPolicy max_history: 15
name: RulePolicy
name: TEDPolicy max_history: 15 epochs: 100 constrain_similarities: True model_confidence: linear_norm

Endre

ChrisRahme · November 4, 2021, 10:25am

It’s because randomness is used in machine learning.

If you really want to compare, you should set random_seed: 1 (or any number) under DIET, ResponseSelector, and TED. Example:

- name: DIETClassifier
  epochs: 100
  model_confidence: linear_norm
  constrain_similarities: true
  random_seed: 1

endreb · November 4, 2021, 11:09am

Thank you for the info!

ChrisRahme · November 4, 2021, 11:13am

Happy to help!

Please test again and tell me if it works or not

endreb · November 4, 2021, 12:12pm

The two tests are now exactly the same! Thanks for the help once again!

ChrisRahme · November 4, 2021, 12:56pm

Glad to know! Please mark my answer as solution to mark the thread as closed