NLU test for the two same config files give me different results

[endreb (endre)]

Hi There,

I have the exact same config files. config2.yml and config3.yml. And I run the following command: rasa test nlu --nlu data/nlu.yml --config config2.yml config3.yml

How come I get different results for the two different config files?

My config files: language: en

pipeline:

• name: WhitespaceTokenizer
• name: LexicalSyntacticFeaturizer
• name: CountVectorsFeaturizer
• name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4
• name: DIETClassifier epochs: 100 constrain_similarities: True model_confidence: linear_norm
• name: EntitySynonymMapper

policies:

• name: MemoizationPolicy max_history: 15
• name: RulePolicy
• name: TEDPolicy max_history: 15 epochs: 100 constrain_similarities: True model_confidence: linear_norm

Endre

[ChrisRahme (Chris Rahmé)]

It’s because randomness is used in machine learning.

If you really want to compare, you should set random_seed: 1 (or any number) under DIET, ResponseSelector, and TED. Example:

- name: DIETClassifier
  epochs: 100
  model_confidence: linear_norm
  constrain_similarities: true
  random_seed: 1

[endreb (endre)]

Thank you for the info!

[ChrisRahme (Chris Rahmé)]

Happy to help!

Please test again and tell me if it works or not

[endreb (endre)]

@ChrisRahme

random_seed: 1

The two tests are now exactly the same! Thanks for the help once again!

[ChrisRahme (Chris Rahmé)]

Glad to know! Please mark my answer as solution to mark the thread as closed