Trained model is very inconsistent

Rasa Open Source
1

Every time i train my model my chatbot is giving different replys. some times i tried forcing the model train using --force and results are exceptionally good. I’m not sure why this is happening. But sometimes it is failing to extract intents and entities which were extracted correctly in previous model. Here is our config.yml

language: en
pipeline:
  - name: "lib.sana_custom_components.SanaSpellCheckerEN"
  - name: "lib.sana_custom_components.SanaTrueCaseConvertor"
  - name: SpacyNLP
  - name: SpacyTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: MappingPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: FormPolicy
  - name: TwoStageFallbackPolicy
    nlu_threshold: 0.3
    core_threshold: 0.3
    fallback_core_action_name: "action_default_fallback"
    fallback_nlu_action_name: "action_default_fallback"
    deny_suggestion_intent_name: "out_of_scope"
2

Welcome Satya,

What version of Rasa are you using?

3 
rasa==1.9.6
rasa-sdk==1.9.0
4

Hi @SriSatyaLokesh to get reproducible training results, you should set a random_seed for some of the NLU components and policies. So your config would look like this:

language: en
pipeline:
  - name: "lib.sana_custom_components.SanaSpellCheckerEN"
  - name: "lib.sana_custom_components.SanaTrueCaseConvertor"
  - name: SpacyNLP
  - name: SpacyTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
    random_seed: 42
  - name: EntitySynonymMapper
  - name: ResponseSelector
    random_seed: 42
    epochs: 100

policies:
  - name: MemoizationPolicy
  - name: MappingPolicy
  - name: TEDPolicy
    random_seed: 42
    max_history: 5
    epochs: 100
  - name: FormPolicy
  - name: TwoStageFallbackPolicy
    nlu_threshold: 0.3
    core_threshold: 0.3
    fallback_core_action_name: "action_default_fallback"
    fallback_nlu_action_name: "action_default_fallback"
    deny_suggestion_intent_name: "out_of_scope"

Please be aware that if you have drastically inconsistent results though, that may be an indicator that your training data needs some improvements. So it may be worth investing some time into adding more examples to your NLU data

1 Like