Rasa train unreasonable stop

Rasa Open Source
1

use rasa train to train the model, but sometimes the training process stoped unreasonable, without error output. Here is the screenshot of the cmd

SharedScreenshot
SharedScreenshot1137×392 86.3 KB
my domain # Configuration for Rasa NLU.

Components

language: “zh”

pipeline:

  • name: “nlp_mitie” model: “data/total_word_feature_extractor_zh.dat”
  • name: “tokenizer_jieba”
  • name: “ner_mitie”
  • name: “ner_synonyms”
  • name: “intent_entity_featurizer_regex”
  • name: “intent_featurizer_mitie”
  • name: “intent_classifier_sklearn”

Configuration for Rasa Core.

Policies

policies:

  • name: MemoizationPolicy
  • name: KerasPolicy
  • name: MappingPolicy

anyone can help me? thanks a lot!

2

Hi @SnowAsh, thank you for your question.

What do you mean by “sometimes”? What changes between instances where it works and instances where it stops? Is it random? And how long do you wait for? Note, that training can take a long time if you have a lot of data.

3

Thanks for your reply. The situation appears ramdomly, sometimes the train completed successfully,but sometimes it failed, there are no rules in the process. But when I change the keraspolicy into sklearnpolicy ,the problem vanished. I don’t know whether the changed policy has any negative effect on my model. So is it a keras bug? how can I fix it?

4

Could you fix the random seed and see if it is random anyway? If so, you might suffer from a hardware problem (overheating that slows down your machine), or the OS is doing something strange. If, on the other hand, fixing the seed fixes the problem (or always triggers the problem), then something might be wrong in the algorithm. What Rasa version do you use?

5

Thank you for you help! I use the rasa 1.5.1,and I don’t know how to fix random seed in a training process…,should I check the keras source code?

6

Many policies have a “random_seed” option, e.g. rasa/embedding_policy.py at dccd2ef76651025bd1e22ecb45a3cb0e0dd47b9a · RasaHQ/rasa · GitHub. You can set it in the config file (see Policies).