# ISSUE WHILE TRAINING - Everything freezes **URL:** \ **Category:** Rasa Open Source\ **Created:** [October 31, 2019, 5:00pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740 "2019-10-31T17:00:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [October 31, 2019, 5:00pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/1 "2019-10-31T17:00:57Z")
Hi, while I train my dataset I get this two warnings: “UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. ‘precision’, ‘predicted’, average, warn\_for)” "c:\users\tjcol\appdata\local\programs\python\python37\lib\site-packages\sklearn\metrics\classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. ‘precision’, ‘predicted’, average, warn\_for) " After I get them, the training freezes and I’m not able to train my model. Please help. 🙂 ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 1, 2019, 9:23am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/2 "2019-11-01T09:23:36Z")
this warning can occur during testing not training ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 1, 2019, 4:11pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/3 "2019-11-01T16:11:27Z")
Well for me this occurs after I run the line “rasa train”, so I don’t know how it is not possible then? ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 2, 2019, 10:28am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/4 "2019-11-02T10:28:42Z")
I tried now with a bit smaller dataset but I still have the same problem, could someone look at my dataset maybe there is a problem with it that I’m overlooking? ---
**Author:** ![JulianGerhard](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/juliangerhard/32/3781_2.png) [@JulianGerhard](https://forum.rasa.com/u/JulianGerhard)\ **Post date:** [November 2, 2019, 2:32pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/5 "2019-11-02T14:32:54Z")
Hi rasafan, have your tried to use a [virtual environment](https://timmyreilly.azurewebsites.net/python-pip-virtualenv-installation-on-windows/) with e.g. python 3.6.8 ? I am suggestion this because the warning you see is familiar to me and usually doesnt lead to a freeze. However I experienced quite different problems with a python version equal and greater to 3.7. Maybe this is worth a try. If that doesnt lead to a working system, feel free to publish your dataset / zip the bot such that I can test it. Regards Julian ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 3, 2019, 6:36pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/6 "2019-11-03T18:36:43Z")
Hi, yeah we tried that too, still doesn’t work. [rasa\_data.zip](https://forum.rasa.com/uploads/short-url/uz5Hk8bFd9432261Mp0BvFhT62F.zip) (39.4 KB) I’ve uploded our dataset, can you please try it out? Thanks. ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 4, 2019, 11:36am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/7 "2019-11-04T11:36:21Z")
@rasafan it could be a memory error ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 4, 2019, 2:53pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/8 "2019-11-04T14:53:45Z")
How can I solve that? ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 5, 2019, 8:20am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/9 "2019-11-05T08:20:49Z")
We’re working on the new version that will significantly reduce amount of memory required. for now, could you please try the following nlu pipeline: ```auto language: "en" pipeline: - name: "WhitespaceTokenizer" - name: "RegexFeaturizer" - name: "CRFEntityExtractor" - name: "EntitySynonymMapper" - name: "CountVectorsFeaturizer" - name: "EmbeddingIntentClassifier" batch_strategy: sequence ``` ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 5, 2019, 9:07am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/10 "2019-11-05T09:07:07Z")
Hi, that training is with supervised embedding and that works for us all the time. We tried now your pipeline and got a model with 93% accuracy that works poorly, it doesn’t understand anything that isn’t literally written as it is in dataset. We are trying to train on pretrained embedding option and that doesn’t work for us, check it out: - language: “en” - pipeline: “pretrained\_embeddings\_spacy” - policies: - epochs: 75 - max\_history: 10 - name: KerasPolicy - max\_history: 10 - name: AugmentedMemoizationPolicy - name: “FallbackPolicy” - nlu\_threshold: 0.2 - core\_threshold: 0.1 - fallback\_action\_name: “action\_default\_fallback” In addition, when I trained now once again with this pipeline I check the memory conditions when it freezes and it was around 1.5GB and my computer has 12GB of memory. ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 5, 2019, 9:38am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/11 "2019-11-05T09:38:04Z")
sorry, I didn’t see that you use spacy pipeline. In this case it is not a memory issue ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 5, 2019, 10:21am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/12 "2019-11-05T10:21:06Z")
Yeah I understand, do you know what should I do? Do you know anyone who could assist? @JulianGerhard, did you maybe found the time to check out our dataset and test it? ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 7, 2019, 7:46pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/13 "2019-11-07T19:46:59Z")
Any ideas? ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 8, 2019, 9:25am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/14 "2019-11-08T09:25:30Z")
did you debug your code to find out the spot where it freezes? ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 8, 2019, 6:05pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/15 "2019-11-08T18:05:39Z")
@Ghostvv How do I debug training with Rasa since that is on the Rasa side of the code? I googled and haven’t found how to debug the training part anywhere… ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 11, 2019, 9:13am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/16 "2019-11-11T09:13:38Z")
did you run in `--debug` mode? If, it doesn’t help, otherwise I’m afraid the solution is to clone GitHub repo, and use the source code ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 12, 2019, 9:38pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/17 "2019-11-12T21:38:44Z")
Can you please tell me step by step how to debug the source code after cloning it? I cannot find any info on to how to do that when I’m using it in another project as a dependency. Thank you. ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 14, 2019, 10:18am UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/18 "2019-11-14T10:18:18Z")
You can find here more people with the same error: > > > Hi, > > when using spacy, rasa won't train the NLU model. It preoceeds until \`\[Pa…rallel(n\_jobs=1)\]: Using backend SequentialBackend with 1 concurrent workers.\` and then just stays like this forever. > I tried several operating systems and python versions, but the problem persists. I don't think it's an issue of "giving it more time". I think it just freezes. > > While I do have a lot of training data, supervised embeddings work just fine (and I would expect this to be slower). There is a thread about this issue on rasa forums which was however never resolved: https://forum.rasa.com/t/can-not-train-rasa-nlu-with-spacy-models/11374/3 > > \*\*Rasa version\*\*: > rasa==1.4.3 > rasa-nlu==0.14.3 > rasa-sdk==1.4.0 > (also happens with older versions) > > \*\*Python version\*\*: > 3.6.8 > (also happens with 3.7) > > \*\*Operating system\*\* (windows, osx, ...): > Both Windows 10 and Ubuntu 18.04 > > \*\*Issue\*\*: > Rasa won't train the NLU model when using spacy. It proceed's until > > \`Fitting 2 folds for each of 6 candidates, totalling 12 fits\` > \`\[Parallel(n\_jobs=1)\]: Using backend SequentialBackend with 1 concurrent workers.\` > > and then just stays like this forever. I left it like this for 12+ hours and nothing happened. > This happens with both \`en\_core\_web\_sm\` and \`en\_core\_web\_md\` > > > \*\*Command or request that led to error\*\*: > \`\`\` > rasa train > \`\`\` > > \*\*Content of configuration file (config.yml)\*\* (if relevant): > \`\`\` > \# Configuration for Rasa NLU. > \# https://rasa.com/docs/rasa/nlu/components/ > language: "en" > pipeline: > \- name: "SpacyNLP" > \- name: "SpacyTokenizer" > \- name: "SpacyFeaturizer" > \- name: "RegexFeaturizer" > \- name: "CRFEntityExtractor" > \- name: "EntitySynonymMapper" > \- name: "CountVectorsFeaturizer" > \- name: "CountVectorsFeaturizer" > analyzer: char\_wb > min\_ngram: 1 > max\_ngram: 4 > lowercase: true > \- name: "SklearnIntentClassifier" > timezone: "Europe/Berlin" > case\_sensitive: false > > \# Configuration for Rasa Core. > \# https://rasa.com/docs/rasa/core/policies/ > policies: > - name: "KerasPolicy" > featurizer: > - name: "MaxHistoryTrackerFeaturizer" > max\_history: 5 > state\_featurizer: > - name: "LabelTokenizerSingleStateFeaturizer" > - name: "MemoizationPolicy" > max\_history: 5 > - name: MappingPolicy > - name: FormPolicy > - name: "FallbackPolicy" > nlu\_threshold: 0.75 > core\_threshold: 0.45 > fallback\_action\_name: "action\_default\_ask\_rephrase" > \`\`\` ---
**Author:** ![Ghostvv](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/ghostvv/32/211_2.png) [@Ghostvv](https://forum.rasa.com/u/Ghostvv)\ **Post date:** [November 14, 2019, 12:40pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/19 "2019-11-14T12:40:38Z")
I would suggest to run your training locally with rasa installed from the cloned repo, then debug the parts of code of rasa that are executed during training ---
**Author:** ![rasafan](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rasafan](https://forum.rasa.com/u/rasafan)\ **Post date:** [November 16, 2019, 2:36pm UTC](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740/20 "2019-11-16T14:36:58Z")
I’ve done that and my error happens while running rasa train nlu --debug since it doesn’t offer any insight in to the freezing error as there are no additional information logged while training. Training still freezes on: Fitting 3 folds for each of 6 candidates, totalling 18 fits [Parallel(n\_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Next page](https://forum.rasa.com/t/issue-while-training-everything-freezes/20740.md?page=2)