ISSUE WHILE TRAINING - Everything freezes

Rasa Open Source
1

Hi,

while I train my dataset I get this two warnings:

“UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. ‘precision’, ‘predicted’, average, warn_for)”

"c:\users\tjcol\appdata\local\programs\python\python37\lib\site-packages\sklearn\metrics\classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. ‘precision’, ‘predicted’, average, warn_for) "

After I get them, the training freezes and I’m not able to train my model.

Please help. :slight_smile:

2

this warning can occur during testing not training

3

Well for me this occurs after I run the line “rasa train”, so I don’t know how it is not possible then?

4

I tried now with a bit smaller dataset but I still have the same problem, could someone look at my dataset maybe there is a problem with it that I’m overlooking?

5

Hi rasafan,

have your tried to use a virtual environment with e.g. python 3.6.8 ?

I am suggestion this because the warning you see is familiar to me and usually doesnt lead to a freeze. However I experienced quite different problems with a python version equal and greater to 3.7. Maybe this is worth a try.

If that doesnt lead to a working system, feel free to publish your dataset / zip the bot such that I can test it.

Regards Julian

6

Hi,

yeah we tried that too, still doesn’t work.

rasa_data.zip (39.4 KB)

I’ve uploded our dataset, can you please try it out? Thanks.

7

@rasafan it could be a memory error

8

How can I solve that?

9

We’re working on the new version that will significantly reduce amount of memory required. for now, could you please try the following nlu pipeline:

language: "en"

pipeline:
- name: "WhitespaceTokenizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "CountVectorsFeaturizer"
- name: "EmbeddingIntentClassifier"
  batch_strategy: sequence
10

Hi, that training is with supervised embedding and that works for us all the time.

We tried now your pipeline and got a model with 93% accuracy that works poorly, it doesn’t understand anything that isn’t literally written as it is in dataset.

We are trying to train on pretrained embedding option and that doesn’t work for us, check it out:

  • language: “en”
  • pipeline: “pretrained_embeddings_spacy”
  • policies:
  • epochs: 75
  • max_history: 10
  • name: KerasPolicy
  • max_history: 10
  • name: AugmentedMemoizationPolicy
  • name: “FallbackPolicy”
  • nlu_threshold: 0.2
  • core_threshold: 0.1
  • fallback_action_name: “action_default_fallback”

In addition, when I trained now once again with this pipeline I check the memory conditions when it freezes and it was around 1.5GB and my computer has 12GB of memory.

11

sorry, I didn’t see that you use spacy pipeline. In this case it is not a memory issue

12

Yeah I understand, do you know what should I do? Do you know anyone who could assist?

@JulianGerhard, did you maybe found the time to check out our dataset and test it?

13

Any ideas?

14

did you debug your code to find out the spot where it freezes?

15

@Ghostvv

How do I debug training with Rasa since that is on the Rasa side of the code? I googled and haven’t found how to debug the training part anywhere…

16

did you run in --debug mode? If, it doesn’t help, otherwise I’m afraid the solution is to clone GitHub repo, and use the source code

17

Can you please tell me step by step how to debug the source code after cloning it? I cannot find any info on to how to do that when I’m using it in another project as a dependency. Thank you.

18

You can find here more people with the same error:

19

I would suggest to run your training locally with rasa installed from the cloned repo, then debug the parts of code of rasa that are executed during training

20

I’ve done that and my error happens while running rasa train nlu --debug since it doesn’t offer any insight in to the freezing error as there are no additional information logged while training.

Training still freezes on: Fitting 3 folds for each of 6 candidates, totalling 18 fits [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.