Hello all, I was trying to run an evaluation using cross-validation, but I’m getting the following errors during each fold. Has someone run into this before and how does one resolve it? /opt/venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior. For additional information, I’ve got an NLU-only model and I’m only using it for intent classification (no entities yet). I’m using DIET in my config (see below) - # Current pipeline configuration for the NLU language: "en" pipeline: - name: ConveRTTokenizer intent_tokenization_flag: true # Set to true to spot multiple intents case_sensitive: False intent_split_symbol: "+" - name: ConveRTFeaturizer - name: RegexFeaturizer - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer - name: CountVectorsFeaturizer analyzer: "char_wb" min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 - name: EntitySynonymMapper - name: ResponseSelector epochs: 100 # Configuration for Rasa Core. # https://rasa.com/docs/rasa/core/policies/ policies: - name: MemoizationPolicy - name: TEDPolicy - name: MappingPolicy Finally, this error appears before generating the confusion matrix. Would also appreciate some help here - /opt/venv/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray Many thanks for your help!

Hi Ganesh, as for your first question, you can choose to ignore the warning. As pointed out also on Stack Overflow , this simply means that your classifier (in this case DIET) didn’t use some of the labels during prediction. In most cases, this will mean that the classifier made some mistakes, but nothing more. If you’re curious, the warning typically originates from here , and the function calls leading to it are in your case probably: rasa.nlu.test.cross_validate > rasa.nlu.test.combine_result > rasa.nlu.test.compute_metrics > rasa.nlu.test._compute_metrics > rasa.test.get_evaluation_metrics As for the second warning, by the nature of it (a warning, not an error) I’d say it’s safe to ignore too. However, if you want to dig into it and see where the warning is coming from, I recommend looking at this Stack Overflow answer . Let me know if you’ve got any further questions.

Precision and F-score related errors from sklearn during cross-validation

Rasa Open Source

ganeshv (Ganesh Iyer) September 25, 2020, 9:39am 3

Thank you @SamS! I’ll try this again once I’m unblocked on running rasa.

Topic		Replies	Views
UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. Rasa Open Source	2	1349	August 29, 2019
Is there any way to check incorrect prediction while cross validation? Rasa Open Source	0	448	May 8, 2019
ISSUE WHILE TRAINING - Everything freezes Rasa Open Source	24	1928	February 19, 2020
Confidence Mismatch Rasa Open Source	10	1038	September 4, 2018
Adding Larger Lookup Table Causes Ill Defined F-Scores Rasa Open Source	7	885	May 19, 2019

Precision and F-score related errors from sklearn during cross-validation

Related topics