Hello all, I was trying to run an evaluation using cross-validation, but I’m getting the following errors during each fold. Has someone run into this before and how does one resolve it? /opt/venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior. For additional information, I’ve got an NLU-only model and I’m only using it for intent classification (no entities yet). I’m using DIET in my config (see below) - # Current pipeline configuration for the NLU language: "en" pipeline: - name: ConveRTTokenizer intent_tokenization_flag: true # Set to true to spot multiple intents case_sensitive: False intent_split_symbol: "+" - name: ConveRTFeaturizer - name: RegexFeaturizer - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer - name: CountVectorsFeaturizer analyzer: "char_wb" min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 - name: EntitySynonymMapper - name: ResponseSelector epochs: 100 # Configuration for Rasa Core. # https://rasa.com/docs/rasa/core/policies/ policies: - name: MemoizationPolicy - name: TEDPolicy - name: MappingPolicy Finally, this error appears before generating the confusion matrix. Would also appreciate some help here - /opt/venv/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray Many thanks for your help!

Precision and F-score related errors from sklearn during cross-validation

ganeshv (Ganesh Iyer) September 25, 2020, 9:39am 3

Thank you @SamS! I’ll try this again once I’m unblocked on running rasa.

Topic		Replies	Views
ISSUE WHILE TRAINING - Everything freezes Rasa Open Source	24	1832	February 19, 2020
UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. Rasa Open Source	2	1295	August 29, 2019
UndefinedMetricWarning: F score is ill-defined and being set to 0.0 in labels with no predicted samples Rasa Open Source	0	1150	April 24, 2019
Confidence Mismatch Rasa Open Source	10	983	September 4, 2018
Getting Warning in diet_classifier.py Rasa Open Source	7	1655	September 29, 2020