Hello all, I was trying to run an evaluation using cross-validation, but I’m getting the following errors during each fold. Has someone run into this before and how does one resolve it? /opt/venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior. For additional information, I’ve got an NLU-only model and I’m only using it for intent classification (no entities yet). I’m using DIET in my config (see below) - # Current pipeline configuration for the NLU language: "en" pipeline: - name: ConveRTTokenizer intent_tokenization_flag: true # Set to true to spot multiple intents case_sensitive: False intent_split_symbol: "+" - name: ConveRTFeaturizer - name: RegexFeaturizer - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer - name: CountVectorsFeaturizer analyzer: "char_wb" min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 - name: EntitySynonymMapper - name: ResponseSelector epochs: 100 # Configuration for Rasa Core. # https://rasa.com/docs/rasa/core/policies/ policies: - name: MemoizationPolicy - name: TEDPolicy - name: MappingPolicy Finally, this error appears before generating the confusion matrix. Would also appreciate some help here - /opt/venv/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray Many thanks for your help!

Precision and F-score related errors from sklearn during cross-validation

ganeshv (Ganesh Iyer) September 25, 2020, 9:39am 3

Thank you @SamS! I’ll try this again once I’m unblocked on running rasa.

Topic		Replies	Views
UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. Rasa Open Source	2	1338	August 29, 2019
Is there any way to check incorrect prediction while cross validation? Rasa Open Source	0	443	May 8, 2019
ISSUE WHILE TRAINING - Everything freezes Rasa Open Source	24	1881	February 19, 2020
Confidence Mismatch Rasa Open Source	10	1014	September 4, 2018
Adding Larger Lookup Table Causes Ill Defined F-Scores Rasa Open Source	7	824	May 19, 2019