Precision and F-score related errors from sklearn during cross-validation

Hello all, I was trying to run an evaluation using cross-validation, but I’m getting the following errors during each fold. Has someone run into this before and how does one resolve it?

/opt/venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior.

For additional information, I’ve got an NLU-only model and I’m only using it for intent classification (no entities yet). I’m using DIET in my config (see below) -

# Current pipeline configuration for the NLU
language: "en"

pipeline:
  - name: ConveRTTokenizer
    intent_tokenization_flag: true
    # Set to true to spot multiple intents
    case_sensitive: False
    intent_split_symbol: "+"
  - name: ConveRTFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
  - name: MappingPolicy

Finally, this error appears before generating the confusion matrix. Would also appreciate some help here -

/opt/venv/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray

Many thanks for your help!

Hi Ganesh, as for your first question, you can choose to ignore the warning. As pointed out also on Stack Overflow, this simply means that your classifier (in this case DIET) didn’t use some of the labels during prediction. In most cases, this will mean that the classifier made some mistakes, but nothing more. If you’re curious, the warning typically originates from here, and the function calls leading to it are in your case probably: rasa.nlu.test.cross_validate > rasa.nlu.test.combine_result > rasa.nlu.test.compute_metrics > rasa.nlu.test._compute_metrics > rasa.test.get_evaluation_metrics

As for the second warning, by the nature of it (a warning, not an error) I’d say it’s safe to ignore too. However, if you want to dig into it and see where the warning is coming from, I recommend looking at this Stack Overflow answer.

Let me know if you’ve got any further questions.

Thank you @SamS! I’ll try this again once I’m unblocked on running rasa.