1 of 9 retrieval_intent has a very low accuracy r_acc 0.045 while the others are fine

Rasa Open Source
1

1 of 9 retrieval_intent has a very low accuracy r_acc 0.045 while the others are fine.

sentiment_1 goes along also with the most data

We use ResponseSelector with Rasa v2. Quite many training data but with 400+ epochs it works well. Now, however, one of the 6 retrieval_intent classes does not generate any accuracy if we do the complete training. However, in a test setting, when training data for the other retrieval_intent classes are removed, the training for the same group works like a charm.

Why do those training data of the different ResponseSelector retrieval_intent groups influence each other?

pipeline:
  - name: WhitespaceTokenizer
    intent_tokenization_flag: False
    intent_split_symbol: "_"
    case_sensitive: False
  - name: RegexFeaturizer
    alias: "regex"
    case_sensitive: False
    use_lookup_tables: True
    use_regexes: True
  - name: LexicalSyntacticFeaturizer
    alias: "lexical"
  - name: CountVectorsFeaturizer
    alias: "cvf_word"
  - name: CountVectorsFeaturizer
    case_sensitive: False
    alias: "cvf_char"
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    random_seed: 42
    constrain_similarities: True
    epochs: 11
    case_sensitive: False
  - name: EntitySynonymMapper
  - name: ResponseSelector
    retrieval_intent: sentiment_1
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    constrain_similarities: True
    epochs: 400
  - name: ResponseSelector
    retrieval_intent: sentiment_2
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    epochs: 300
  - name: ResponseSelector
    retrieval_intent: sentiment_3
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    constrain_similarities: True
    epochs: 245
  - name: ResponseSelector
    retrieval_intent: sentiment_4
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    constrain_similarities: True
    epochs: 265
  - name: ResponseSelector
    retrieval_intent: sentiment_5
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    epochs: 300
  - name: ResponseSelector
    retrieval_intent: sentiment_6
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    epochs: 245
  - name: ResponseSelector
    retrieval_intent: sentiment_7
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    epochs: 255
  - name: ResponseSelector
    retrieval_intent: sentiment_8
    use_text_as_label: False
    ranking_length: 6
    scale_loss: False
    epochs: 50
  - name: ResponseSelector
    retrieval_intent: sentiment_9
    use_text_as_label: False
    case_sensitive: False
    epochs: 50
2

I had quite a complex lookup-tables structure with the training data and entities aligned. I tried to remove all entities and look-ups from the training data. Now I am able to do the training appropriately for all ReponseSelectors.

3

Still struggling. In the mean time I tried out different random_seed values for each ResponseSelector. It seems that this somehow worked… But not sure if this was the reason. Also I removed randomly some training examples.