TED policy test shows no output

Rasa Open Source
1

I have been running dozens of tests using rasa test but the results files are showing no outputs from the TED Policy. Other models’ results are fine. I don’t remember it working, even at the very beginning, using defaults for all initial rasa projects; just changing my domain and nlu and creating test conversations.

Why is this?

The confusion matrix shows that all entities are predicted as no_entity, TEDPolicy_confusion_matrix.png:

image
image1964×2196 276 KB

The TEDPolicy_errors.json file shows no predicted entities. The TEDPolicy_report.json file shows all scores are zero. I’m not even sure why TED is predicting entities, even when I set entity_recognition: false in the config.yml file. Why are there no actions being predicted? I thought that was it’s main purpose, not NER.

Using rasa 3.0.6.

2

@tomp can you please share the config.yml file as ref ?

3 
# The config recipe.
# https://rasa.com/docs/rasa/model-configuration/
recipe: default.v1

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
# See https://rasa.com/docs/rasa/tuning-your-model for more information.
#  - name: WhitespaceTokenizer
  - name: SpacyNLP
    model: "en_core_web_md"
  - name: SpacyTokenizer
    token_pattern: "(\\d+|[^\\s\\d\\W]+|[^\\w\\s]+)"
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
#    epochs: 200
    constrain_similarities: true
    random_seed: 42
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
    #epochs: 200  #  Mixed results
    constrain_similarities: true
    random_seed: 42
  #  Use nlu_fallback intent class if other intent confidences are lower than "threshold" (default 0.3)
  #  or not further apart than "ambiguity_threshold" (default 0.1).
  - name: FallbackClassifier
    threshold: 0.1
    ambiguity_threshold: 0.05
    random_seed: 42


# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
# See https://rasa.com/docs/rasa/policies for more information.
  - name: MemoizationPolicy
  - name: RulePolicy
  - name: UnexpecTEDIntentPolicy
    max_history: 5
    epochs: 100
    random_seed: 42
  - name: TEDPolicy
    max_history: 5
    epochs: 100
    constrain_similarities: true
    random_seed: 42
    entity_recognition: true
    tensorboard_log_directory: ./results/

Adding the random_seed, constrain_similarities, and tensorboard_log_directory makes no difference.

4

Hello guys! I’m facing the same problem here. I tested it in Rasa versions: 3.1.0 and 2.8.1 and the TEDPolicy results are always the same, any solution?

rasa_error
rasa_error495×547 20.6 KB

My config.yml

recipe: default.v1

language: pt

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
    "case_sensitive": False
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
  - name: FallbackClassifier
    threshold: 0.6
    ambiguity_threshold: 0.1

policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100

  - name: RulePolicy
    core_fallback_threshold: 0.4
    core_fallback_action_name: "utter_nao_entendi"
    enable_fallback_prediction: True
1 Like
5

Hello all! I have the exact same issue with Rasa 3.0. AugmentedMemoizationPolicy works fine, but the TedPolicy never extract any entities. All the scores reported in TedPolicy_report.json are zero (no matter what changes I make). Any resolution on this? (Please see my config.yml below):

language: en

pipeline:
  - name: WhitespaceTokenizer
    intent_split_symbol: '+'
  - name: RegexEntityExtractor 
    case_sensitive: false 
    use_lookup_tables: true 
    use_regexes: true
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: LanguageModelFeaturizer
    model: "bert"
    model_weights: 'rasa/LaBSE'
  - name: DIETClassifier
    use_masked_language_model: false
    epochs: 200 #260 #450
    #similarity_type: "cosine"
    #maximum_positive_similarity: 0.9
    #maximum_negative_similarity: -0.1
    embedding_dimension: 28
    #number_of_transformer_layers: 2 #2
    #number_of_attention_heads: 5
    #transformer_size: 180
    #hidden_layer_sizes: 
    #text: [256, 128]
    #connection_density: 0.2
    constrain_similarities: true
    #model_confidence: linear_norm
    #loss_type: cross_entropy
    batch_size: [64, 128]
    learning_rate: 4e-4 #0.001 #8e-4 #4e-4
    #evaluate_on_number_of_examples: 20
    #evaluate_every_number_of_epochs: 10
    #regularization_constant: 0.03
    #drop_rate_attention: 0.1
    random_seed: 2
    tensorboard_log_directory: ./.tensorboard/DIET
    tensorboard_log_level: epoch
    checkpoint_model: true
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 150
     
     
policies:
# # No configuration for policies was provided. The following default policies were used to train your model.
# # If you'd like to customize them, uncomment and adjust the policies.
# # See https://rasa.com/docs/rasa/policies for more information.
  - name: AugmentedMemoizationPolicy
    max_history: 5
  - name: TEDPolicy
    max_history: 5
    epochs: 400
    constrain_similarities: true
    #model_confidence: linear_norm
    #encoding_dimension: 25
    batch_size: [32, 64]
    random_seed: 1
    learning_rate: 1e-3
  - name: UnexpecTEDIntentPolicy
    max_history: 5
    epochs: 200
6

I am experiencing the same thing! Using rasa 3.2.6.