Mapping Policy ignores NLU threshold

Rasa Open Source
1

I have NLU Threshold configured with default 0.3 value. “greet” intent is mapped to “action_greet”. When I get incoming user intent classified as “greet” with 0.24 confidence, I would expect the classification to fail and use the fallback policy. Instead the mapping policy is invoked. So you get the following:

2019-05-21 11:30:25 DEBUG rasa_core.processor - Received user message ‘attending training tomorrow’ with intent ‘{‘name’: ‘greet’, ‘confidence’: 0.24704762503877342}’ and entities ‘[{‘start’: 19, ‘end’: 27, ‘text’: ‘tomorrow’, ‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘confidence’: 1.0, ‘additional_info’: {‘values’: [{‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘grain’: ‘day’, ‘type’: ‘value’}], ‘value’: ‘2019-05-22T00:00:00.000-07:00’, ‘grain’: ‘day’, ‘type’: ‘value’}, ‘entity’: ‘time’, ‘extractor’: ‘DucklingHTTPExtractor’}]’

2019-05-21 11:30:25 DEBUG rasa_core.policies.fallback - NLU confidence 0.24704762503877342 is lower than NLU threshold 0.3.

2019-05-21 11:30:25 DEBUG rasa_core.policies.memoization - There is a memorised next action ‘35’ 2019-05-21 11:30:25 DEBUG rasa_core.policies.form_policy - There is an active form ‘student_loan_form’ 2019-05-21 11:30:25 DEBUG rasa_core.policies.ensemble - Predicted next action using policy_4_MappingPolicy

Is there any way for the MappingPolicy to only work when the NLU confidence is greater than threshold?

2

Hm this is a good point, might require discussion but seems to me like a bug. Would you mind opening a github issue?

3

already opened here:

1 Like
4

Hi I’m having the same issue even though I comment out the Mapping Policy. The bot doesn’t consider NLU threshold. What could be the other reason/s?

Thanks

5

@serdar what’s your config?

6

Hello my config file is

- name: "customComponent.expand_contractions.expanding_contractions"
- name: WhitespaceTokenizer
- name: RegexFeaturizer
- name: CRFEntityExtractor
- name: EntitySynonymMapper
- epochs: 4
  max_sentence_length: 100
  name: chatbot.classifiers.text_cnn_intent_classifier.TextCnnIntentClassifier
  num_cnn_layers: 7
- name: "customComponent.confidence_level.confidence_levels"
policies:
- name: "KerasPolicy"
  epochs: 200
  batch_size: 36
  max_history: 5
  priority: 1
- name: MemoizationPolicy
  max_history: 8
  priority: 2
- name: FallbackPolicy
  nlu_threshold: 0.5
  core_threshold: 0.3
  fallback_action_name: action_default_fallback
  priority: 3
- name: FormPolicy
  priority: 4
- name: MappingPolicy
  priority: 5```
7

@serdar does the issue still persist if you get rid of the personalized priorities? these are what we fixed when we fixed that bug. also what version are you on?

8

Hi @erohmensing , I’m using Rasa 1.0.6 version. I figured out that Rasa does not consider NLU threshold level to evaluate or test the model, right? For example it doesn’t consider NLU threshold level to calculate F1 score, correct? I wanted to how the F1 score changes based on NLU threshold level I created a custom component. Thank you very much for response in advance

9

Are you talking about core or NLU evaluation?

10

Hi @erohmensing NLU evaluation

11

Right, so NLU evaluation does not take your dialogue policies into account, it is merely testing how well your NLU model performs. You could completely remove the dialogue part of your model and it would perform the same way.