DIETClassifier not working properly for french bot

Rasa Open Source
1

Hi all,

I’am using Rasa 2.8.x and i would like to create a french model using this config

language: fr pipeline:

  • name: WhitespaceTokenizer
  • name: RegexFeaturizer
  • name: LexicalSyntacticFeaturizer
  • name: CountVectorsFeaturizer
  • name: CountVectorsFeaturizer analyzer: “char_wb” min_ngram: 1 max_ngram: 4
  • name: DIETClassifier epochs: 500
  • name: EntitySynonymMapper
  • name: ResponseSelector epochs: 100
  • name: FallbackClassifier threshold: 0.7 policies:
  • name: RulePolicy core_fallback_threshold: 0.7 core_fallback_action_name: “action_default_fallback” enable_fallback_prediction: True

The problem is that: when i ask a question that does not exist in my database, this question will be predicted by an existing intent with high confidence (0.9-0.8). This make the bot modele unusable. Can some one help me please.

Thanks for help

2

@Asmacats Hi, Asma! can you share some example ( rasa shell --debug ) and why you are using epochs to 500?, I hope you provided the good sets of training examples?

3

I have put epochs equals 500 because DIETClassifier does not performe well for me with 100 and 300.

Examples

{ “text”: “je veux acheter une table”, “intent”: { “id”: -927290318229052698, “name”: “Homeworking_expenses”, “confidence”: 0.9878547787666321 }, “entities”: [], “intent_ranking”: [ { “id”: -927290318229052698, “name”: “Homeworking_expenses”, “confidence”: 0.9878547787666321 }, { “id”: 76971132114904350, “name”: “HR_ref_info”, “confidence”: 0.004166799131780863 }, { “id”: -1234260648247950797, “name”: “Bank_IdentityModification”, “confidence”: 0.0031558743212372065 }, { “id”: -1145694641947141512, “name”: “Income_tax”, “confidence”: 0.0015534815611317754 }, { “id”: -1215719506263518657, “name”: “Disability”, “confidence”: 0.0007306730840355158 }, { “id”: 7803130917047708293, “name”: “HomeWorking_Modification”, “confidence”: 0.0006981872720643878 }, { “id”: -5802361719868036702, “name”: “RPS”, “confidence”: 0.0006207775440998375 }, { “id”: 9040549092417786293, “name”: “VIE_need”, “confidence”: 0.0004581536340992898 }, { “id”: -3085731226668345978, “name”: “HappyNewYear”, “confidence”: 0.00041413516737520695 }, { “id”: -7674930881790180033, “name”: “Termination_documents”, “confidence”: 0.00034716585651040077 } ] }

{ “text”: “bagra”, “intent”: { “id”: -7436027423513636610, “name”: “RealEstate_topics”, “confidence”: 0.8366689085960388 }, “entities”: [], “intent_ranking”: [ { “id”: -7436027423513636610, “name”: “RealEstate_topics”, “confidence”: 0.8366689085960388 }, { “id”: -3089545106919507808, “name”: “Bye”, “confidence”: 0.09386001527309418 }, { “id”: -4866337184457216915, “name”: “Acompte_request”, “confidence”: 0.022813159972429276 }, { “id”: 8577390014233000451, “name”: “Out_Of_Scope”, “confidence”: 0.013151166960597038 }, { “id”: -8690539986265921609, “name”: “Brad_Notice”, “confidence”: 0.008503349497914314 }, { “id”: 9040549092417786293, “name”: “VIE_need”, “confidence”: 0.007288624998182058 }, { “id”: -4821310619412684425, “name”: “test”, “confidence”: 0.006394628901034594 }, { “id”: -1664471251747425475, “name”: “Apprentice_need”, “confidence”: 0.005033596884459257 }, { “id”: -6198125073179470449, “name”: “Pay_Information”, “confidence”: 0.0033796532079577446 }, { “id”: 9078668446656761219, “name”: “Insulte”, “confidence”: 0.0029068393632769585 } ] }

{ “text”: “je vais cuisiner”, “intent”: { “id”: 8858069660738245532, “name”: “Parttime_get”, “confidence”: 0.7362473011016846 }, “entities”: [], “intent_ranking”: [ { “id”: 8858069660738245532, “name”: “Parttime_get”, “confidence”: 0.7362473011016846 }, { “id”: -8726565943689357929, “name”: “Business_trip_abroad”, “confidence”: 0.0803333967924118 }, { “id”: -1226918115395225022, “name”: “Not_HR_related”, “confidence”: 0.06672504544258118 }, { “id”: 2926905526687052247, “name”: “Revision_days_alternant”, “confidence”: 0.03680917248129845 }, { “id”: 4863940678474918023, “name”: “Vacation_HowToTake”, “confidence”: 0.02194966934621334 }, { “id”: 7801167326249256644, “name”: “HomeWorking_WishToBecome”, “confidence”: 0.018881049007177353 }, { “id”: -1999849757420176469, “name”: “Covid19_homeworking”, “confidence”: 0.016685040667653084 }, { “id”: -8409544892037923603, “name”: “Digiposte_NoPay”, “confidence”: 0.008241781033575535 }, { “id”: 690491568216209152, “name”: “HRTopics”, “confidence”: 0.0074796308763325214 }, { “id”: -309450883116865338, “name”: “Vacation_Report”, “confidence”: 0.006647932808846235 } ] }

4

@Asmacats

policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: RulePolicy
    nlu_threshold: 0.4
    core_threshold: 0.4
    fallback_action_name: "action_default_fallback"

There is no MemorisationPolicy and TEDPolicy can you update and try delete older model and train again.

5

@nik202 My problem concerns the NLU part. Th DIETClassifier does not perform well when predict the intent. I think adding policies will not impact the NLU part.

6

@Asmacats can you share some files related to your use case? How many training examples you had provided for the intents?

8

I have about 117 intentsconfig.yml (564 Bytes) rules.yml (12.2 KB)

9

@Asmacats can you even share the intent file i.e nlu.yml ?

10

@Asmacats Try update your config

language: fr
pipeline:
- name: SpacyNLP
  model: fr_core_news_lg
  case_sensitive: false
- name: SpacyTokenizer
- name: SpacyFeaturizer
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
- name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 4
- name: DIETClassifier
  epochs: 100
- name: CRFEntityExtractor
- name: EntitySynonymMapper
- name: ResponseSelector
  epochs: 100
policies:
- name: MemoizationPolicy
  max_history: 3
- name: TEDPolicy
  max_history: 5
  epochs: 100
- name: RulePolicy
  core_fallback_threshold: 0.7
  core_fallback_action_name: "action_default_fallback"
  enable_fallback_prediction: True

I guess this is much now the case of your pipeline.

11

@nik202 there is no needto MemozationPolicy and TED as i have no story. I am trying to construct FAQ chatbot. Unfortunately, using ReponseSelection, there is no way to set a fallback policy if the response selector has a lower confidence level. For that Reason i would like to use only RulePolicy.

I’am so surprised because after first fallback i have obtained this warning: “Circuit breaker tripped. Stopped predicting more actions for sender” and all the next predicted intent will be associated to the action fallback.

[state 1] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_listen [state 2] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: utter_Bank_IdentityModification [state 3] user intent: Attestation_Get | user entities: (‘Attestation’,) | previous action name: action_listen [state 4] user intent: Attestation_Get | user entities: (‘Attestation’,) | previous action name: utter_Attestation_Get [state 5] user intent: nlu_fallback | previous action name: action_listen [state 6] user intent: nlu_fallback | previous action name: action_default_fallback [state 7] user intent: nlu_fallback | previous action name: action_default_fallback [state 8] user intent: nlu_fallback | previous action name: action_default_fallback [state 9] user intent: nlu_fallback | previous action name: action_default_fallback [state 10] user intent: nlu_fallback | previous action name: action_default_fallback [state 11] user intent: nlu_fallback | previous action name: action_default_fallback [state 12] user intent: nlu_fallback | previous action name: action_default_fallback [state 13] user intent: nlu_fallback | previous action name: action_default_fallback [state 14] user intent: nlu_fallback | previous action name: action_default_fallback [state 15] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 16] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 17] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 18] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 19] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 20] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 21] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 22] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 23] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback [state 24] user intent: Bank_IdentityModification | user entities: (‘RIB’,) | previous action name: action_default_fallback

Any idea @nik202 please ?

12

@Asmacats can you share the above output in screenshot, its very hard to read.

@Asmacats is your DIET Classifier is working now?

13

@nik202

image
image1206×453 11.7 KB

14

@Asmacats can you run the above code in --debug so that I can see more detail for the same.

15

@nik202 debug.txt (39.8 KB)

16

@Asmacats I am trying to understand your issue, but now me confused.

  1. DIET Classifier is working for your use case yes or no?
  2. You want to implement action_fallback policy?
  3. You are using Rules and not using stories for your use case?
  4. Why you using Rules, as you can use stories any specific reason?

Any other issue?

17
  1. DIET Classifier is working for your use case yes or no? → DIET CLASSIFIER Not perform well for my case. I think results with 500 epochs is better than those with 100 and 300 epochs.
  2. You want to implement action_fallback policy? → No
  3. You are using Rules and not using stories for your use case? → my initial idea is to use FAQ. i have faced the same issue as How can set a fallback policy if the response selector has a lower confidence level? - #4 by DanielOlarte → For example if the classifier predict intent faq with confidence 1 but the right answer is predicted with confidence 0.5, while the fallback threshold is 0.7 → In such a case, the classifier will not predict fallaback.
  4. Why you using Rules, as you can use stories any specific reason? As i treat FAQ, i have prefered to use Rules
18

In this case the fallback should be trigger as that is not the right response for the user intent if the right prediction is 1.0 then it should return the right response else fallback message should be displayed.

You’re the second case if the right answer is predicted with a confidence of 0.5 which is less then fallback set threshold 0.7, it by default reply the set fallback message.