Improve NLU accuracy for FAQ (several responses)

Rasa Open Source
1

Hi everyone !

I’ve been using RASA for a project since several weeks / months and I still have problems with the accuracy of the NLU.

To be quick the project is to provide a Chatbot factory for specific entities which can use a spreadsheet and we convert this spreadsheet in Rasa files (domain.yml).

One entity is doing tests with us and we have a major problem. The NLU, despite several modification on the configuration file, recognize exact sentences with few accuracy (like 50%).

Do you have an idea how to improve this ? We only use RASA for intent recognition (no entities).

Config file:

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: fr
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: 'char_wb'
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    entity_recognition: false
    epochs: 20
  - name: ResponseSelector
    epochs: 100

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
    max_history: 1
  - name: FallbackPolicy
    nlu_threshold: 0.6
    core_threshold: 0.5
    fallback_action_name: 'utter_phrase_hors_sujet_0'

Exemple of the domain.yml file:

version: '2.0'

intents:
  - intent_id
  - other_intent_id
  - another_intent_id

responses:
  utter_intent_id_0:
    - text: 'First response'
  utter_intent_id_1:
    - text: 'Second response'
      buttons:
        - tilte: 'Title'
          payload: other_intent_id
  utter_other_intent_id_0:
    - text: 'First response'
  utter_other_intent_id_1:
    - text: 'Second response'
  utter_another_intent_id_0:
    - text: 'First response'
  utter_another_intent_id_1:
    - text: 'Second response'

nlu:
  - intent: intent_id
    examples: |
      - Example 1
      - Example 2
  - intent: other_intent_id
    examples: |
      - Other Example 1
      - Other Example 2
  - intent: another_intent_id
    examples: |
      - Another Example 1
      - Another Example 2

stories:
  - story: intent_id
    steps:
      - intent: intent_id
      - action: utter_intent_id_0
      - action: utter_intent_id_1
  - story: other_intent_id
    steps:
      - intent: other_intent_id
      - action: utter_other_intent_id_0
      - action: utter_other_intent_id_1
  - story: another_intent_id
    steps:
      - intent: another_intent_id
      - action: utter_another_intent_id_0
      - action: utter_another_intent_id_1

session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true

Some data usefull:

We are currently training the model with more than 800 intents and more than 3000 examples (this number is growing). Even with 50 or 100 epochs on DIETClassifier the result is disappointing.

Rasa Version : 2.0

Thanks.

2

Hi @vincentlaine. Would you be willing to share your training data? It’s quite difficult to give recommendations on the config alone