Case Sensitivity

Rasa Open Source
1

I’m testing my chatbot, I inserted several entity values which are all in uppercase. I would like my NLU model to ignore the case sensitivity but despite removing the t"itle" and “upper” components it does not identify them. Also in order to conduct any form of entity extraction my crf component needs a “upper” component due to them all being upper case values.

How do I remove case sensitivity from my CRF component?

1 Like
2

Did you manage to fix this? I have the same problem

1 Like
3

which pipeline do you use?

4

This is the pipeline I use

language: "es"

pipeline:
- name: "tokenizer_whitespace"
- name: "ner_crf"
  features: [
              ["low", "upper"],
              ["bias", "low", "prefix5", "prefix2", "suffix5", "suffix3",
               "suffix2", "upper", "digit", "pattern"],
              ["low", "upper"]
            ]
- name: "ner_synonyms"
- name: "intent_featurizer_count_vectors"
- name: "intent_classifier_tensorflow_embedding"
# nn architecture
  "num_hidden_layers_a": 2
  "hidden_layer_size_a": [256, 128]
  "num_hidden_layers_b": 0
  "hidden_layer_size_b": []
  "batch_size": 32
  "epochs": 50
  # embedding parameters
  "embed_dim": 20
  "mu_pos": 0.8  # should be 0.0 < ... < 1.0 for 'cosine'
  "mu_neg": -0.4  # should be -1.0 < ... < 1.0 for 'cosine'
  "similarity_type": "cosine"  # string 'cosine' or 'inner'
  "num_neg": 10
  "use_max_sim_neg": true  # flag which loss function to use
  # regularization
  "C2": 0.002
  "C_emb": 0.8
  "droprate": 0.2
  # flag if to tokenize intents
  "intent_tokenization_flag": false
  "intent_split_symbol": "_"
5

you would need to add an option there like case_sensitive and if it is False then use text.lower()

Do you mind creating a PR

1 Like
6

I am having a similar issue. My goal is for our bot to be case insensitive. I am using a very similar pipeline setup to @Matotias, and I also added case_sensitive: false to tokenizer_whitespace (now called WhitespaceTokenizer in newer versions of Rasa). Our model still seems to be case sensitive after these changes. An example is, I type the word “bar” and our entity extractor identifies it properly as a place_type. When I type “BAR”, the entity extractor does not recognize the word.

Any thoughts? Am I missing something?

7

there are various config options for crf: Components

8

Hi @Ghostvv. Thanks! I did some research on those and followed the advice from other posts about removing “title” like OP. This is what my pipeline looks like. Based on what I’ve read here, my model should be case insensitive, but it is not.

language: "en"

pipeline: #"supervised_embeddings"
  
  - name: "WhitespaceTokenizer"
    case_sensitive: false
  - name: "RegexFeaturizer"
  - name: "CRFEntityExtractor"
    features: [
              ["low", "upper"],
              ["bias", "low", "prefix5", "prefix2", "suffix5", "suffix3",
               "suffix2", "upper", "digit", "pattern"],
              ["low", "upper"]
            ]
  - name: "EntitySynonymMapper"
  - name: "CountVectorsFeaturizer"
    oov_token: oov
  - name: "EmbeddingIntentClassifier"
    epochs: 50
    intent_tokenization_flag: true
    intent_split_symbol: "+"
2 Likes
9

The problem is that WhitespaceTokenizer doesn’t have case_sensitive option. Do you mind creating a PR to fix that?

1 Like
10

raised the PR.

1 Like
11

thanks, reviewed

12

This is closed, PR merged.

1 Like
13

great work @sibbsnb

14

hello,

is it working for you ?