Lookup table is not working

Rasa Open Source
1

Example :slight_smile: Failed cases :

     1- it is a red
     2- chf
     3- rub
     4- aziz
     5- yellow

intent:check_balance

  • what is my balance
  • Could I pay in yen?
  • USD

intent:Greet

intent:ColorsIntent

##intent:Bored

  • I am bored
  • it is boring
  • it is bored

lookup:currency

  • Yen
  • USD
  • Euro
  • PKR
  • jpy
  • chf
  • cad
  • mxn
  • rub

lookup:name

  • aziz
  • ron
  • rehan
  • sahal

lookup:color

  • red
  • yellow
  • orange

language: “en”

pipeline:

  • name: “nlp_spacy” model: “en_core_web_md” case_sensitive: false
  • name: “tokenizer_spacy”
  • name: “intent_entity_featurizer_regex”
  • name: “ner_duckling” dimensions: [“time”,“number”,“phone-number”,“email”]
  • name: “ner_crf” features:

    features for word before token

    • [“low”, “title”, “upper”]

    features of token itself

    • [“bias”, “low”, “prefix5”, “prefix2”,“suffix5”, “suffix3”, “suffix2”,“digit”,“upper”, “title”, “pattern”]

    features for word after the token we want to tag

    • [“low”, “title”, “upper”] max_iterations: 50 L1_c: 0.1 L2_c: 0.1
  • name: “ner_synonyms”
  • name: “intent_featurizer_count_vectors”
  • name: “intent_classifier_tensorflow_embedding”
2

You need to include few examples with the content from your lookup table for it to work.

For example, as you have red, yellow, orange in your lookup, add these

##intent:ColorsIntent
- i like red(color)
- I am interested in yellow(color)
- it is a black color(color)
1 Like
3

How much for how many data ??

4

There is no specific number. 3~4 examples from my lookup table works for me.

5

what if have different values of hundreds ? There is alot of variations ? There are not same length ? Somethinglike these ?? colors

c3 ii
c4
tour
travato
vectra
via
view and view
view
vista
rialta
adventure
adventure cb500x
cruiser
cruiser 1300 custom line
cruiser ctx700n
cruiser fury
cruiser goldwing f6b
cruiser rebel
cruiser shadow line
dual sport
dual sport crf250l
super
super cb1000r
t@b
trailblazer (lite) by komfort
a.c.e.

@akelad what do i have to do ? if have hurdreds of data kike this data ? bcz loopup data is not working, do I have to train on each of these ??

6

I’m not sure if it’ll work for such large variation, never tried it out. However, I guess your training data must include all such variations for it to work.

7

you have to include some examples of these entities in your training data

8

@akelad @Juste @juste_petr can you please look into this issue Error while trying to create a look up table

9

Hi all, do I have to give examples for all the existing entities in my lookup table? I am failing to understand, if i have to give all the entities in the training sets how is lookup table helpful?

3 Likes
10

Hi, I have very domain specific entities that are not well recognized in my model. So, I am trying to improve the accuracy of entity recognition by adding lookup tables. But I don’t see any improvement. Currently, I’m using DIET classifier and based on this post “Entity lookup in DIET Classifier” we can use lookup tables with DIET model as long as “RegexFeaturizer” is provided in the pipeline. I also added more training data with the same entity in my nlu data. But still I don’t see any changes. The other thing that I tried was adding “CRFEntityExtractor” which I’m not sure if it is a good idea or not, but still it didn’t help. Can you share your experience that what possibly could be the solution?

1 Like
11

Having similar problems here - did anyone get a response to this?

12

Just want to chime in that it’s Feb 2022 and this still does not seem to work. I have a lookup table that shares the same name as the entity I am extracting. The table contains 14 elements. My training data uses 4-5 of those elements in a good number of different sentence structures. When entering the same EXACT phrasing as in my training data, but only changing the entity that the lookup table should provide, it does not detect any entity at all.

Example: In my NLU: Tell me about the rockwell.

In my lookup table:

  • rockwell
  • destin

Train, start, ask: Tell me about the destin

I get no matches on the entity.

Its just flat out not working at all. And yes I have more training data than the 1 example and a larger lookup table than the one listed.

1 Like
13

use RegexEntityExtractor, it must resolve your problem.

    pipeline:
    - name: RegexEntityExtractor
      # text will be processed with case insensitive as default
      case_sensitive: False
      # use lookup tables to extract entities
      use_lookup_tables: True
      # use regexes to extract entities
      use_regexes: True
      # use match word boundaries for lookup table
      "use_word_boundaries": True
1 Like
14

I am also unable to understand how is lookup table helpful if I have to add every single element of lookup table into examples because I have a list of all countries and was building a tell time bot and in lookup table I have added around 150 country names now do I also have to add 150 examples mentioning all of the countries?

15

Thank you @azizullah2017 it worked

1 Like
16

Hi, it didn’t work for me I made a topic about it yesterday.