# Lookup table not working for entities with multiple words

**URL:** <https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305>\
**Category:** Rasa Open Source\
**Created:** [February 26, 2020, 11:04am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305 "2020-02-26T11:04:24Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![vishu1994](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vishu1994/32/4970_2.png) [@vishu1994](https://forum.rasa.com/u/vishu1994)\
**Post date:** [February 26, 2020, 11:04am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/1 "2020-02-26T11:04:25Z")

</div>

I am using lookup table for extracting an entity “country” which has some 196 values. Lookup table looks something like this.

```
Sierra Leone
Puerto Rico
Belgium
Palau
Belize
Indonesia
Brunei
Macao
Hong Kong
Nicaragua
South Africa
Montserrat
Syria
Australia
Jordan
Guinea
Libya
Paraguay
St. Lucia
Israel 
Nigeria
Barbados
Kazakstan
Aland Islands

```

**Results :**

```
get me downloads count in hong kong
    {
      "intent": {
        "name": "getAppDownloadsCount",
        "confidence": 0.9998836517333984
      },
      "entities": [
        {
          "start": 26,
          "end": 30,
          "value": "hong",
          "entity": "country",
          "confidence": 0.5102069739663814,
          "extractor": "CRFEntityExtractor"
        },
        {
          "start": 31,
          "end": 35,
          "value": "kong",
          "entity": "country",
          "confidence": 0.7573467555695853,
          "extractor": "CRFEntityExtractor"
        }
      ],
      "intent_ranking": [
        {
          "name": "getAppDownloadsCount",
          "confidence": 0.9998836517333984

```

Ideally i should get hong kong as one single value.Can somebody helps or atleast tell why it happens in case of lookup table, as crf works fine with multiple words entities present in training data and if these values are in lookup table we get such results.

---

<div class="post-metadata">

**Author:** ![Tanja](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tanja/32/4696_2.png) [@Tanja](https://forum.rasa.com/u/Tanja)\
**Post date:** [March 3, 2020, 10:13am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/2 "2020-03-03T10:13:30Z")

</div>

What version of Rasa are you using? And how does your config file look like?

---

<div class="post-metadata">

**Author:** ![vishu1994](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vishu1994/32/4970_2.png) [@vishu1994](https://forum.rasa.com/u/vishu1994)\
**Post date:** [March 4, 2020, 12:44pm UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/3 "2020-03-04T12:44:02Z")

</div>

rasa version : 1.7.0

```
# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline: pretrained_embeddings_spacy
  # Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: KerasPolicy
  - name: MappingPolicy
```

---

<div class="post-metadata">

**Author:** ![Tanja](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tanja/32/4696_2.png) [@Tanja](https://forum.rasa.com/u/Tanja)\
**Post date:** [March 5, 2020, 3:34pm UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/4 "2020-03-05T15:34:44Z")

</div>

I just tried to reproduce the issue. For me everything seems to be working. Just to clarify, if you add the lookup table to your NLU data, Rasa predicts

```auto
"entities": [
        {
          "start": 26,
          "end": 30,
          "value": "hong",
          "entity": "country",
          "confidence": 0.5102069739663814,
          "extractor": "CRFEntityExtractor"
        },
        {
          "start": 31,
          "end": 35,
          "value": "kong",
          "entity": "country",
          "confidence": 0.7573467555695853,
          "extractor": "CRFEntityExtractor"
        }

```

But if you train without the lookup table, Rasa will combine the two entities into one?

---

<div class="post-metadata">

**Author:** ![vishu1994](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vishu1994/32/4970_2.png) [@vishu1994](https://forum.rasa.com/u/vishu1994)\
**Post date:** [March 6, 2020, 6:18am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/5 "2020-03-06T06:18:45Z")

</div>

ya but “hong kong” are not two different entities, this should come as one single value.

I am assuming hong kong is not present in training data.Because if [hong kong] is there in training data it comes as single entity value, and entities which are there in lookup table having multiple words gives me this issue.

---

<div class="post-metadata">

**Author:** ![Tanja](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tanja/32/4696_2.png) [@Tanja](https://forum.rasa.com/u/Tanja)\
**Post date:** [March 6, 2020, 7:47am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/6 "2020-03-06T07:47:18Z")

</div>

I was able to reproduce the error and created an issue for it ([CRFEntityExtractor splits one entity into two · Issue #5377 · RasaHQ/rasa · GitHub](https://github.com/RasaHQ/rasa/issues/5377)).

It should not be related to the lookup tables. It seems to be related to the `BILOU_flag`. Can you try to train your bot with the following pipeline and check if it works? Thanks.

```auto
language: "en"

pipeline:
- name: "SpacyNLP"
- name: "SpacyTokenizer"
- name: "SpacyFeaturizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
  BILOU_flag: True
- name: "EntitySynonymMapper"
- name: "SklearnIntentClassifier"

```

---

<div class="post-metadata">

**Author:** ![vishu1994](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vishu1994/32/4970_2.png) [@vishu1994](https://forum.rasa.com/u/vishu1994)\
**Post date:** [March 6, 2020, 8:07am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/7 "2020-03-06T08:07:43Z")

</div>

Ya i trained the bot with the mentioned pipeline and i get same same results.

```
get me downloads count in hong kong
{
  "intent": {
    "name": "getAppDownloadsCount",
    "confidence": 0.9592514339348922
  },
  "entities": [
    {
      "start": 26,
      "end": 30,
      "value": "hong",
      "entity": "country",
      "confidence": 0.7003851556965186,
      "extractor": "CRFEntityExtractor"
    },
    {
      "start": 31,
      "end": 35,
      "value": "kong",
      "entity": "country",
      "confidence": 0.7234106247445744,
      "extractor": "CRFEntityExtractor"
    }
  ],
```

---

<div class="post-metadata">

**Author:** ![HuppeJ](https://avatars.discourse-cdn.com/v4/letter/h/3be4f8/32.png) [@HuppeJ](https://forum.rasa.com/u/HuppeJ)\
**Post date:** [March 31, 2020, 1:44am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/8 "2020-03-31T01:44:30Z")

</div>

Hello,

I had the same problem, but I solved it by adding nlu data with multiple words:

```auto
[…] [Costa Rica](country) […]
[…] [Dominican Republic](country) […]
[…] [Central African Republic](country) […]
[…] […] [Papua New Guinea](country) […]
[…] [Papua New Guinea](country) […]
...

```

---

<div class="post-metadata">

**Author:** ![vishu1994](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vishu1994/32/4970_2.png) [@vishu1994](https://forum.rasa.com/u/vishu1994)\
**Post date:** [March 31, 2020, 6:52am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/9 "2020-03-31T06:52:56Z")

</div>

Ya, i also did the same thing, basically the training data should cover few values of each type of variation present in the lookup table, that way it learns better @Tanja

---

<div class="post-metadata">

**Author:** ![vish0701](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vish0701/32/9441_2.png) [@vish0701](https://forum.rasa.com/u/vish0701)\
**Post date:** [June 21, 2020, 9:11am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/10 "2020-06-21T09:11:32Z")

</div>

Hmm. Similar problem here. I did add several examples but not quite there.

Here’s what I have in nlu.md (listing only the relevant examples)

```
- list all [experts](expert_name) in the country of [South Africa]{"entity":"country", "value":"South Africa"} 
- list all [experts](expert_name) in the country of [New Zealand]{"entity":"country", "value":"New Zealand"}
- [United Kingdom](country)
- [United States](country)

```

My config.yml is as follows

```
    language: en
    pipeline:
      - name: SpacyNLP
      - name: DeSymbolizer
      - name: SpacyTokenizer
      - name: SpacyFeaturizer
      - name: RegexFeaturizer
      - name: LexicalSyntacticFeaturizer
      - name: CountVectorsFeaturizer
      - name: CountVectorsFeaturizer
        analyzer: "char_wb"
        min_ngram: 1
        max_ngram: 4
      - name: DIETClassifier
        epochs: 100
      - name: EntitySynonymMapper
      - name: ResponseSelector
        epochs: 100

    # Configuration for Rasa Core.
    # https://rasa.com/docs/rasa/core/policies/
    policies:
      - name: MemoizationPolicy
      - name: TEDPolicy
        max_history: 5
        epochs: 100
      - name: MappingPolicy

```

It still didn’t catch “El Salvador”; ended up extracting “Salvador” for country name and dropped “El”. Or “Marshall Islands”. Picked up only Islands. Maybe it recognizes Marshall as a person’s name and dropped it?

Do I need to add more samples? And, I also have countries with “&”. Looks like I will need to a few samples of those as well for it to work?

---

<div class="post-metadata">

**Author:** ![vishu1994](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/vishu1994/32/4970_2.png) [@vishu1994](https://forum.rasa.com/u/vishu1994)\
**Post date:** [June 22, 2020, 5:58am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/11 "2020-06-22T05:58:57Z")

</div>

Yes you need to cover all entities in the training data.

---

<div class="post-metadata">

**Author:** ![iszainab](https://avatars.discourse-cdn.com/v4/letter/i/ad7895/32.png) [@iszainab](https://forum.rasa.com/u/iszainab)\
**Post date:** [September 14, 2020, 8:30am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/12 "2020-09-14T08:30:53Z")

</div>

Hi can I get some guidance please? I’m using the CRFEntityExtractor with BILOU\_flag set as True and my model seems to be splitting the entities. I was under the impression that if I needed my compound entities to be treated as a single entity, I need to use BILOU tagging which seemed to be working fine for me till now. I recently switched from SklearnIntentClassifier to DIETClassifier (configured for only intent classification).

I know one solution is to add training phrases and I’ve done that but since I can’t possibly add all entities in training, I need a better solution. A solution at the config level that is.

---

<div class="post-metadata">

**Author:** ![Tanja](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/tanja/32/4696_2.png) [@Tanja](https://forum.rasa.com/u/Tanja)\
**Post date:** [September 21, 2020, 9:21am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/13 "2020-09-21T09:21:06Z")

</div>

@iszainab Can you please share some more details? What Rasa version are you using? How does your pipeline look like? And can you please give an example of an entity that is split? Thanks.

---

<div class="post-metadata">

**Author:** ![cicciob95](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/cicciob95/32/11049_2.png) [@cicciob95](https://forum.rasa.com/u/cicciob95)\
**Post date:** [January 29, 2021, 11:17am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/14 "2021-01-29T11:17:45Z")

</div>

I have the same problem. The rasa version I am using is 1.10.9, while the NLU pipeline is:

- name: “SpacyNLP” “case\_sensitive”: False
- name: “SpacyTokenizer”
- name: “SpacyFeaturizer” “pooling”: “mean”
- name: “RegexFeaturizer” “case\_sensitive”: False
- name: “CRFEntityExtractor” “epochs”: 300
- name: “LexicalSyntacticFeaturizer”
- name: “CountVectorsFeaturizer”
- name: “CountVectorsFeaturizer” “analyzer”: “char\_wb” “min\_ngram”: 3 “max\_ngram”: 5
- name: “DIETClassifier” “entity\_recognition”: False “epochs”: 300
- name: “EntitySynonymMapper”

---

<div class="post-metadata">

**Author:** ![osterhult](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/osterhult/32/12771_2.png) [@osterhult](https://forum.rasa.com/u/osterhult)\
**Post date:** [January 29, 2021, 3:22pm UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/15 "2021-01-29T15:22:02Z")

</div>

Just a thought regarding your case. Can your issue have something todo that you have two different entity extractors in your pipeline? “CRFEntityExtrator” and “DIETClassifier”?

I had an issue with that myself some time ago. (I am a newbie with Rasa) 🙂

---

<div class="post-metadata">

**Author:** ![cicciob95](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/cicciob95/32/11049_2.png) [@cicciob95](https://forum.rasa.com/u/cicciob95)\
**Post date:** [January 29, 2021, 4:37pm UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/16 "2021-01-29T16:37:48Z")

</div>

I don’t know…but i don’t think this is the problem since i use the DIET only to classify the intent (“entity\_recognition” is set to False).

---

<div class="post-metadata">

**Author:** ![osterhult](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/osterhult/32/12771_2.png) [@osterhult](https://forum.rasa.com/u/osterhult)\
**Post date:** [January 29, 2021, 8:29pm UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/17 "2021-01-29T20:29:22Z")

</div>

Ok. I thought it was worth a shot at least. 🙂

---

<div class="post-metadata">

**Author:** ![cicciob95](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/cicciob95/32/11049_2.png) [@cicciob95](https://forum.rasa.com/u/cicciob95)\
**Post date:** [February 27, 2021, 12:05pm UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/18 "2021-02-27T12:05:17Z")

</div>

I currently have the same problem. Were you able to solve it somehow? Thanks

---

<div class="post-metadata">

**Author:** ![GeethPriyanka](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/geethpriyanka/32/13193_2.png) [@GeethPriyanka](https://forum.rasa.com/u/GeethPriyanka)\
**Post date:** [June 1, 2021, 11:15am UTC](https://forum.rasa.com/t/lookup-table-not-working-for-entities-with-multiple-words/25305/19 "2021-06-01T11:15:39Z")

</div>

Were you able to solve the problem. ? I’m also facing the same
