# NLU only detecting entities explicitly present in training data

**URL:** <https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506>\
**Category:** Rasa Open Source\
**Created:** [January 21, 2019, 6:07am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506 "2019-01-21T06:07:11Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![shubham21197](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@shubham21197](https://forum.rasa.com/u/shubham21197)\
**Post date:** [January 21, 2019, 6:07am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/1 "2019-01-21T06:07:12Z")

</div>

The only entities showing up in the interpreter are the ones given in training data. For the given dataset:

```
  {
    "text": "My name is alice",
    "intent": "name",
    "entities": [
      {
        "start": 11,
        "end": 16,
        "value": "alice",
        "entity": "name1"
      }
    ]
  },
  {
    "text": "I am Josh",
    "intent": "name",
    "entities": [
      {
        "start": 5,
        "end": 9,
        "value": "Josh",
        "entity": "name1"
      }
    ]
  },

  {
    "text": "my name is Greg",
    "intent": "name",
    "entities": [
      {
        "end": 19,
        "start": 15,
        "value": "Greg",
        "entity": "name1"
      }
    ]
  },
  {
    "text": "I am david",
    "intent": "name",
    "entities": [
      {
        "end": 10,
        "start": 5,
        "value": "david",
        "entity": "name1"
      }
    ]
  },

  {
    "text": "john is my name",
    "intent": "name",
    "entities": [
      {
        "end": 15,
        "start": 11,
        "value": "john",
        "entity": "name1"
      }
    ]
  },

```

When a query like “My name is alice” (present in training data) comes, it is correctly classified as the entity ‘name1’. But something like “My name is peter” (not present in training data) doesn’t give any entities. What is going wrong here? P.S. I tried ner\_spacy but it only classifies names with first letter capital as entities.

---

<div class="post-metadata">

**Author:** ![flynnJ](https://avatars.discourse-cdn.com/v4/letter/f/b77776/32.png) [@flynnJ](https://forum.rasa.com/u/flynnJ)\
**Post date:** [January 21, 2019, 6:29am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/2 "2019-01-21T06:29:23Z")

</div>

Not an expert but as far as I can say, with the ner\_crf extractor, you need to train it with “My name is peter” too

---

<div class="post-metadata">

**Author:** ![rakesh](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@rakesh](https://forum.rasa.com/u/rakesh)\
**Post date:** [January 21, 2019, 6:41am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/3 "2019-01-21T06:41:33Z")

</div>

@shubham21197. I am also currently searching for an answer on this problem. Please let me know if you get any answer Thank you

---

<div class="post-metadata">

**Author:** ![shubham21197](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@shubham21197](https://forum.rasa.com/u/shubham21197)\
**Post date:** [January 21, 2019, 6:47am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/4 "2019-01-21T06:47:01Z")

</div>

But we can’t train on all the names out there. What can we do in a practical application where it should automatically extracts users name?

---

<div class="post-metadata">

**Author:** ![rakesh](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@rakesh](https://forum.rasa.com/u/rakesh)\
**Post date:** [January 21, 2019, 7:50am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/5 "2019-01-21T07:50:21Z")

</div>

@shubham21197I only came across these way seems good but need a data preparation.

> **[NLU Training Data](https://rasa.com/docs/rasa/nlu-training-data/)**
>
> Read more about how to format training data with Rasa NLU for open source natural language processing.

have a look at it , we need to provide some .txt file in the addotional lookups

---

<div class="post-metadata">

**Author:** ![flynnJ](https://avatars.discourse-cdn.com/v4/letter/f/b77776/32.png) [@flynnJ](https://forum.rasa.com/u/flynnJ)\
**Post date:** [January 22, 2019, 4:13pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/6 "2019-01-22T16:13:46Z")

</div>

@shubham21197Have you tried this with [https://github.com/RasaHQ/starter-pack-rasa-stack](https://github.com/RasaHQ/starter-pack-rasa-stack)

It already has a bunch of name entity extraction and intents, might work for you

---

<div class="post-metadata">

**Author:** ![flynnJ](https://avatars.discourse-cdn.com/v4/letter/f/b77776/32.png) [@flynnJ](https://forum.rasa.com/u/flynnJ)\
**Post date:** [January 22, 2019, 4:36pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/7 "2019-01-22T16:36:46Z")

</div>

@shubham21197 Try out the rasa starter pack with both Tensor Flow pipeline and the Sklearn pipeline, you’d see a few differences especially for your query “My name is Peter”

---

<div class="post-metadata">

**Author:** ![khaerulumam42](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/khaerulumam42/32/952_2.png) [@khaerulumam42](https://forum.rasa.com/u/khaerulumam42)\
**Post date:** [May 6, 2019, 2:36am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/8 "2019-05-06T02:36:26Z")

</div>

ner\_crf working with calculate weight of token based on prefix, suffix, word before and word after, you must variate training data to make it more generalize enough. Maybe you can try with lowering name, variate phrase, longer phrase.

---

<div class="post-metadata">

**Author:** ![jainsaurav22](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jainsaurav22/32/9217_2.png) [@jainsaurav22](https://forum.rasa.com/u/jainsaurav22)\
**Post date:** [May 29, 2020, 5:51am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/9 "2020-05-29T05:51:43Z")

</div>

The same problem is what i am facing. If you are able to solve yours, kindly help me out on this too.

---

<div class="post-metadata">

**Author:** ![vaidehi](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vaidehi](https://forum.rasa.com/u/vaidehi)\
**Post date:** [June 4, 2020, 7:41am UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/10 "2020-06-04T07:41:40Z")

</div>

This can happen if you have less training data in nlu for that statement.

I have a few options for experimenting

1. Increase nlu data

2. Also, DIET classifier in the pipeline can better extract entities do try out with this. If this also doesn’t work if entities are limited then

3. lookup tables in rasa can be of some use.@Juste @maddymantha may be you can give more clarity

---

<div class="post-metadata">

**Author:** ![jerry](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jerry/32/16367_2.png) [@jerry](https://forum.rasa.com/u/jerry)\
**Post date:** [August 7, 2021, 7:51pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/11 "2021-08-07T19:51:40Z")

</div>

Hi there , I am facing a similar issue, i have a entity named **location** and an intent named **inform\_state\_district** , i have a look up table for location , still most of the times abt 80% it fails to detect the entity in the intent, i will attach the screenshots of all below

 ![entoty_prob](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/3X/2/2/223fe5230e7e1aa32fc21d4db293109d5b677153.png) ![intent_nlu_yml](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/3X/9/d/9d34d1a53a68fe8c16d976e5abd06a8bf83ba2ed.png) ![lookup table](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/3X/f/0/f0f58d0bef50bec3dbcae3b50022a9a74ac99523.png) , the first white pic shows places of where it fails and where it doesnt, it would be great if someone can shed some light into why this behaviour occurs, (the slot district is mapped to location , if anyone is wondering y ) @nik202 any idea 😅😅

---

<div class="post-metadata">

**Author:** ![ChrisRahme](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/chrisrahme/32/15572_2.png) [@ChrisRahme](https://forum.rasa.com/u/ChrisRahme)\
**Post date:** [August 7, 2021, 10:11pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/12 "2021-08-07T22:11:42Z")

</div>

Are the places where it fails (like Durg and Ernakulam) in the lookup?

---

<div class="post-metadata">

**Author:** ![jerry](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jerry/32/16367_2.png) [@jerry](https://forum.rasa.com/u/jerry)\
**Post date:** [August 8, 2021, 1:15pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/13 "2021-08-08T13:15:23Z")

</div>

Yup all those are there, just attaching lookup for reference.[district\_data.yml](https://forum.rasa.com/uploads/short-url/r2SPJB0uL0ziEjkuAKLhPLNzwfS.yml) (6.9 KB) [state\_data.yml](https://forum.rasa.com/uploads/short-url/ixq49MtsIpeMfsUwkvEGtJF6hju.yml) (804 Bytes)

---

<div class="post-metadata">

**Author:** ![ChrisRahme](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/chrisrahme/32/15572_2.png) [@ChrisRahme](https://forum.rasa.com/u/ChrisRahme)\
**Post date:** [August 8, 2021, 2:27pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/14 "2021-08-08T14:27:05Z")

</div>

Both files are lookups for the `location` entity. You should merge them and try again.

---

<div class="post-metadata">

**Author:** ![nik202](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nik202/32/16098_2.png) [@nik202](https://forum.rasa.com/u/nik202)\
**Post date:** [August 8, 2021, 4:44pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/15 "2021-08-08T16:44:27Z")

</div>

@jerry share config.yml

Hope you following this benchmark [NLU Training Data](https://rasa.com/docs/rasa/nlu-training-data/#lookup-tables)

---

<div class="post-metadata">

**Author:** ![jerry](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jerry/32/16367_2.png) [@jerry](https://forum.rasa.com/u/jerry)\
**Post date:** [August 8, 2021, 6:41pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/16 "2021-08-08T18:41:10Z")

</div>

Still No difference yet

---

<div class="post-metadata">

**Author:** ![jerry](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/jerry/32/16367_2.png) [@jerry](https://forum.rasa.com/u/jerry)\
**Post date:** [August 8, 2021, 6:48pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/17 "2021-08-08T18:48:19Z")

</div>

As for the training data I have used the entity and intent stuff like in the previous screen shot, like the one mentioned in the docs, for config.yml i havent changed anything except for the fact that i added rulepolicy for forms, here is the file[config.yml](https://forum.rasa.com/uploads/short-url/jkooBtI4905hX70OtscjJRGeOHi.yml) (1.3 KB)

---

<div class="post-metadata">

**Author:** ![nik202](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nik202/32/16098_2.png) [@nik202](https://forum.rasa.com/u/nik202)\
**Post date:** [August 8, 2021, 7:44pm UTC](https://forum.rasa.com/t/nlu-only-detecting-entities-explicitly-present-in-training-data/4506/18 "2021-08-08T19:44:55Z")

</div>

@jerry check this video : [https://youtu.be/gvyfQZMnHPY](https://youtu.be/gvyfQZMnHPY)