# Regex entity extractor generated a incomplete report

**URL:** <https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529>\
**Category:** Rasa Open Source\
**Created:** [November 25, 2021, 2:57pm UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529 "2021-11-25T14:57:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dsmendes](https://avatars.discourse-cdn.com/v4/letter/d/7cd45c/32.png) [@dsmendes](https://forum.rasa.com/u/dsmendes)\
**Post date:** [November 25, 2021, 2:57pm UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529/1 "2021-11-25T14:57:15Z")

</div>

Hi, I think that I have a problem when generating RegexEntityExtractor\_errors.json. The report is generated by running `rasa test nlu`. Analysing this report I notice that the entities are only extracted by CRFEntityExtractor. Bellow I pasted an example from the file generated. However, when I used `rasa shell nlu` the two extractors can extract correctly the entity.

RegexEntityExtractor\_errors.json

```auto
  {
  "text": "Compare my tariff with tariff_type_2",
    "entities": [
      {
        "start": 23,
        "end": 36,
        "value": "tariff_type_2",
        "entity": "tariff_type"
      }
    ],
    "predicted_entities": [
      {
        "entity": "tariff_type",
        "start": 23,
        "end": 36,
        "confidence_entity": 0.9832901695523858,
        "value": "tariff_type_2",
        "extractor": "CRFEntityExtractor"
      }
    ]
  },

```

NLU:

```auto
{
  "text": "Compare my tariff with tariff_type_2",
  "intent": {
    "name": "tariff_comparison",
    "confidence": 0.984128006208084
  },
  "entities": [
    {
      "entity": "tariff_type",
      "start": 23,
      "end": 36,
      "value": "tariff_type_2",
      "extractor": "RegexEntityExtractor"
    },
    {
      "entity": "tariff_type",
      "start": 23,
      "end": 36,
      "confidence_entity": 0.9947785145759475,
      "value": "tariff_type_2",
      "extractor": "CRFEntityExtractor"
    }
  ],

```

Can anyone explain to me why is it happening?

Another thing is. During the cross validation (5 folds) I found the following warning:

```auto
UserWarning: No lookup tables or regexes defined in the training data that have a name equal to any entity in the training data. In order for this component to work you need to define valid lookup tables or regexes in the training data.

```

I have 50 examples where 19 don’t have any entity of lookup table Thank you.

---

<div class="post-metadata">

**Author:** ![MatthiasLeimeister](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/matthiasleimeister/32/18158_2.png) [@MatthiasLeimeister](https://forum.rasa.com/u/MatthiasLeimeister)\
**Post date:** [December 1, 2021, 2:21pm UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529/2 "2021-12-01T14:21:04Z")

</div>

Hi @dsmendes, welcome to the community forum! 🙂 Could you please provide more information on your setup? What Rasa version are you using (`rasa --version`)? Could you also please post all associated YAML files (config.yml, domain.yml, nlu.yml, etc.) that lead to the observed issue? Please enclose the contents in code blocks (using `````). Thanks!

---

<div class="post-metadata">

**Author:** ![dsmendes](https://avatars.discourse-cdn.com/v4/letter/d/7cd45c/32.png) [@dsmendes](https://forum.rasa.com/u/dsmendes)\
**Post date:** [December 2, 2021, 11:05am UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529/3 "2021-12-02T11:05:55Z")

</div>

Hi,

my rasa version:

```auto
└─ $ rasa --version
Rasa Version : 2.8.2
Minimum Compatible Version: 2.8.0
Rasa SDK Version : 2.8.2
Rasa X Version : None
Python Version : 3.8.0
Operating System : Linux-5.4.0-90-generic-x86_64-with-glibc2.27
Python Path : xxxxxxxxx

```

I compress the rasa files here [data.zip](https://forum.rasa.com/uploads/short-url/zRyJZiaQwcsDkwioZy26lZEvtbz.zip) (8.3 KB)

Thanks

---

<div class="post-metadata">

**Author:** ![MatthiasLeimeister](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/matthiasleimeister/32/18158_2.png) [@MatthiasLeimeister](https://forum.rasa.com/u/MatthiasLeimeister)\
**Post date:** [December 2, 2021, 9:59pm UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529/4 "2021-12-02T21:59:00Z")

</div>

Hi @dsmendes, thanks for sending the config and data files. I was able to reproduce the issue and this looks like a bug in the creation of the cross-validation folds, where the lookup tables are not kept when the train and test data objects are created. I filed a bug report here:

> <https://github.com/RasaHQ/rasa/issues/10448>
>
> \### Rasa Open Source version
> 
> 2.8.15
> 
> \### Rasa SDK version
> 
> \_No response\_
> 
> \### R…asa X version
> 
> \_No response\_
> 
> \### Python version
> 
> 3.8
> 
> \### What operating system are you using?
> 
> OSX
> 
> \### What happened?
> 
> Based on this \[forum report\](http://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529), which I was able to reproduce, it looks like using \`rasa test nlu\` with cross validation does not properly work for \`RegexEntityExtractor\` with lookup tables.
> 
> When running cross validation with the provided NLU data, the following warning shows up:
> 
> \`\`\`
> UserWarning: No lookup tables or regexes defined in the training data that have a name equal to any entity in the training data. In order for this component to work you need to define valid lookup tables or regexes in the training data.
> \`\`\`
> 
> Afterwards, the report shows no entities extracted by \`RegexEntityExtractor\` with confusion matrix
> 
> \<img src="https://user-images.githubusercontent.com/10855680/144508400-f037d3cc-8e71-43e5-a508-c5aa0eecc73b.png" width=300 height=300\>
> 
> \### Source of the problem
> 
> Stepping through the code showed that when folds are generated from the training data in \[generate\_folds\](https://github.com/RasaHQ/rasa/blob/2.8.x/rasa/nlu/test.py#L1506), the \`TrainingData\` objects created \[here\](https://github.com/RasaHQ/rasa/blob/2.8.x/rasa/nlu/test.py#L1524) for the folds don't have the \`lookup\_tables\` parameter set, resulting in empty lookup tables for both train and test data.
> 
> \### Proposed solution
> 
> Add the \`lookup\_tables\` parameter to take over the lookup tables from the original training data object.
> 
> 
> \### Command / Request
> 
> \`\`\`shell
> rasa test nlu --nlu data/nlu.yml --cross-validation --runs 1 --folds 2
> \`\`\`
> 
> 
> \### Relevant log output
> 
> \`\`\`shell
> (rasa) matthias@Matthiass-MBP forum-49529 % rasa test nlu --nlu data/nlu.yml --cross-validation --runs 1 --folds 2
> 2021-12-02 22:54:43 INFO rasa.cli.test - Test model using cross validation.
> 2021-12-02 22:54:46 INFO rasa.nlu.utils.spacy\_utils - Trying to load spacy model with name 'en\_core\_web\_md'
> 2021-12-02 22:54:47 INFO rasa.nlu.components - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en\_core\_web\_md'.
> 2021-12-02 22:54:47 INFO rasa.nlu.model - Starting to train component SpacyNLP
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Starting to train component SpacyTokenizer
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Starting to train component RegexFeaturizer
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Starting to train component SpacyFeaturizer
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Starting to train component RegexEntityExtractor
> /Users/matthias/Workspace/rasa/rasa/shared/utils/io.py:97: UserWarning: No lookup tables or regexes defined in the training data that have a name equal to any entity in the training data. In order for this component to work you need to define valid lookup tables or regexes in the training data.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Starting to train component CRFEntityExtractor
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Starting to train component EntitySynonymMapper
> 2021-12-02 22:54:48 INFO rasa.nlu.model - Finished training component.
> \`\`\`

---

<div class="post-metadata">

**Author:** ![MatthiasLeimeister](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/matthiasleimeister/32/18158_2.png) [@MatthiasLeimeister](https://forum.rasa.com/u/MatthiasLeimeister)\
**Post date:** [December 16, 2021, 5:02pm UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529/5 "2021-12-16T17:02:37Z")

</div>

Hi @dsmendes, the issue got fixed in the latest release [3.0.3](https://github.com/RasaHQ/rasa/releases/tag/3.0.3). If you update to that the CV report should be corrected.

---

<div class="post-metadata">

**Author:** ![dsmendes](https://avatars.discourse-cdn.com/v4/letter/d/7cd45c/32.png) [@dsmendes](https://forum.rasa.com/u/dsmendes)\
**Post date:** [December 16, 2021, 5:14pm UTC](https://forum.rasa.com/t/regex-entity-extractor-generated-a-incomplete-report/49529/6 "2021-12-16T17:14:07Z")

</div>

Thank you for your help 🙂