# Rasa com

**URL:** <https://forum.rasa.com/t/rasa-com/27810>\
**Category:** Rasa Open Source\
**Created:** [April 21, 2020, 12:57pm UTC](https://forum.rasa.com/t/rasa-com/27810 "2020-04-21T12:57:21Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![keerti](https://avatars.discourse-cdn.com/v4/letter/k/db5fbb/32.png) [@keerti](https://forum.rasa.com/u/keerti)\
**Post date:** [April 21, 2020, 12:57pm UTC](https://forum.rasa.com/t/rasa-com/27810/1 "2020-04-21T12:57:21Z")

</div>

# I am trying a create for NLU. I want to preprocess the user input before we do anything in the NLU pipeline like- lemmatization. I took the code from rasa GitHub and modified it as per my requirement. I printed the input in this and I am able to see the input getting modified. But still, It is not adding in the input text for intent and entity recognization. Here is my code:-

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 6:08pm UTC](https://forum.rasa.com/t/rasa-com/27810/2 "2020-04-21T18:08:39Z")

</div>

Hi @keerti, welcome to the Rasa forum! It’s cool that you’re building a custom component. Are you adding your custom component in your pipeline? if so, what does it look like?

---

<div class="post-metadata">

**Author:** ![keerti](https://avatars.discourse-cdn.com/v4/letter/k/db5fbb/32.png) [@keerti](https://forum.rasa.com/u/keerti)\
**Post date:** [April 21, 2020, 6:16pm UTC](https://forum.rasa.com/t/rasa-com/27810/3 "2020-04-21T18:16:34Z")

</div>

Hi @mloubser, I have added the custom component in the pipeline this way:-

> ```
> pipeline:
> - name: "Lemmatizationtokenizer"
> - name: WhitespaceTokenizer
> - name: RegexFeaturizer
> - name: CRFEntityExtractor
> - name: EntitySynonymMapper
> - name: CountVectorsFeaturizer
> - name: EmbeddingIntentClassifier
> 
> ```

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 6:20pm UTC](https://forum.rasa.com/t/rasa-com/27810/4 "2020-04-21T18:20:18Z")

</div>

thanks. What version of Rasa are you on? looks like something pre-1.8 right?

---

<div class="post-metadata">

**Author:** ![keerti](https://avatars.discourse-cdn.com/v4/letter/k/db5fbb/32.png) [@keerti](https://forum.rasa.com/u/keerti)\
**Post date:** [April 21, 2020, 6:22pm UTC](https://forum.rasa.com/t/rasa-com/27810/5 "2020-04-21T18:22:35Z")

</div>

I am using rasa version:- 1.6.1

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 6:26pm UTC](https://forum.rasa.com/t/rasa-com/27810/6 "2020-04-21T18:26:03Z")

</div>

Ok thanks. It looks like you’re using two constant names that aren’t available - `TEXT_ATTRIBUTE` and `INTENT_ATTRIBUTE`. I think you want `INTENT` and `TEXT` respectively.

\*Edit: Sorry, this is true for higher versions of rasa, not the one you’re on.

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 6:28pm UTC](https://forum.rasa.com/t/rasa-com/27810/7 "2020-04-21T18:28:07Z")

</div>

I’m not sure what code you pulled to start this - could you link to the file on Github that you were looking at?

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 6:34pm UTC](https://forum.rasa.com/t/rasa-com/27810/8 "2020-04-21T18:34:43Z")

</div>

Another note - doesn’t look like you’re using the stop words you are loading anywhere?

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 6:47pm UTC](https://forum.rasa.com/t/rasa-com/27810/10 "2020-04-21T18:47:16Z")

</div>

Sorry, edited my comment on that - if you upgrade to a higher version of rasa, that will be true, in rasa1.6.1, you have the right names.

re. stop words - I see that you’re loading them, but where are they being used?

What happens when you run `rasa train nlu`? Does it train successfully?

If yes, what happens with `rasa shell nlu` and then entering something like `test message-things-to ? remove .`. ?

---

<div class="post-metadata">

**Author:** ![keerti](https://avatars.discourse-cdn.com/v4/letter/k/db5fbb/32.png) [@keerti](https://forum.rasa.com/u/keerti)\
**Post date:** [April 21, 2020, 7:06pm UTC](https://forum.rasa.com/t/rasa-com/27810/11 "2020-04-21T19:06:09Z")

</div>

sorry, my mistake. I removed the stop-words logic before submitting this question here. I trained the model the same you mentioned above:- “rasa train nlu” It trained successfully.

I run it using “rasa shell --debug”.

Suppose I am entering input like:- “flights timings to India?”. I am using print statement in the custom component. There I am getting “flight timing to india”.

But for intent and entity recognize there my log shows:- **"rasa.core.processor - Received user message ‘flight timings to india?’ with intent …"**

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 21, 2020, 7:39pm UTC](https://forum.rasa.com/t/rasa-com/27810/12 "2020-04-21T19:39:28Z")

</div>

Ok, I see what’s happening here. In these two functions in rasa.nlu.training\_data.message, you see how the `text` attribute is handled differently than the other attributes? so `message.data["text"]` will be what you’re expecting, but `message.text` will be left untouched.

```auto
    def set(self, prop, info, add_to_output=False) -> None:
        self.data[prop] = info
        if add_to_output:
            self.output_properties.add(prop)

    def get(self, prop, default=None) -> Any:
        if prop == TEXT_ATTRIBUTE:
            return self.text
        return self.data.get(prop, default)

```

You can get around this by doing:

```auto
    def process(self, message: Message, **kwargs: Any) -> None:
        message.text = self.tokenize(message.text)

```

However, since you’re creating a tokenizer, it would make sense for your class to return tokens, not text, which would be handled correctly. Using `WhitespaceTokenizer` over pre-tokenized text wouldn’t really make sense, in my opinion.

For example, I passed the sentence “Are you a bot?” to the model, and Lemmatizationtokenizer returns

```auto
be -PRON- a bot?

```

But after `WhitespaceTokenizer` is done with it, you get

```auto
be pron a bot?

```

---

<div class="post-metadata">

**Author:** ![keerti](https://avatars.discourse-cdn.com/v4/letter/k/db5fbb/32.png) [@keerti](https://forum.rasa.com/u/keerti)\
**Post date:** [April 21, 2020, 8:14pm UTC](https://forum.rasa.com/t/rasa-com/27810/13 "2020-04-21T20:14:52Z")

</div>

I changed the code to this :

But I am still facing the same problem. And I am handling the -PRON- using stopwords in my code.

I don’t want to create Tokenizer. I just want to take the input and do Lemmatization and forward that input to next nlu pipeline.

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 22, 2020, 1:29am UTC](https://forum.rasa.com/t/rasa-com/27810/14 "2020-04-22T01:29:44Z")

</div>

Ok, if you don’t want to tokenize then it makes sense to edit the text, I was going off of the class name.

Did you remove everything from your tokenize function? With the code below (basically the same as your original, except the train and process functions), I got the results that follow:

config.yml

```auto
language: en
pipeline: 
  - name: Lemmatizer.Lemmatizer
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: CRFEntityExtractor
  - name: EntitySynonymMapper
  - name: CountVectorsFeaturizer
  - name: EmbeddingIntentClassifier

```

Lemmatizer.py

```auto
import re
from typing import Any, Optional, Text, Dict, List, Type
from rasa.nlu.config import RasaNLUModelConfig
from rasa.nlu.tokenizers.tokenizer import Token, Tokenizer
from rasa.nlu.training_data import Message, TrainingData
from rasa.nlu.constants import (
    TEXT_ATTRIBUTE,
    TOKENS_NAMES,
    MESSAGE_ATTRIBUTES,
)
from rasa.nlu.components import Component
import os
import pickle
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
import spacy
nlp = spacy.load("en")
class Lemmatizer(Component):
    @classmethod
    def required_components(cls) -> List[Type[Component]]:
        """Specify which components need to be present in the pipeline."""
        return []
    #language_list = ["en", "fr"]
    language_list = None
    defaults = {}
    provides = ["text"]
    name = "Lemmatizationtokenizer"
    def __init__ (self, component_config: Dict[Text, Any] = None) -> None:
        super(). __init__ (component_config)
    def train(
        self, training_data: TrainingData, config: RasaNLUModelConfig, **kwargs: Any
    ) -> None:
        for example in training_data.training_examples:
            example.set(
                TEXT_ATTRIBUTE,
                self.tokenize(example.get(TEXT_ATTRIBUTE))
            )
    def process(self, message: Message, **kwargs: Any) -> None:
        message.text = self.tokenize(message.text)
        message.set(TEXT_ATTRIBUTE, self.tokenize(message.text))
    def tokenize(self, text: Text):
        ### English Lemmatization using spacy
        if text:
            text=text.replace("-"," ")
            text=text.lower()
            doc=nlp(str(text))
            lst=[]
            for token in doc:
                lst.append(token.lemma_)
            text=' '.join(lst)
            text=text.replace(" ?","?").replace(" .",".")
            text=text.lstrip().rstrip()
        return text
    @classmethod
    def load(
        cls,
        meta: Dict[Text, Any],
        model_dir: Optional[Text] = None,
        model_metadata: Optional["Metadata"] = None,
        cached_component: Optional["Component"] = None,
        **kwargs: Any,
    ) -> "Component":
        """Load this component from file."""
        if cached_component:
            return cached_component
        else:
            return cls(meta)

```

`rasa shell nlu`

```auto
NLU model loaded. Type a message and press enter to parse it.
Next message:
Flights to india
{
  "intent": {
    "name": "greet",
    "confidence": 0.533901035785675
  },
  "entities": [],
  "intent_ranking": [
    {
      "name": "greet",
      "confidence": 0.533901035785675
    },
    {
      "name": "goodbye",
      "confidence": 0.46609896421432495
    }
  ],
  "text": "flight to india"
}
Next message:
are you a bot?
{
  "intent": {
    "name": "greet",
    "confidence": 0.533901035785675
  },
  "entities": [],
  "intent_ranking": [
    {
      "name": "greet",
      "confidence": 0.533901035785675
    },
    {
      "name": "goodbye",
      "confidence": 0.46609896421432495
    }
  ],
  "text": "be -PRON- a bot?"
}
Next message:
I-have-dashes-everywhere
{
  "intent": {
    "name": "greet",
    "confidence": 0.533901035785675
  },
  "entities": [],
  "intent_ranking": [
    {
      "name": "greet",
      "confidence": 0.533901035785675
    },
    {
      "name": "goodbye",
      "confidence": 0.46609896421432495
    }
  ],
  "text": "i have dash everywhere"
}
```

---

<div class="post-metadata">

**Author:** ![mloubser](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/mloubser/32/13021_2.png) [@mloubser](https://forum.rasa.com/u/mloubser)\
**Post date:** [April 24, 2020, 6:53pm UTC](https://forum.rasa.com/t/rasa-com/27810/16 "2020-04-24T18:53:03Z")

</div>

Ok, I think those two issues are unrelated to your lemmatizer. The first says you should check your annotation of custom entities so you don’t have e.g. `This is an [entity](ent) with trailing whitespace ` or `this [entity,](ent) includes punctuation`.

the second warning says you should check that you haven’t defined two values for a single entity e.g.

```auto
- I work for [Rasa](company:rasa)
- I work for [Rasa](company:RasaTechnologies)

```

That is considered conflicting synonym definitions