# Custom preprocessor

**URL:** <https://forum.rasa.com/t/custom-preprocessor/52536>\
**Category:** Rasa Open Source\
**Created:** [April 12, 2022, 10:17am UTC](https://forum.rasa.com/t/custom-preprocessor/52536 "2022-04-12T10:17:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![artemsnegirev](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/artemsnegirev/32/16465_2.png) [@artemsnegirev](https://forum.rasa.com/u/artemsnegirev)\
**Post date:** [April 12, 2022, 10:17am UTC](https://forum.rasa.com/t/custom-preprocessor/52536/1 "2022-04-12T10:17:54Z")

</div>

I want create custom component to preprocess input message before tokenizer. This component replace several words.

What is type of component should i pick? I really want to separate this component from tokenizer component.

```python
    class ComponentType(Enum):
        """Enum to categorize and place custom components correctly in the graph."""

        MESSAGE_TOKENIZER = 0
        MESSAGE_FEATURIZER = 1
        INTENT_CLASSIFIER = 2
        ENTITY_EXTRACTOR = 3
        POLICY_WITHOUT_END_TO_END_SUPPORT = 4
        POLICY_WITH_END_TO_END_SUPPORT = 5
        MODEL_LOADER = 6

```

---

<div class="post-metadata">

**Author:** ![stephens](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/stephens/32/1632_2.png) [@stephens](https://forum.rasa.com/u/stephens)\
**Post date:** [April 12, 2022, 5:10pm UTC](https://forum.rasa.com/t/custom-preprocessor/52536/2 "2022-04-12T17:10:22Z")

</div>

I think the `MESSAGE_TOKENIZER` would be the best choice since you are modifying the user utterance like a tokenizer would do.

---

<div class="post-metadata">

**Author:** ![artemsnegirev](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/artemsnegirev/32/16465_2.png) [@artemsnegirev](https://forum.rasa.com/u/artemsnegirev)\
**Post date:** [April 13, 2022, 6:49am UTC](https://forum.rasa.com/t/custom-preprocessor/52536/3 "2022-04-13T06:49:07Z")

</div>

Thank you for your help!

I tried `MESSAGE_FEATURIZER`, it works too! I just always return None in features.

```python
    def _get_features(self, message, attribute) -> Tuple:
        if not message.get(attribute):
            return None, None

        text = message.get(attribute)
        text = self.keyword_processor.replace_keywords(text)
        
        message.set(
            attribute,
            text,
            add_to_output=True
        )

        return None, None

```

When i use `MESSAGE_TOKENIZER`, should i return empty tokens list?

---

<div class="post-metadata">

**Author:** ![stephens](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/stephens/32/1632_2.png) [@stephens](https://forum.rasa.com/u/stephens)\
**Post date:** [April 13, 2022, 3:47pm UTC](https://forum.rasa.com/t/custom-preprocessor/52536/4 "2022-04-13T15:47:50Z")

</div>

I think so but I would review the source code for the other featurizers.
