# DIETClassifier splits one entity into list of token

**URL:** <https://forum.rasa.com/t/dietclassifier-splits-one-entity-into-list-of-token/37395>\
**Category:** Rasa Open Source\
**Created:** [November 25, 2020, 1:43pm UTC](https://forum.rasa.com/t/dietclassifier-splits-one-entity-into-list-of-token/37395 "2020-11-25T13:43:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nadachaabani1](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nadachaabani1/32/5919_2.png) [@nadachaabani1](https://forum.rasa.com/u/nadachaabani1)\
**Post date:** [November 25, 2020, 1:43pm UTC](https://forum.rasa.com/t/dietclassifier-splits-one-entity-into-list-of-token/37395/1 "2020-11-25T13:43:55Z")

</div>

I am migrating from RASA 1.5.3 to 1.10.14

In the previous version entity extraction is working fine, but now the slot witch have a special character as `/ or -` is splitted into list of token. For exemple, I want to extract `IT / Engineering` as a department entity but it’s extracted as a list `["it", "engineering"]`

 ![entity extraction](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/2X/d/dad186d7d3499665afb342e46229af91e4b5b8a6.png)

This is my NLU data

> ## intent: inform
> 
> - i am looking for IT / Engineering fields
> - i work for analyst / business strategy services industry in operations department
> - i am an engineer in Manufacturing / Aerospace

Department slot in the domain looks like:

> ```
> slots:
> department:
> type: unfeaturized
> 
> ```

And this is the pipeline I am using:

> ```
> # Configuration for Rasa NLU.
> language: en
> pipeline:
> - name: WhitespaceTokenizer
> - name: RegexFeaturizer
> - name: LexicalSyntacticFeaturizer
> - name: CountVectorsFeaturizer
> analyzer: char_wb
> min_ngram: 1
> max_ngram: 4
> - name: DIETClassifier
> batch_strategy: sequence
> epochs: 90
> ranking_length: 5
> - name: EntitySynonymMapper
> - name: "ResponseSelector"
> retrieval_intent: zigzag
> scale_loss: false
> 
> ```

I think that `DIETClassifier` component caused this error because when I remove it everything works fine. Which hyperparamater should I change to fix this problem please?

---

<div class="post-metadata">

**Author:** ![Arjaan](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/arjaan/32/5754_2.png) [@Arjaan](https://forum.rasa.com/u/Arjaan)\
**Post date:** [December 1, 2020, 6:17pm UTC](https://forum.rasa.com/t/dietclassifier-splits-one-entity-into-list-of-token/37395/2 "2020-12-01T18:17:50Z")

</div>

Hi @nadachaabani1 ,

Please see the explanation of this behavior, and possible solutions, in [this forum post](http://forum.rasa.com/t/rasa-not-picking-special-characters-in-an-entity/24169).

---

<div class="post-metadata">

**Author:** ![nadachaabani1](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nadachaabani1/32/5919_2.png) [@nadachaabani1](https://forum.rasa.com/u/nadachaabani1)\
**Post date:** [December 7, 2020, 9:44am UTC](https://forum.rasa.com/t/dietclassifier-splits-one-entity-into-list-of-token/37395/3 "2020-12-07T09:44:05Z")

</div>

Thank you so much @Arjaan. Now I’am using `SpacyTokenizer` and everything works fine.