# Advices for creating a data set **URL:** \ **Category:** Rasa Open Source\ **Created:** [August 15, 2018, 9:08pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480 "2018-08-15T21:08:21Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![datistiquo](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/datistiquo/32/124_2.png) [@datistiquo](https://forum.rasa.com/u/datistiquo)\ **Post date:** [August 15, 2018, 9:08pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/1 "2018-08-15T21:08:21Z")
I think you guys have made same experiences. If you have multiple intents and add examples for one intent to do recognition better, afterwards the classification of other Intents might suffer. It is like chasing chicken or egg. Sometimes sentences only differ in one word for different intents like `How much I get` or differntly from `How I get`… Any advices? ---
**Author:** ![akelad](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/akelad/32/69_2.png) [@akelad](https://forum.rasa.com/u/akelad)\ **Post date:** [August 24, 2018, 1:28pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/2 "2018-08-24T13:28:02Z")
Could you provide an example of intents where this would be the case? ---
**Author:** ![datistiquo](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/datistiquo/32/124_2.png) [@datistiquo](https://forum.rasa.com/u/datistiquo)\ **Post date:** [August 24, 2018, 1:49pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/3 "2018-08-24T13:49:44Z")
No it is in general. I start building it from empty set and fill it sentence for sentence. So, I found the quantity is also influencing the classification. I struggle how find the right number of examples per intent.Of course you can have intents where examples within having similiar structure/words like above short example? Also, when you have a more common intent, then you need somehow a little oversampling for this intent. Otherwise the misclassification is too high. How does imbalance within one intent for a specific sentence type (structure/count of specific words) is influencing the algo? You have one intent and add various sentences to it and they have different structures but you have more examples of one type compared to other types. ---
**Author:** ![akelad](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/akelad/32/69_2.png) [@akelad](https://forum.rasa.com/u/akelad)\ **Post date:** [August 27, 2018, 11:02am UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/4 "2018-08-27T11:02:50Z")
“In general” is always hard to answer. It depends, if you really need to handle these differently then create a new intent. But if you have something that can be distinguished by entities, then use entities. If you have a lot of intents that are hard to distinguish between, then you should probably reconsider your intent classes. The tensorflow embedding pipeline is better at handling these cases though. As for more examples for an intent, you should try to keep the number of examples a little balanced, but it’s not a huge problem. But really you shouldn’t worry too much about these things in general, only when these issues actually arise ---
**Author:** ![AnilLG](https://avatars.discourse-cdn.com/v4/letter/a/e47774/32.png) [@AnilLG](https://forum.rasa.com/u/AnilLG)\ **Post date:** [September 10, 2018, 10:54am UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/5 "2018-09-10T10:54:22Z")
I have many sentences like, 1. Can I pay with PayPal? 2. Can I pay with VIsa? My entity name is payment\_channels, It is possible to write one sentence for many same sentences only entity name is changed? ---
**Author:** ![akelad](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/akelad/32/69_2.png) [@akelad](https://forum.rasa.com/u/akelad)\ **Post date:** [September 12, 2018, 7:48am UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/6 "2018-09-12T07:48:32Z")
Yeah you can do that ---
**Author:** ![datistiquo](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/datistiquo/32/124_2.png) [@datistiquo](https://forum.rasa.com/u/datistiquo)\ **Post date:** [September 21, 2018, 3:04pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/7 "2018-09-21T15:04:29Z")
Hey, I need some tips for NER\_CRF. I have 1000 sentences and and a list of entities. I want to use them for training my NER since intent classification works so far good. The way of going would be now to use the sentences and list of entities with chatito to get training data. But you need to take care of the right balances of different sentences structures. It would be not so good to throw all sentences in training data duplicated with other entity values, right? Then arises the problem that you need to find all different sentcence structures/types (manually) and take only one of each and use it with different entity values. So you get let’s say 20 sentences of each type just with different entities. Would this be the right strategy? Since this is manual labour. I thought you can do simple document classification like topic modelling to filter all different sentences structures. Such that I have maybe 50 different structures out of my 1000 examples. Those 50 I can use to sample with entities… 😀 @akelad ---
**Author:** ![datistiquo](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/datistiquo/32/124_2.png) [@datistiquo](https://forum.rasa.com/u/datistiquo)\ **Post date:** [September 24, 2018, 4:42pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/8 "2018-09-24T16:42:11Z")
It is really difficult to train NER\_CRF. I experience that often you need both surrounding words in training data such that it will be recognised. I use the ngram features but it generalizes though very badly. ---
**Author:** ![datistiquo](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/datistiquo/32/124_2.png) [@datistiquo](https://forum.rasa.com/u/datistiquo)\ **Post date:** [September 27, 2018, 12:41pm UTC](https://forum.rasa.com/t/advices-for-creating-a-data-set/480/9 "2018-09-27T12:41:04Z")
I found the following results: The feature `low` for NER\_CRF is very bad because then just it pays too much attention to seen entities. Removing it increases results drastically. I have many sentences with surrounding words of entity but also trained typical sentences where entity is at the end of sentence. But in testing, the entities at end of sentence are rarely reconized! It is really difficult to see why this happens. But I think I have to oversample those sentences with entity at the end maybe?