Grab the NLU training dataset!

Hi.
I just downloaded and training the smalltalk.md and during training I started to get the warnings:

At runtme, I started to get the following warnings:

  • 2019-01-22 04:22:54+0000 [-] /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/label.py:151: builtins.DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use array.size > 0 to check that an array is not empty.

I noticed something related:

And in the smalltalk.md there are duplicate intents.

## intent:smart
- study
- you should study better
- you must learn
- be clever
- be more clever
- be smarter
- be smart
- get qualified

and ## intent:smart

  • you are so intelligent
  • you are a genius
  • smart
  • brilliant
  • clever
  • you are clever
  • you are so brainy
  • you’re really smart
  • you’re really brainy
  • you know a lot
  • you know a lot of things
  • you have a lot of knowledge
  • you know so much
  • how smart you are
  • how brainy you are
  • how clever you are
  • how brilliant you are
  • you are intelligent
  • you are qualified
  • you are really smart
  • you’re very smart
  • you are so smart
  • you are too smart
  • you are very clever
  • you are very intelligent
  • you are very smart
  • you’re intelligent
  • you’re a genius
  • you’re a smart cookie
  • you’re clever
  • you’re pretty smart
  • you’re qualified
  • why are you so smart
  • you are so clever

Same applies for intent:good - it’s there twice and maybe more.

Is it the duplicates that cause this problem? As the confidence level went to near zero after the retrain with this smalltalk.md: