[SOLVED]
Basically, I was training my DIETClassifier to recognize some words in my nlu examples as entities that are already recognized by spacy. So both Spacy and my DIETClassifier were extracting those words.
Running rasa shell nlu helped me debug