NLU gets one-word entity right, misses extraction

Rasa Open Source
1

I have a terse intent which after training Rasa gets with extreme (near perfect) accuracy (typical confidence: 0.8774649)

However it is very consistently missing the extracted entity, even though user’s response is a single token. Its only extracting those entities that are part of the training set.

Now, it clearly knows the user’s intent, and there is only a single token. How can I help Rasa get this basic case right?

NB. the user can response more verbosely (and Rasa tends to get those) but typically, the responses will be terse, just a single word. Rasa nails the intent, I just need it to grab that token for the entity.

Anyone have any hints? Thanks! First post!

2

Hey @kombucha, welcome to the community!

To help you I have some questions first:

How much training data do you have for this entity? And what kind of entity is it? Would also be very helpful if you can post the pipeline you’re using for this, as well as the Rasa version.

3

Hi Akela,

The entity is unfeaturized.

My question here is really twofold:

  1. Why would it not correctly extract the entity, when it correctly predicts with high confidence a single token intent? I get that that there are no surrounding tokens to provide context for the CRF, but given it nails the prediction, shouldn’t it know that is the targeted entity?
  2. Is there a way to force the answer, ie to tell Rasa, look, if you predict this intent and it is a single token, label that as the targeted entity?

I’ve tried with differing size training sets up to 200+ examples. I’ve tried with ner_spacy before ner_crf, but in generally ner_crf first tends to give better results.

language: "en"

pipeline:
- name: "nlp_spacy"
- name: "tokenizer_spacy"
- name: "ner_crf"
- name: "ner_spacy"
- name: "intent_featurizer_spacy"
- name: "intent_classifier_sklearn"

To be clear, I’d like to understand how this works more than simply “getting it to work.”