Entity roles: experimentation and issues

Hey!

Our team has lately been experimenting with entity roles in Rasa as it’s a great way of having better implementable dialogue capabilities. While we think it’s a great feature and we know it’s still pretty new and experimental, we are finding some problems while trying to implement models that use this feature.

Before entering into details, here is our setup.

  • Rasa 1.10.14 (NLU only, we use our own dialogue manager)
  • Language: Swedish (we adapt the examples in this post to English, however)
  • Config:
language: "sv"
pipeline:
  - name: HFTransformersNLP
    model_name: "bert"
    model_weights: "models/tokenization-models/bert-base-swedish-cased/"
  - name: LanguageModelTokenizer
  - name: LanguageModelFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "word"
    min_ngram: 1
    max_ngram: 5
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 10
  - name: DIETClassifier
    epochs: 300
  - name: EntitySynonymMapper

What we want:

  • Some user input to be labeled without a role. E.g. the utterance “Bill Smith” (entity: person name).
  • Some user input to be labeled with role(s). E.g. the utterance “My name is Amy Andersson” (entity: person name, role: caller).

Our problem is that we sometimes get an undesired mix of with and without role in the output. For instance:

  • In the utterance “Bill Smith”, “Bill” is labeled without a role (entity: person name) and “Smith” with a role (entity: person name, role: caller).

However, what we want is to have “Bill Smith” without a role. Sometimes in dialogue you want to have entity roles as the context provides enough information to disambiguate the role, but in some cases just the content of the utterance is not enough to assign a role.

To give more information, the data used for training Rasa looks as follows:

## intent:call
- my name is [Amanda]{"entity": "person_name", "role": "caller"} and I want to talk to [Frederick]{"entity": "person_name", "role": "contact"}
- I want to talk to [Amy Smith]{"entity": "person_name", "role": "contact"}
[...]

## intent:answer
- [Ellie](person_name)
- [Johan](person_name)
- [Johnson](person_name)
- [Emily Torres](person_name)
- [Bill Armstrong](person_name)
[...]

As you can see in this extract data, examples with roles and without roles are actually separated in two different intents. And the intents are actually alright in the predictions, but the roles get all mixed up and even affect the intent that should not have them. Then, we sometimes get mixed interpretations with entities mixed (with and without roles) like the one explained above.

Is there any way in Rasa (NLU) to avoid a behaviour like the one we are having? Something like making entities and roles (or the absence of them in this case) being more dependent of the guessed intent. Maybe some component in the pipeline that we have missed.

Or maybe there is some helpful new tool or improvement in Rasa 2.X that can help us with this. We still didn’t have the time to experiment with this new version, but we plan to do so!

We would be really glad to hear your input and ideas on this topic. Thanks!

Hi @jcanosan! Thanks for your interest in entity roles :slight_smile: You have quite an interesting use case. I’ll try to answer all your questions.

In general your pipeline looks good. You are not missing any component. One thing that I would try first is to increase the number of epochs of the DIETClassifier. We made the experience that entity roles need a bit longer to train. So maybe increase the number of epochs to 400 and check again. It might be that your problem is already solved.

Our problem is that we sometimes get an undesired mix of with and without role in the output.

I am afraid we don’t have a solution for that. We have a method that “cleans up” some of those mixed entity predictions, but we don’t have it in place for roles. Might be something we should to add.

Something like making entities and roles (or the absence of them in this case) being more dependent of the guessed intent.

You cannot influence that. As the DIETClassifier is training the intent and entities together, the classifier can “learn” that some entities just occur together with one intent. So model architecture allows for that, but there is no model parameter that you can tune to enforce this behaviour even more.

Or maybe there is some helpful new tool or improvement in Rasa 2.X that can help us with this.

Entity roles did not change in Rasa 2.x. So upgrading should not improve the performance of entity roles.

So please try to increase the number of epochs. Let me know if that worked.