Hey!
Our team has lately been experimenting with entity roles in Rasa as it’s a great way of having better implementable dialogue capabilities. While we think it’s a great feature and we know it’s still pretty new and experimental, we are finding some problems while trying to implement models that use this feature.
Before entering into details, here is our setup.
- Rasa 1.10.14 (NLU only, we use our own dialogue manager)
- Language: Swedish (we adapt the examples in this post to English, however)
- Config:
language: "sv"
pipeline:
- name: HFTransformersNLP
model_name: "bert"
model_weights: "models/tokenization-models/bert-base-swedish-cased/"
- name: LanguageModelTokenizer
- name: LanguageModelFeaturizer
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
analyzer: "word"
min_ngram: 1
max_ngram: 5
- name: CountVectorsFeaturizer
analyzer: "char_wb"
min_ngram: 1
max_ngram: 10
- name: DIETClassifier
epochs: 300
- name: EntitySynonymMapper
What we want:
- Some user input to be labeled without a role. E.g. the utterance “Bill Smith” (entity: person name).
- Some user input to be labeled with role(s). E.g. the utterance “My name is Amy Andersson” (entity: person name, role: caller).
Our problem is that we sometimes get an undesired mix of with and without role in the output. For instance:
- In the utterance “Bill Smith”, “Bill” is labeled without a role (entity: person name) and “Smith” with a role (entity: person name, role: caller).
However, what we want is to have “Bill Smith” without a role. Sometimes in dialogue you want to have entity roles as the context provides enough information to disambiguate the role, but in some cases just the content of the utterance is not enough to assign a role.
To give more information, the data used for training Rasa looks as follows:
## intent:call
- my name is [Amanda]{"entity": "person_name", "role": "caller"} and I want to talk to [Frederick]{"entity": "person_name", "role": "contact"}
- I want to talk to [Amy Smith]{"entity": "person_name", "role": "contact"}
[...]
## intent:answer
- [Ellie](person_name)
- [Johan](person_name)
- [Johnson](person_name)
- [Emily Torres](person_name)
- [Bill Armstrong](person_name)
[...]
As you can see in this extract data, examples with roles and without roles are actually separated in two different intents. And the intents are actually alright in the predictions, but the roles get all mixed up and even affect the intent that should not have them. Then, we sometimes get mixed interpretations with entities mixed (with and without roles) like the one explained above.
Is there any way in Rasa (NLU) to avoid a behaviour like the one we are having? Something like making entities and roles (or the absence of them in this case) being more dependent of the guessed intent. Maybe some component in the pipeline that we have missed.
Or maybe there is some helpful new tool or improvement in Rasa 2.X that can help us with this. We still didn’t have the time to experiment with this new version, but we plan to do so!
We would be really glad to hear your input and ideas on this topic. Thanks!
You have quite an interesting use case. I’ll try to answer all your questions.