Hello everyone,
I am running some entity self-learning experiments on Rasa version 1.10.14 and I want to access the entity confidence and the dictionary of the DIETClassifier to identify the out-of-vocabulary words of a user’s input. From what I have found, in this version, the only option would be to load the DIETClassifier somehow and access them directly.
Am I able to do that in that version? If so, how?
Thanks in advance.
@vkon using custom action:
To get the latest intent name inside a custom action:
intent= tracker.latest_message[‘intent’].get(‘name’)
To extract all entity names:
tracker.latest_message[‘entities’]
OR
you can extract the value using this:
ent = tracker.latest_message['entities'][0]['confidence']
“ent” is an array with a dict inside, so the “[0]” is for get in the dict and you can obtain anything inside using the name of the key, in this case “confidence”.
Ref:
"entities": [{
"end": 53,
"entity": "time",
"start": 48,
"value": "2017-04-10T00:00:00.000+02:00",
"confidence": 1.0,
"extractor": "DIETClassifier"
}]
I hope this will solve your issue? OR I hope you looking this?
Thank you for your quick response.
Unfortunately, in the RASA version I am using, this method is not viable. The entities contained in the tracker are as in the following form:
[{ ‘entity’: ‘time’, ‘start’: 12, ‘end’: 20, ‘value’: ‘2017-04-10T00:00:00.000+02:00’, ‘extractor’: ‘DIETClassifier’, ‘processors’: [‘EntitySynonymMapper’] }]
This is what “tracker.latest_message[‘entities’]” returns, while “ent = tracker.latest_message[‘entities’][0][‘confidence’]” results in an error as the confidence field is missing.
What I seek to achieve is some other way to access the ‘confidence’ of the entities and to also retrieve the vocabulary of the DIETClassifier (if it uses one).
@vkon can you check the tracker.events printout and confirm what you seeing the outputs?
I checked the “tracker.events”, but the same issue still holds. It returns confidence for the intents, but not for the entities.
Referring to DIETClassifier adds a confidence value to entity predictions, the DIETClassifier was programmed to return the confidence of entities from Rasa 2.0 and later, but my current RASA version is 1.10.14.
@vkon yes, I’m aware of this DIETClassifier. But, still it should have some solution, that why suggesting you. can you share the screenshot what you get in return for tracker.events in logs?
The response I am getting is a very long one (it would need 3 full screen screenshots) and it contains some sensitive data, making it difficult to share here. I have reviewed all of this information before posting, but I had not luck of finding what I was looking there.
It seems that the CRF layer of the DIETClassifier in old RASA versions did not have the option to return the confidence of predictions, that is why the tracker has nothing related to entity confidence.
From what I have seen, a solution would be to extend the CRF layer with a custom one that is able to return the confidence as well. Though, I am not sure what are the steps to achieve that and if the RASA model pipeline would pose a problem, that might require changes as well.
Is there any other direction we could take to retrieve the entities’ confidence?
@vkon as per my knowledge I tried to give you all possible solutions, as we have version constraints.
@vkon can you confirm me from the rasa shell -debug that are you able to see the payload: metadata? and in that te confidence value of the entity?
I tried running “rasa shell --debug”. It gives a lot of information, but unfortunately nothing related to entity confidence.
This is the only result related to entities from the whole debug output.
Are you sure that the reason is not that Diabetes is mapped via the EntitySynonymMapper and therefore the confidence is always 1 and not shown? RASA either recognizes the synonym or not.
Just my 2 cents, did not test my theory though :).
