Thank you for all the replies, got something working now!
However, I was trying to create some test data to check the performance and I noticed something strange, potentially it has to do with the fact that I am using Duckling to extract numbers and at the same time using roles.
For example, after running rasa test nlu , I get the following error in the test-results RegexEntityExtractor_errors.json file:
{
"text": "2 nights for 3 persons",
"entities": [
{
"start": 0,
"end": 1,
"value": "2",
"entity": "number",
"role": "nights"
},
{
"start": 13,
"end": 14,
"value": "3",
"entity": "number",
"role": "people"
}
],
"predicted_entities": [
{
"entity": "number",
"start": 0,
"end": 1,
"confidence_entity": 0.9993997812271118,
"role": "nights",
"confidence_role": 0.9364138841629028,
"value": "2",
"extractor": "DIETClassifier"
},
{
"entity": "number",
"start": 13,
"end": 14,
"confidence_entity": 0.9996367692947388,
"role": "nights",
"confidence_role": 0.9225713014602661,
"value": "3",
"extractor": "DIETClassifier"
}
]
}
But as you can see, this is clearly correct.
I will write some rules to cope with these incorrect errors, but just want to check if I was missing something here.
Does this have something to do with the fact that Pretrained extractors like the DucklingHTTPExtractor are not evaluated? (found this in the docs)