# Evaluate stories against a model not working as expected **URL:** \ **Category:** Rasa Open Source\ **Created:** [September 30, 2019, 10:46am UTC](https://forum.rasa.com/t/evaluate-stories-against-a-model-not-working-as-expected/17705 "2019-09-30T10:46:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![lgonzalez](https://avatars.discourse-cdn.com/v4/letter/l/fbc32d/32.png) [@lgonzalez](https://forum.rasa.com/u/lgonzalez)\ **Post date:** [September 30, 2019, 10:46am UTC](https://forum.rasa.com/t/evaluate-stories-against-a-model-not-working-as-expected/17705/1 "2019-09-30T10:46:36Z")
Hi there, I want to evaluate my stories against an already trained model, without evaluating the NLU part. For this, I am using the HTTP API, with **e2e=false**. My test stories contains only intents that are in the domain file (actually, I am using exactly one of the stories that was used to train the model). My test story looks like: ![image](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/2X/6/67185d0748ea29d88844ed07c7dff9ec5efc3125.png) When I evaluate the story against the trained model I get low accurracy and the following warning: WARNING rasa\_core.training.dsl - Found unknown intent ‘None’ on line 3. Please, make sure that all intents are listed in your domain yaml. WARNING rasa\_core.training.dsl - Found unknown intent ‘None’ on line 5. Please, make sure that all intents are listed in your domain yaml. However, if I perform and end to end evaluation ( **e2e=true** ), using a story that looks like this: ![image](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/2X/a/acfe92ad4bdda7cbd2d12fb0b09c7944fd546007.png) I get 1 of accuracy and no warnings. **It seems like in e2e=false mode, RASA is trying to perform NLU evaluation too.** Could anyone help me with this issue? Thank you in advance. I am using **RASA CORE 0.14.5** ---
**Author:** ![imLew](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/imlew/32/3744_2.png) [@imLew](https://forum.rasa.com/u/imLew)\ **Post date:** [October 2, 2019, 12:11pm UTC](https://forum.rasa.com/t/evaluate-stories-against-a-model-not-working-as-expected/17705/2 "2019-10-02T12:11:09Z")
Hi lgonzalez, it looks like a bug. A couple things you could do to find out more: - Remove the NL input from the story you’re evaluating. It could be that rasa does not recognize this format if `e2e=false`. - Try the same thing on a newer version of rasa. ---
**Author:** ![lgonzalez](https://avatars.discourse-cdn.com/v4/letter/l/fbc32d/32.png) [@lgonzalez](https://forum.rasa.com/u/lgonzalez)\ **Post date:** [October 3, 2019, 9:38am UTC](https://forum.rasa.com/t/evaluate-stories-against-a-model-not-working-as-expected/17705/3 "2019-10-03T09:38:43Z")
Hi Nikolai, I’ve already tried “e2e=false” without NL input (see the first snippet in my previous post). The thing is, whether I set e2e=false or e2e=true, if I include the NL input I get 1of accuracy. Therefore it seems that the e2e flag is being ignored and it tries to do the NLU evaluation in both cases. If I change to a newer version of RASA I have to move to 1.X, which means major changes that I cannot assume right now. ---
**Author:** ![imLew](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/imlew/32/3744_2.png) [@imLew](https://forum.rasa.com/u/imLew)\ **Post date:** [October 4, 2019, 10:22am UTC](https://forum.rasa.com/t/evaluate-stories-against-a-model-not-working-as-expected/17705/4 "2019-10-04T10:22:54Z")
Sorry, I misunderstood your post. Perhaps the best way would be to include the nlu part but have the input always be one of the examples. That way NLU accuracy would be (close to) 100% and the results of the evaluation would reflect only the core model. ---
**Author:** ![lgonzalez](https://avatars.discourse-cdn.com/v4/letter/l/fbc32d/32.png) [@lgonzalez](https://forum.rasa.com/u/lgonzalez)\ **Post date:** [October 4, 2019, 1:46pm UTC](https://forum.rasa.com/t/evaluate-stories-against-a-model-not-working-as-expected/17705/5 "2019-10-04T13:46:52Z")
Thank you for your comments Nikolai, that’s actually the workaround I was implementing. However, I would like to know if someone managed to solve this.