Rasa.test() using Jupyter Notebooks

Hi ALL,

I had used the step-by-step guide to use RASA on Jupyter Notebook by following the link here Jupyter Notebooks. Sequently, I tried learning the Rasa tutorial (Building a simple FAQ Assistant) provided in this link, Tutorial: Building Assistants ,by applying those command on jupyter notebook. I am able to create and train a chatbot and I am very happy and thankful for the materials provided!!

However, I am having issues with testing the chatbot. From the Rasa tutorial (Tutorial: Building Assistants), it says to test our model against the test file, run the command:

rasa test --stories tests/conversation_tests.md

where conversation_tests.md stored all the end-to-end used tests. I would like to ask how can I run the above command in jupyter notebook. I understand from Jupyter Notebooks that the code to test the chatbot in Jupyter notebook is:

rasa.test(model_path, stories_directory, nlu_data_directory)

From what I understand we are supposed to test the chatbot by coming up with end-to-end used tests in conversation_tests.md and use rasa test to evaluate if our model(or chatbot) can indeed perform well in those cases we listed in conversation_tests.md. However, the test function in this case, passes in the path to the saved model and directories containing the stories and nlu data to evaluate on. This is not what I wanted, i wanted to evaluate my model on conversation_tests.md and would appreciate help on how this can be done using Jupyter notebook!!!

I read up on rasa.train and tried rasa.test(model_path, “tests/conversation_tests.md”, nlu_data_directory) and it didnt return me what I wanted.

Here is the reason why I wanted to evaluate my model on conversation_tests.md. Conceptually, we train the model using our stories and nlu data. So, when we test the model, we should not use back the same stories we used to train our model to test our model right? It will be expected to get 100% accuracy all the times?? This is what I am getting using rasa.test(model_path, stories_directory, nlu_data_directory) no matter how I changed my conversation_tests.md. I also looked into the temp files created by the command: stories_directory, nlu_data_directory = data.get_core_nlu_directories(training_files). Indeed, those temp files only contained the stories and nlu data that I used to train the data. I want to test the conversation_tests.md file!

Sorry for the long detailed description. It’s my first time posting on a forum to ask for help and I just wanted to get my point across. I have been spending quite some time reading forum and searching for resources online but I can’t find the answer that I am looking for. Would greatly appreciate if anyone can help. Thank you in advance!!.