How can i test a model without setting it as default model

sachin_333_nv · June 10, 2020, 1:38pm

Hi, i created a model using REST api (e.g. http://localhost:5005/model/train)

it returns newly created model file name.

Now i wants to evaluate/test this model using REST api before setting it as a default model.

currently what i’m trying, i’m invoking below rest api as : http://localhost:5005/model/test/intents?model=generated_model_file_name.tar.gz

with payload

intent:greet

hey
hello
hi
good morning
good evening
hey there

response i’m getting response OK with response as

{ “intent_evaluation”: null, “entity_evaluation”: {}, “response_selection_evaluation”: null }

is what i’m doing is right? and is this right way to test a model?

erohmensing · June 11, 2020, 12:28pm

Hey @sachin_333_nv, that request looks okay to me, I’m not sure why it’s not running an evaluation, unless this test data that you passed was also the training data for that model? If so, your model needs to have at least 2 intents to make predictions or run evaluations.

If you have 2 or more intents in generated_model_file_name.tar.gz, can you try running the server with the --debug flag and posting the logs when you try to hit the testing endpoint?

sachin_333_nv · June 11, 2020, 4:14pm

thanks @erohmensing

–debug flag really helped to find the problem. It says - There is no trained model for ‘ResponseSelector’: The component is either not trained or didn’t receive enough training data.

can u let me know what is going wrong.

below is the console log for creating model using REST API (http://localhost:5005/model/train)

it creates a file named - 20200611-154124.tar.gz

2020-06-11 15:51:42 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpcdpabgot/nlu.md' is 'md'.
2020-06-11 15:51:42 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpcdpabgot/responses.md' is 'unk'.
2020-06-11 15:51:42 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpcdpabgot/stories.md' is 'unk'.
2020-06-11 15:51:42 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpcdpabgot/nlu.md' is 'md'.
2020-06-11 15:51:42 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpcdpabgot/nlu.md' is 'md'.
2020-06-11 15:51:42 DEBUG    rasa.model  - Extracted model to '/tmp/tmp196q4l1i'.
2020-06-11 15:51:42 INFO     rasa.model  - Data (core-config) for Core model section changed.
2020-06-11 15:51:42 INFO     rasa.model  - Data (nlu-config) for NLU model section changed.
2020-06-11 15:51:42 INFO     rasa.model  - Data (nlg) for NLG templates section changed.
2020-06-11 15:51:42 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Generated trackers will be deduplicated based on their unique last 5 states.
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Number of augmentation rounds is 3
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Starting data generation round 0 ... (with 1 trackers)
Processed Story Blocks: 100%|██████████| 5/5 [00:00<00:00, 1024.60it/s, # trackers=1]
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Finished phase (5 training samples found).
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Data generation rounds finished.
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Found 0 unused checkpoints
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Starting augmentation round 0 ... (with 5 trackers)
Processed Story Blocks: 100%|██████████| 5/5 [00:00<00:00, 667.16it/s, # trackers=5]
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Finished phase (30 training samples found).
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Starting augmentation round 1 ... (with 30 trackers)
Processed Story Blocks: 100%|██████████| 5/5 [00:00<00:00, 189.10it/s, # trackers=20]
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Finished phase (125 training samples found).
2020-06-11 15:51:42 DEBUG    rasa.core.training.generator  - Starting augmentation round 2 ... (with 50 trackers)
Processed Story Blocks: 100%|██████████| 5/5 [00:00<00:00, 131.76it/s, # trackers=24]
2020-06-11 15:51:43 DEBUG    rasa.core.training.generator  - Finished phase (231 training samples found).
2020-06-11 15:51:43 DEBUG    rasa.core.training.generator  - Found 231 training trackers.
2020-06-11 15:51:43 DEBUG    rasa.core.training.generator  - Subsampled to 226 augmented training trackers.
2020-06-11 15:51:43 DEBUG    rasa.core.training.generator  - There are 5 original trackers.
2020-06-11 15:51:43 DEBUG    rasa.core.agent  - Agent trainer got kwargs: {}
2020-06-11 15:51:43 DEBUG    rasa.core.featurizers  - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(NoneType))...
Processed trackers: 100%|██████████| 5/5 [00:00<00:00, 1081.79it/s, # actions=16]
2020-06-11 15:51:43 DEBUG    rasa.core.featurizers  - Created 16 action examples.
Processed actions: 16it [00:00, 5178.95it/s, # examples=16]
2020-06-11 15:51:43 DEBUG    rasa.core.policies.memoization  - Memorized 16 unique examples.
2020-06-11 15:51:43 DEBUG    rasa.core.featurizers  - Creating states and action examples from collected trackers (by MaxHistoryTrackerFeaturizer(LabelTokenizerSingleStateFeaturizer))...
Processed trackers: 100%|██████████| 231/231 [00:00<00:00, 356.95it/s, # actions=126]
2020-06-11 15:51:43 DEBUG    rasa.core.featurizers  - Created 126 action examples.
2020-06-11 15:51:43 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow train graph...
2020-06-11 15:51:48 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow train graph.
Epochs: 100%|██████████| 100/100 [00:14<00:00,  7.05it/s, t_loss=0.078, loss=0.003, acc=1.000]
2020-06-11 15:52:02 INFO     rasa.utils.tensorflow.models  - Finished training.
2020-06-11 15:52:03 INFO     rasa.core.agent  - Persisted model to '/tmp/tmp1w3mjti0/core'
2020-06-11 15:52:03 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpcdpabgot/nlu.md' is 'md'.
2020-06-11 15:52:03 INFO     rasa.nlu.training_data.training_data  - Training data stats:
2020-06-11 15:52:03 INFO     rasa.nlu.training_data.training_data  - Number of intent examples: 43 (7 distinct intents)
2020-06-11 15:52:03 INFO     rasa.nlu.training_data.training_data  -   Found intents: 'greet', 'goodbye', 'mood_unhappy', 'affirm', 'bot_challenge', 'deny', 'mood_great'
2020-06-11 15:52:03 INFO     rasa.nlu.training_data.training_data  - Number of response examples: 0 (0 distinct responses)
2020-06-11 15:52:03 INFO     rasa.nlu.training_data.training_data  - Number of entity examples: 0 (0 distinct entities)
2020-06-11 15:52:03 DEBUG    rasa.nlu.training_data.training_data  - Validating training data...
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2020-06-11 15:52:03 DEBUG    rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - No text provided for response attribute in any messages of training data. Skipping training a CountVectorizer for it.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2020-06-11 15:52:03 DEBUG    rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - No text provided for response attribute in any messages of training data. Skipping training a CountVectorizer for it.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:03 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
2020-06-11 15:52:03 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow train graph...
2020-06-11 15:52:10 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow train graph.
Epochs: 100%|██████████| 100/100 [00:15<00:00,  6.55it/s, t_loss=1.456, i_loss=0.075, i_acc=1.000]
2020-06-11 15:52:25 INFO     rasa.utils.tensorflow.models  - Finished training.
2020-06-11 15:52:25 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:25 INFO     rasa.nlu.model  - Starting to train component EntitySynonymMapper
2020-06-11 15:52:25 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:25 INFO     rasa.nlu.model  - Starting to train component ResponseSelector
2020-06-11 15:52:25 INFO     rasa.nlu.selectors.response_selector  - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents.
2020-06-11 15:52:25 DEBUG    rasa.nlu.classifiers.diet_classifier  - **Cannot train 'ResponseSelector'. No data was provided. Skipping training of the classifier**.
2020-06-11 15:52:25 INFO     rasa.nlu.model  - Finished training component.
2020-06-11 15:52:25 INFO     rasa.nlu.model  - Successfully saved model into '/tmp/tmp1w3mjti0/nlu'

------- payload data (created with default RASA config,domain,nlu,stories file)-------

**payload :** 

{
   "domain":"intents:
  - greet
  - goodbye
  - affirm
  - deny
  - mood_great
  - mood_unhappy
  - bot_challenge

responses:
  utter_greet:
  - text: "Hey! How are you?"

  utter_cheer_up:
  - text: "Here is something to cheer you up:"
    image: "https://i.imgur.com/nGF1K8f.jpg"

  utter_did_that_help:
  - text: "Did that help you?"

  utter_happy:
  - text: "Great, carry on!"

  utter_goodbye:
  - text: "Bye"

  utter_iamabot:
  - text: "I am a bot, powered by Rasa."
",
   "config":"language: en
pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: MappingPolicy",
   "nlu":"## intent:greet
- hey
- hello
- hi
- good morning
- good evening
- hey there


## intent:goodbye
- bye
- goodbye
- see you around
- see you later

## intent:affirm
- yes
- indeed
- of course
- that sounds good
- correct

## intent:deny
- no
- never
- I don't think so
- don't like that
- no way
- not really

## intent:mood_great
- perfect
- very good
- great
- amazing
- wonderful
- I am feeling very good
- I am great
- I'm good

## intent:mood_unhappy
- sad
- very sad
- unhappy
- bad
- very bad
- awful
- terrible
- not very good
- extremely sad
- so sad

## intent:bot_challenge
- are you a bot?
- are you a human?
- am I talking to a bot?
- am I talking to a human?

",
   "responses":"utter_greet:
  - text: "Hey! How are you?"

  utter_cheer_up:
  - text: "Here is something to cheer you up:"
    image: "https://i.imgur.com/nGF1K8f.jpg"

  utter_did_that_help:
  - text: "Did that help you?"

  utter_happy:
  - text: "Great, carry on!"

  utter_goodbye:
  - text: "Bye"

  utter_iamabot:
  - text: "I am a bot, powered by Rasa."",
   "stories":"## happy path
* greet
  - utter_greet
* mood_great
  - utter_happy

## sad path 1
* greet
  - utter_greet
* mood_unhappy
  - utter_happy
  - utter_did_that_help
* affirm
  - utter_happy

## sad path 2
* greet
  - utter_greet
* mood_unhappy
  - utter_happy
  - utter_did_that_help
* deny
  - utter_goodbye

## say goodbye
* goodbye
  - utter_goodbye

## bot challenge
* bot_challenge
  - utter_iamabot",
   "force":false,
   "save_to_default_model_directory":true
}

--------( Perform an intent evaluation)------------ http://localhost:5005/model/test/intents?model=models/20200611-154124.tar.gz

payload

## intent:greet
- hey
- hello
- hi
- good morning
- good evening

response

{
    "intent_evaluation": null,
    "entity_evaluation": {},
    "response_selection_evaluation": null
}

console log

/opt/venv/lib/python3.7/site-packages/rasa/utils/common.py:364: UserWarning: No valid configuration given to load agent.
2020-06-11 16:10:16 DEBUG    rasa.model  - Extracted model to '/tmp/tmpecgio5tu'.
2020-06-11 16:10:16 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-11 16:10:17 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-11 16:10:17 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-11 16:10:19 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-11 16:10:19 DEBUG    rasa.nlu.classifiers.diet_classifier  - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpecgio5tu/nlu' doesn't exist?
2020-06-11 16:10:19 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-11 16:10:20 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-11 16:10:20 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-11 16:10:21 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-11 16:10:21 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
2020-06-11 16:10:21 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-11 16:10:22 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-11 16:10:22 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-11 16:10:24 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-11 16:10:24 DEBUG    rasa.nlu.classifiers.diet_classifier  - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpecgio5tu/nlu' doesn't exist?
2020-06-11 16:10:24 DEBUG    rasa.nlu.training_data.loading  - Training data format of '/tmp/tmpfe5v2nch' is 'md'.
2020-06-11 16:10:24 INFO     rasa.nlu.test  - Running model for predictions:
  0%|          | 0/5 [00:00<?, ?it/s]2020-06-11 16:10:24 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-06-11 16:10:24 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2020-06-11 16:10:24 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-06-11 16:10:24 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2020-06-11 16:10:24 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-06-11 16:10:24 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2020-06-11 16:10:24 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-06-11 16:10:24 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
 80%|████████  | 4/5 [00:00<00:00, 39.98it/s]2020-06-11 16:10:24 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-06-11 16:10:24 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
100%|██████████| 5/5 [00:00<00:00, 37.31it/s]
2020-06-11 16:10:24 INFO     rasa.nlu.test  - Entity evaluation results:

erohmensing · June 16, 2020, 12:09pm

If you are not trying to use the response selector, this should be fine! I think there must be a different issue here

erohmensing · June 16, 2020, 12:17pm

Are there more logs from the end of your last console log? Looks like it was about to show entity evaluation results – I’d like to see those for intent evaluation.

If you want to remove the ResponseSelector errors, you can remove

  - name: ResponseSelector
    epochs: 100

from your pipeline since that bot isn’t using it.

Topic		Replies	Views
Is it possible to train Core without specifying model? Rasa Open Source	2	694	October 31, 2019
Solution for file models create by API Rasa Open Source testing	1	214	August 3, 2023
Ran training, but it finished without a trained model Rasa Open Source	4	884	June 7, 2021
Trying to understand model training and testing Rasa Open Source	0	318	February 15, 2022
Http API for model evaluation using cross validation Rasa Open Source	0	290	March 4, 2020

How can i test a model without setting it as default model

intent:greet

Related topics