ValueError: You can only specify one policy per model for comparison

Rasa Open Source
1

@akelad @souvikg10 Do you know about this issue ?

python -m rasa_core.train compare -c policy_config1.yml policy_config2.yml -d domain.yml -s stories_folder -o comparison_models --runs 3 --percentages 0 5 25 50 70 90 95

2018-12-13 14:09:08 INFO     root  - Starting run 1/3
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/aziz/Documents/compare/rasa_core/train.py", line 361, in <module>
    additional_args)
  File "/home/aziz/Documents/compare/rasa_core/train.py", line 277, in do_compare_training
    additional_arguments)
  File "/home/aziz/Documents/compare/rasa_core/train.py", line 229, in train_comparison_models
    raise ValueError("You can only specify one policy per " ValueError: You can only specify one policy per model for comparison

ValueError: You can only specify one policy per model for comparison

reference: Evaluating and Testing

2

Hey @azizullah2017!

My initial guess here is that there is more than one policy defined in your config files. Could you post the contents of policy_config1.yml and policy_config2.yml for me please?

3

yes, i do have that.

4

Yeah it’s designed to test individual policies, secifically KerasPolicy vs EmbeddingPolicy for example. Meaning only one policy definition per file!

5

@akelad @MetcalfeTom thanks for the reply.

policy_config1.yml

policies:
- name: EmbeddingPolicy
  epochs: 2000
  attn_shift_range: 5
- name: FormPolicy

policy_config2.yml

policies:
  - name: "KerasPolicy"
    featurizer:
    - name: MaxHistoryTrackerFeaturizer
      max_history: 10
      state_featurizer:
        - name: BinarySingleStateFeaturizer
  - name: FormPolicy

is the error is becuase of form policy I have mentioned in my policies ??

6

Yup, remove the FormPolicy from both files and try again

1 Like
7

@MetcalfeTom does this mean I need to strip all Form-related stories to be able to compare results? Otherwise I receive this error:

2019-02-17 13:12:35 INFO     root  - Starting to train EmbeddingPolicy round 1/7 with 0% exclusion
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/rasa_core/train.py", line 361, in <module>
    additional_args)
  File "/usr/local/lib/python3.6/site-packages/rasa_core/train.py", line 276, in do_compare_training
    additional_arguments)
  File "/usr/local/lib/python3.6/site-packages/rasa_core/train.py", line 246, in train_comparison_models
    dump_stories=dump_stories)
  File "/usr/local/lib/python3.6/site-packages/rasa_core/train.py", line 178, in train_dialogue_model
    policies=policies)
  File "/usr/local/lib/python3.6/site-packages/rasa_core/agent.py", line 199, in __init__
    "You have defined a form action, but haven't added the "
rasa_core.domain.InvalidDomain: You have defined a form action, but haven't added the FormPolicy to your policy ensemble.
8

Apparently, you cannot have forms in your domain file when using rasa_core.train compare