Is the lookup table working?

Rasa Open Source
1

I downloaded this lookup table demo:

and run the ‘food’ experiment, but found no difference after lookup table is used:

100%|██████████| 57/57 [00:00<00:00, 63.74it/s]
F1-Score:  0.7751422415446251
Precision: 0.8254441057849204
Accuracy:  0.8395061728395061
Classification report: 
              precision    recall  f1-score   support

        food       0.75      0.04      0.08        67
   no_entity       0.84      1.00      0.91       338

   micro avg       0.84      0.84      0.84       405
   macro avg       0.80      0.52      0.50       405
weighted avg       0.83      0.84      0.78       405

100%|██████████| 57/57 [00:00<00:00, 64.51it/s]
F1-Score:  0.7751422415446251
Precision: 0.8254441057849204
Accuracy:  0.8395061728395061
Classification report: 
              precision    recall  f1-score   support

        food       0.75      0.04      0.08        67
   no_entity       0.84      1.00      0.91       338

   micro avg       0.84      0.84      0.84       405
   macro avg       0.80      0.52      0.50       405
weighted avg       0.83      0.84      0.78       405

Why are the numbers exactly the same? The first set of numbers are food without lookup table, and the 2nd has lookup table.

2

Hey @twittmin, could you describe what you changed exactly between each run? (config files, command line arguments, etc)

3

Hi, @fede, I barely changed anything. Just run the program and got these numbers. Also, at the end, it receied this error:

Traceback (most recent call last):
  File "/Users/lingvisa/Nlp/chatbot/rasa_lookup_demo/run_lookup.py", line 234, in <module>
    run_demo(key, disp_bar=disp_bar)
  File "/Users/lingvisa/Nlp/chatbot/rasa_lookup_demo/run_lookup.py", line 127, in run_demo
    plot_metrics(metric_list)
  File "/Users/lingvisa/Nlp/chatbot/rasa_lookup_demo/run_lookup.py", line 178, in plot_metrics
    print_metrics(metric_list)
  File "/Users/lingvisa/Nlp/chatbot/rasa_lookup_demo/run_lookup.py", line 156, in print_metrics
    key = metric_list[0]["key"]
IndexError: list index out of range
4

@twittmin - what exact commands are you running? Since the numbers are exactly the same, maybe double check that you are really evaluating two different models?

5

@amn41I simply run this as instructed in the github instruction:

python run_lookup.py food

The code from the function:

    def run_demo(key, disp_bar=True):
        # runs the demo specified by key

        # get the data for this key and the configs
        training_data, training_data_lookup, test_data, model_dir = get_path_dicts(key)
        config_file = "configs/config.yaml"
        config_baseline = "configs/config_no_features.yaml"

        # run a baseline
        model_loc = train_model(training_data, config_baseline, model_dir)
        evaluate_model(test_data, model_loc)

        # run with more features in CRF
        model_loc = train_model(training_data, config_file, model_dir)
        evaluate_model(test_data, model_loc)

        # run with the lookup table
        model_loc = train_model(training_data_lookup, config_file, model_dir)
        evaluate_model(test_data, model_loc)

        # get the metrics
        metric_list = strip_metrics(key)

        # either print or plot them
        if disp_bar:
            plot_metrics(metric_list)
        else:
            print_metrics(metric_list)