[SOLVED]RASA story misbehaving

Rasa Open Source
1

path1

  • sequence
    • action_check_intent
    • slot{“conv_id”: “1”}
  • inform
    • action_check_reg
    • slot{“regid”: “1”}
  • inform
    • action_check_proc
    • slot{“procid”: “1”}
  • inform
    • action_check_iss
    • slot{“issid”: “1940”}
  • inform
    • action_check_com
    • slot{“comid”: “1001”}
    • action_validate
    • slot{“is_valid”: true}
    • action_seq
  • affirm OR deny
    • utter_ack

path2

  • posting_date
    • action_check_intent
    • slot{“conv_id”: “1”}
  • inform
    • action_check_reg
    • slot{“regid”: “1”}
  • inform
    • action_check_proc
    • slot{“procid”: “1”}
  • inform
    • action_check_iss
    • slot{“issid”: “1940”}
  • inform
    • action_check_com
    • slot{“comid”: “1001”}
    • action_validate
    • slot{“is_valid”: true}
    • utter_file
  • file
    • action_post
  • affirm OR deny
    • utter_ack

Everything runs fine apart from the fact that only path 1 gets executed irrespective of the intent . Which makes to wonder , is rasa core stories indeed full proof ? FYI i have used interactive learning to generate these stories.

Any help is appreciated :slight_smile:

2

It is a machine learning model so it is never full proof. However there are steps to avoid falling in the trap

I am not fully sure what is going wrong for you here,

Path 1 is working fine and path 2 is not but at which point? irrespecive of which intent? What are your hyperparameters?

Thanks for clarifying your problem

3

As you can see both the paths start with different intents , even if i enter an input whose intent is classified as “posting_date” it takes “path1” in place of “path2” . When i enter an input which get classified as “sequence” it follows path2(probably) , executes “utter_file” and then executes “action_seq” . Sound crazy right ? but that’s what’s happening . Interestingly , as i separate the two paths in to different story files and execute it they seem to work flawlessly.

following is the dialogue trainer i used :

agent = Agent(domain_file, policies = [MemoizationPolicy(), KerasPolicy()])

data = agent.load_data(training_data_file)

agent.train(data,epochs = 200,batch_size = 10,validation_split = 0.2)

agent.persist(model_path)

I haven’t done any changes to the usual hyperparameters , if you feel i should change them do suggest. FYI i am already getting an accuracy of above 80% with the default hyperparameters , still no luck

It seems rasa story is still a mystery :disappointed_relieved:

4

Your accuracy is based on your training set- that is not the proper indication of accuracy. that is your training accuracy.

maybe set augmentation to 0 , augmentation factor randomly glues together stories to create larger stores from your stories file.

run rasa with --debug flag on to check at which point the problem occurs. After interactive learning, when you have exported your training data, did you retrain the model?

5

augmentation can be set in agent.load_data() but by saying “run rasa with --debug” did you mean run rasa_core_sdk with --debug ?? and yes i did retrain my model after interactive learning . The issue appears as i amalgamate all the stories , unlike when i keep them in different story files they seem to work just fine.

thanks for the help though :slight_smile:

6

I meant running rasa core server with debug flag

7

hey will you be able to tell me how to add this in python api , since i have scripts for every task i would like to know if “–debug” can be set through python api . I am not finding a way to do so

8

well it depends on how you are running the server? is it flask? I would recommend using the run.py script from rasa as you will be able to take advantage of the server setup.

i think if you add this by importing utils from rasa

utils.configure_colored_logging(DEBUG)
    utils.configure_file_logging(DEBUG,
                                 logfile)

maybe it could work

9

You say that it works fine when you seperate the stories in different files. maybe when you keep it in one story file the newline for each story might not be given so it considers as one single story. this could be problem with the editor you are using or you simply forgot to give newline at the end of a story/path. Sometimes the solution is that simple haha

stories.md

#story_01
*sequence
       - action_check_intent
       - slot{“conv_id”: “1”}
*inform
       - action_check_reg
       - slot{“regid”: “1”}

#story_02
* posting_date
      - action_check_intent
      - slot{“conv_id”: “1”}
* inform
      - action_check_reg
      - slot{“regid”: “1”}

Try the format you never know it might be that simple!

1 Like
10

@stalinsabu sorry to let you know but things ain’t that simple all the time , please don’t joke around that’s totally unnecessary

11

will try it out

12

I don’t think @stalinsabu is wrong though.

since splitting the stories in multiple files shouldn’t really matter for training the model as all stories are concatenated together before training them.

if it works putting them in two separate files and then doesn’t work upon merging, there could be something wrong on how you merge before providing data to the API.

keep in mind you can also provide the folder path to the rasa API who will merge the stories together before training.

how you organise your training data has no impact on how the model is trained unless you are preparing the data to train by code, you should look out for format errors

13

You misunderstood , by saying stories in different file, i meant in two different bots. The above 2 stories represents 2 questions asked for the bot.when i create 2 bots handling the 2 questions it works fine but as soon as i allow a single bot to handle the 2 questions by amalgamating stories , it misbehaves.So i don’t think that suggestion will work. Even i feel i might have to tune some hyperparameters.

14

I am sorry but i don’t think, i had a misunderstanding. having two stories file does not necessarily means you have trained two models. It wasn’t quite clear to me what you meant with two stories file. we split our stories in 10 but train one model.

Maybe add a few more stories, it is a neural network you are trying to train so it will require data.

You can reduce the max_history feature which by default is 5 to 3 for Memoization policy to pick up the next action which is built for precision

15

Yes i totally understood your point and as a matter of fact for training the nlu model i did the same . Issue is , our bot has a very limited paths of execution (As it’s a banking and finance bot), hence we are unable to add more data into the stories.md file .

Regarding max_history it’s a hyperparameter of memoization_policy and by default it’s set to None(in python) , i did try tuning it from 1 to 3 . Didn’t work

FYI i tried setting augmentation_factor to 0 , it didn’t work .

Is there any specific way of adding data to stories.md file ? i have added a few but other than that i am out of ideas. It seems rasa’s actual story execution doesn’t conform with their docs , it says memoization_policy only considers the stories provided by us and does no additional prediction but i tried that by removing keras_policy and guess what no luck :confused:

16

Max history is not by default None but 5, ( if you take a look at singleStateFeaturizer) - you can set it to 0, if you want precision, and max history is a parameter for both Keras and Memoization

for Keras Policy - usually max history of 3 will suffice

You can refer to my article on each policies

best way as far as i know is using interactive learning. sounds odd but talk to your bot.

I always argue the fact of using RNN with limited data and if your path is really simple, try a simpler classifier like Decision tree or Sklearn SVM to adapt to unforeseen paths while still using a deterministic path for a particular process.

Memoization does work deterministically if you set history to 0 and augmentation 0, however it is designed for precision and no recall, meaning for false positives you have no answers with memoization. though i find it really complicated to understand as how it uses the features. I am trying to see if there is a simple policy that don’t depend on features as such but rather copies the path in memory to determine next step, some of it can also be solved using FormActions

17

I set max_history within the policy but i found it only within memoization

But this one says max_history is None .

it has no max_history attribute

“Try a simpler classifier like Decision tree or Sklearn SVM to adapt to unforeseen paths while still using a deterministic path for a particular process.” By saying this are you suggesting of using sklearn_policy ? as i guess i can select my own model with it

18

You can try the skearn policy as well - it is using logistic regression

also when i say defaults of max history, it is the featurizer passed to the Keras policy by the default train script - i think you should start from what is given by Rasa as defaults before making your own. the default train is more generalized to common examples and is a good starting point. You should check the featurizers given

19

Ohk i shall then set max_history of single state featurizer to 0 and pass it to keras/memoization . Lets see if it works (fingers crossed)

20

For keras,i would use max history of 3 to start with and for memoization i would choose 0

Keras should be used as a fallback when memoization can’t find an action.