Loops in stories evaluation and visualisation

Rasa Open Source
1

I tried to notify issues in the rasa_core repository, but for some reason I can only raise issues in the rasa_nlu repo. Anyway.

I am experiencing two big problems with rasa_core when I add a loop in my stories using a checkpoint. In other words, my user is able to ask questions, and when the question is properly answer, the bot goes back to uttering “anything else I can do for you?”, and I write this in the stories using a checkpoint for that stage of the conversation, and anytime a question’s story is over, it ends by going back to that checkpoint. This creates loops.

The evaluation script lasts for about 6 hours (but I never let it ran through because it’s faster to just read my stories, although counter-productive), and the visualization script never ends. I would like to be able to deal with loops since they are extremely frequent when designing automated conversations.

Anyone else experiencing something similar? Any suggestions of solutions, or is anything on the roadmap?

Cheers everyone!

1 Like
2

how many loops do you have? By abusing checkpoints you create a huge amount of stories that requires a lot of time to process. Did you consider using forms for your loops?

3

It creates as many loops as the number of questions that I have in my FAQ chatbot. I am not really “abusing check points” in the sense that I have a gazillion checkpoints, I really just have one per question-topic and if my chatbot can tackle 10 questions then there’s 1 checkpoint but I use the same checkpoint in each story corresponding to each question. Something like

## story_greet
- greeting
   * how_can_i_help
> faq

## story_q1
> faq
(story here q1)
  * utter_anything_else
> faq

## story_q2
> faq
(story here q2)
  * utter_anything_else
> faq
...

This creates a great bot that I cannot evaluate because of how the evaluation process works : it goes through all the possible loops and evaluates everything. This cannot work with loops; the number of possible paths grows way too fast (and if it really is a for loop through all the possible loops, the combinations are infinite because of the cycle). Ideally if we know the number of steps in the past that the model can look at, we can put a bound over which we stop looping and then this process will be considerably shorter, but that’s not how the algorithm is working now.

4

Creating loops with checkpoints is exactly what I meant by abusing checkpoints, it doesn’t matter how many checkpoints you use.

The algorithm doesn’t go infinitely over loops, we create linear graph approximation from the looped graph.

For the example story above I suggest to use forms instead

5

But I am already using forms within that check points, since some questions are answered by a simple answer or a custom action, and some other require a form. If I would replace that checkpoint by a form, basically my whole bot would be a form, that would be way too complex.

But the point is that the bot is working fine when I test manually (and I tested extensively). I just cannot evaluate it because the algorithm takes forever to do so. I don’t wanna make a different bot just because evaluating it is painful… that’s the best bot I could come up with, and it’s a great one! and it is really easy to modify also. Just very painful to test. I don’t really know how to deal with this right now… will keep thinking about it

Could you link something to the theory behind the technique of linear graph approximation? I understand “linear approximation” and “graph” (graph of a function or graph as in graph theory, that I am not sure which of the two you meant) but the only linear graph approximation that I know of would be the standard first-order linear approximation via Taylor expansion. I’m not quite sure that’s what you mean since it doesn’t seem to fit in the context. I’d like to understand the algorithm used there, maybe try to tweak it!

6

I don’t have a link, but it is done here: rasa/structures.py at c98fa44209d8987b2f44b6b522bb14084c271db4 · RasaHQ/rasa · GitHub

1 Like
7

I will have a look, thanks @Ghostvv !

8

Hey, @PatrickDS

I have structured my bot the same way as you (using checkpoints and forms) and am experiencing the issue that you describe. Have you found a workaround already?

9

Hey krasi0,

Not really. In the end at our company we dropped the project, which is a sad thing since I found it really interesting. Ideally we should have a look into it and make a PR to Rasa. If you wanna give it a try, we could try it together; just get in touch with me!

10

@PatrickDS I’d be up for the challenge! We have a bot that is mostly form-based but that must support some FAQs at any point in the conversation. I was able to work without loops so far but that makes it much more painful to maintain and I feel less complete. I will definitely try your approach and if time allow, I’d be up for working on a PR.

11

@nbeuchat I believe that I managed to implement something similar, it’s just that the current evaluation algorithm for the core model is not fit for evaluating those kinds of conversations. For loop-less conversational graphs, a linear graph approximation works fine, but for graphs with loops this is not working, because they seem to run indefinitely. At this point it’s a matter of choice: either you implement your loops and your custom evaluation script by inspiring yourself from the current evaluation script, or you just dump loops altogether.

To be honest, I am surprised that Rasa isn’t considering this a big problem for its users since it was an enormous problem to me. But that doesn’t show that Rasa doesn’t know what it’s doing, just that their users mostly don’t struggle with that and they focus their priorities, which is fine. What this means for you though, is that you have to take a decision.

I think that your individual problem can be solved by just writing a script that generates stories by going through those loops once or twice and you can train your model on top of that. This will most probably teach your model to keep looping infinitely if required. If your conversational graph doesn’t have too many loop points (say one point, the point where FAQs have to be answered), this should work. And in this way, since your stories become free of checkpoint loops, the evaluation algorithm should do fine. Basically, what I picture is going through your stories, using regexes to figure out where the checkpoints are, removing the checkpoints and copy-pasting (via a script of course) the FAQ parts in a new set of generated stories. It will take some testing and won’t be exactly trivial but it’s a simple enough idea.

2 Likes
12

agree Rasa should provide an option for this, typical use case

13

we have the RulePolicy and Forms that allow you to define loop behaviors

14

Are there any news on this topic?

I find it a shame that the use of loops is restricted to rules and forms which make the process of creating them really difficult if you want to do more than just extracting a few slots. Checkpoints are much easier to implement, easier to read inside stories, more flexible and in general more intuitive. I had several occasions where I wanted to expand my bots stories which would be a question of minutes with checkpoints but became a job of hours without them.

Is there any hope that they will be supported in the future? (maybe a option a la --max-loops to tell the bot to automatically stop after a maximum number of looping trought the same checkpoint?

If not, can maybe someone share a more advanced use of forms and rules to deal with this? All examples I could found were either just simple slot extractions or a complex mess of custom actions. thank you :slight_smile: