The current documentation for Checkpoints does not define what checkpoints actually are. It mentions that you can “use them to modularize and simplify your training data” (sounds promising…), but also that " “[they] should be used with caution, if at all”… hmmm… OK, so what are they?
I’ll read further…:
Checkpoints can be useful, but do not overuse them. Using lots of checkpoints can quickly make your example stories hard to understand, and will slow down training.
Here is an example of stories that contain checkpoints:
Then it goes on with an example that does indeed use checkpoints. Looking at the example,
- checkpoint: check_asked_question
Reading the example, I can infer that maybe a checkpoint is a way to label a sequence of steps so they can be reused in multiple stories. But I would have to play around and test to figure out, and so would anyone else who was trying to learn about them. Questions:
How do I create a checkpoint?
How do I reference that checkpoint? (guessing that’s just using a step “checkpoint: abc”)
Are they officially deprecated in favor or Rules and Response Selectors?
If someone can tell me enough to answer these questions, I’m happy to update the documentation, to make the Rasa development experience a bit better for the next person.
@TomV My thoughts on that is not to use much of the checkpoints, as it basic nature is to connect two stories together. It can be either the first or the last step in a story. Further, using a lots of checkpoints can quickly make your stories hard to understand. It makes sense to use them if a sequence of steps is repeated often in different stories, on the otherhand stories without checkpoints are easier to read and write. Even, using lots of checkpoints can quickly make your example stories hard to understand, and even slow down your training.
I’d suggest not to use checkpoints, but If you want to strict your stories, you can go ahead. But, user can break the flow by giving an intent that is not expected. How you will deal that scenario? If the user says “Heya” two times then the flow breaks and the bot gets unpredictable and confused.
I hope you have seen this doc for creating checkpoints : Stories
Linking two stories: second story run after first story
@TomV
I agree checkpoints are still this nebulous nugget that isn’t well defined in the docs. I learn more by looking at working code than by reading a lot of documentation. Now that I think about it, I haven’t noticed the use of checkpoints in any of Rasa’s examples.
I utilize checkpoints like the old GOTO statements in Basic. If I build a new story that utilizes the same steps I’ll break that out into a checkpoint I don’t feel their hard to follow, no harder than reading python so long as you name them to make sense.
I’ll use them if I story “drains” out to a set of steps like creating a ticket, or the agent hand-off, little scenarios that could happen a lot in a conversation. Another good example for a support bot would be to ask if this was helpful. Adding those steps at the end of every story would be a pain, especially if you decided to change how that interaction worked.
To my knowledge there is no return back from a checkpoint so you can’t have one in the middle of a story. BUT you could have a story branch to call say two different checkpoints and the end of those 2 checkpoints call a new common checkpoint.
I’m not entirely sure the technical reason it would slow down training (other than that’s what the docs say) unless the trainer has to spin through and re-assemble them in order to train on them or somehow otherwise keep track of where they are in memory.
It would be a fun experiment to build 3 or 4 stories that all have a set of the same steps, then break out checkpoints to see the difference in training time and or actual accuracy when chatting.
I’m up for a collaboration of some sort if you are
@nik202 Thanks for your reply and the link to the docs. Unfortunately, it was those exact docs that are missing a clear definition of checkpoint, and more specifically how to create one.
I’m currently guessing (since there is no example or documentation on it) that to create a checkpoint is identical to creating a story (e.g., replace “story” with “checkpoint”, and you now have set of steps you can “include” in other stories.
My point is anyone reading the docs should not have to guess, it should be there in the docs.
@nik202 – I especially appreciate your example of why a checkpoint, because it is not flexible would break easily when given an unexpected intent (which seems to be the most likely intent )
@jonathanpwheat – Thanks so much for an excellent clarification. Now that I do get the basic concept, confirming what I thought they were, how do you create a checkpoint?
Is it as simple as replacing “story” with “checkpoint”? I’m guessing that’s the case.
@jonathanpwheat do you have a few examples where you used checkpoints that seem to be “good” uses of this tool.
(Good or bad, I’ll update the docs so at least someone will know how to create a checkpoint!)
Yes, you can pretty much turn any story (exact same format) into a checkpoint just by adding a - checkpoint: some_name as the first entry under steps: Mind you, you have to reference that checkpoint from somewhere or else I think you get an error, or at least a warning when training or validating.
Here’s an example from my general incident stories. These don’t all connect to each other in a straight line here (hopefully that’s not confusing), but they show how to structure calls to and from checkpoints.
This story is called when followup information is necessary and links to a checkpoint that calls a form to collect information
version: "2.0"
stories:
- story: General issue incident submission - Get More Information
steps:
- intent: triage_submit_incident_request_more_information
- action: utter_tell_inform_thank_you
- checkpoint: get_general_information
Here is what I call a “story fragment” (my term not Rasa’s) - because it starts with the checkpoint for the above story AND it calls a checkpoint at the end.
- story: General Collect information and create an incident
steps:
- checkpoint: get_general_information
- action: account_activation_followup_questions_form
- slot_was_set:
- specific_application: Outlook
- active_loop: account_activation_followup_questions_form
- active_loop: null
- checkpoint: support_triage_ask_handoff_to_agent
Here are 2 more stories that link up to the ask about a hand-off or ticket
- story: Account Activation lapse 30 to 90 days
steps:
- intent: account_lapse_90
- action: utter_tell_account_lapse_30to90
- action: utter_tell_account_reactivate_cant_resolve
- checkpoint: support_triage_ask_handoff_to_agent
- story: Account Activation lapse greater than 90 days
steps:
- intent: account_lapse_90plus
- action: utter_tell_account_lapse_greater_than_90
- checkpoint: support_triage_ask_handoff_to_agent
These two checkpoints are where I “drain” a story and will integrate handing off to an agent or simply submitting a ticket
- story: Support Triage Ask and Choose Handoff to support agent
steps:
- checkpoint: support_triage_ask_handoff_to_agent
- action: utter_ask_triage_hand_off_to_agent
- intent: triage_handoff_to_agent
- action: utter_tell_triage_handoff_to_agent_connecting
- action: utter_tell_triage_end_of_integration
- story: Support Triage Ask and Choose Incident Report
steps:
- checkpoint: support_triage_handoff_to_agent
- action: utter_ask_triage_hand_off_to_agent
- intent: triage_submit_incident_request
- action: utter_tell_triage_submit_incident_request
- action: utter_tell_triage_end_of_integration
So those examples cover all the different ways to use a checkpoint that I know of anyway.
@jonathanpwheat Thanks so much for the examples and detail. It was certainly not obvious to me that the same line can either reference a defined checkpoint, or create one. I actually thought a ‘checkpoint’ was peer element to story. (with its own steps element, etc.), rather than a way to “tag” a story for reuse. In the docs, I will clarify that the valid step elements are action, intent, or checkpoint. (are there any other valid step elements?)
I’ve got a full plate today, but will try to create pull request for the docs in this week, and will post the pull request link here for your review and improvement. Thanks so much for taking the time to clarify this for me (and others!)
Valid steps would be any step within a valid story, so you’d also potentially have active_loop: to properly activate a form, slot_was_set: for specifying a slot value during training and the newer or: step that allows a story to be activated by multiple intents.
)