Experimental Feature: Markers
One of the questions that we receive from our community is “how do we evaluate the performance of an assistant beyond individual model comparison and validation?”. To help answer this question, we are releasing an experimental feature in Rasa Open Source 3.0 called Markers. Simply put, markers are conditions that allow you to describe and mark points of interest in the dialogue, such as whether certain actions have been executed, intents have been detected, or slots have been set. When a condition is met, the relevant events are marked for further analysis or inspection.
Here is an example of a marker condition:
marker_cheer_up_succeeded:
seq:
- intent: mood_unhappy
- action: utter_cheer_up
- action: utter_did_that_help
- intent: affirm
Here are the two main ways of using markers when developing your assistant in a larger evaluation workflow:
- You can use them to define and track your assistant’s Key Performance Indicators (KPIs), such as dialogue completion or task success.
- You can use them to diagnose and improve your assistant by surfacing important events for inspection. For example, you might observe that your assistant handles one task really well, but fails when performing a different one. You could define a marker to quantify how often the failed behavior occurs and surface relevant dialogues for review as part of Conversation Driven Development (CDD).
Shipping Markers in 3.0 is an important first step in evaluating assistants.
Read more about Markers here. If you have feedback about this feature, please share it with us on this thread!
