Hi, I’ve created a pretty simple example trying to set slots for a product search. My problem is that the LLM sets a slot without any user input. When asked the LLM sees its error and corrects but of course in rasa it wont’t get corrected. The problem is (most times) reproducable in Chat-GPT. Model is GPT 4, 3.5 Turbe seems to be not able to fill multiple slots from one response. So here’s a dialog that describes my problem:
generated rasa prompt:
Your task is to analyze the current conversation context and generate a list of actions to start new business processes that we call flows, to extract slots, or respond to small talk and knowledge requests.
These are the flows that can be started, with their description and slots:
collect_product: this flow lets the user search a product
slot: modelname (the modelname of the product)
slot: color (the color of the product)
slot: length (the length of the product)
slot: width (the width of the product)
slot: antislip (product should have antislip coating), allowed values: [True, False]
slot: perleffect (product should have perleffect coating), allowed values: [True, False]
get_delivery_time: this flow gets the delivery time by backend query.
===
Here is what happened previously in the conversation:
USER: find product
AI: what is the modelname of the product?
USER: saniform white 1700x700
AI: should the product be coated with antislip?
USER: no
===
You are currently in the flow "collect_product".
You have just asked the user for the slot "antislip" (product should have antislip coating).
Here are the slots of the currently active flow:
- name: modelname, value: saniform white, type: text, description: the modelname of the product of Kaldewei
- name: color, value: white, type: text, description: the color of the product
- name: length, value: 1700.0, type: float, description: the length of the product
- name: width, value: 700.0, type: float, description: the width of the product
- name: antislip, value: undefined, type: bool, description: product should have antislip coating, allowed values: [True, False]
- name: perleffect, value: undefined, type: bool, description: product should have perleffect coating, allowed values: [True, False]
If you start a flow, first start the flow and then optionally fill that flow's slots with information the user provided in their message.
The user just said """no""".
===
Based on this information generate a list of actions you want to take. Your job is to start flows and to fill slots where appropriate. Any logic of what happens afterwards is handled by the flow engine. These are your available actions:
* Slot setting, described by "SetSlot(slot_name, slot_value)". An example would be "SetSlot(recipient, Freddy)"
* Starting another flow, described by "StartFlow(flow_name)". An example would be "StartFlow(transfer_money)"
* Cancelling the current flow, described by "CancelFlow()"
* Clarifying which flow should be started. An example would be Clarify(list_contacts, add_contact, remove_contact) if the user just wrote "contacts" and there are multiple potential candidates. It also works with a single flow name to confirm you understood correctly, as in Clarify(transfer_money).
* Intercepting and handle user messages with the intent to bypass the current step in the flow, described by "SkipQuestion()". Examples of user skip phrases are: "Go to the next question", "Ask me something else".
* Responding to knowledge-oriented user messages, described by "SearchAndReply()"
* Responding to a casual, non-task-oriented user message, described by "ChitChat()".
* Handing off to a human, in case the user seems frustrated or explicitly asks to speak to one, described by "HumanHandoff()".
===
Write out the actions you want to take, one per line, in the order they should take place.
Do not fill slots with abstract values or placeholders.
Only use information provided by the user.
Only start a flow if it's completely clear what the user wants. Imagine you were a person reading this message. If it's not 100% clear, clarify the next step.
Don't be overly confident. Take a conservative approach and clarify before proceeding.
If the user asks for two things which seem contradictory, clarify before starting a flow.
If it's not clear whether the user wants to skip the step or to cancel the flow, cancel the flow.
Strictly adhere to the provided action types listed above.
Focus on the last message and take it one step at a time.
Use the previous conversation steps only to aid understanding.
Your action list:
Model answer:
SetSlot(antislip, False)
SetSlot(perleffect, False)
Asking it why:
why did you set slot perleffect
Answer:
t looks like I made an error in setting the "perleffect" slot without your instruction. I should have only set the "antislip" slot to False based on your response and then asked you about the "perleffect" coating. Let's correct that:
SetSlot(perleffect, undefined)
Should the product have a perleffect coating?
Asking why:
yes, you first made an error. Is there a reason in the prompt? Could I have better prompted?
Answer:
Your prompt was clear and well-structured for the task you wanted to achieve. The mistake was on my part; I misinterpreted your last message and preemptively set a slot value for "perleffect" without having a directive from you to do so. Your direction in the prompt was appropriate, focusing on sequentially filling each slot as required by the user's input in the context of the "collect_product" flow.
In future interactions, to prevent similar mistakes, ensuring each user input is specifically directed towards the request could help. For instance, asking direct questions about each slot sequentially, as you have been doing, is a good practice. This helps maintain clarity and ensures each response from the user directly corresponds to a specific request or question.
