A context engineer is evaluating a Databricks customer support agent that handles complex multi-turn conversations involving subscription changes, billing adjustments, and service cancellation requests.
The agent uses Lakebase-backed session storage to maintain conversation history. MLflow 3 tracing is enabled, and each inference trace includes a session identifier.
During testing, the engineer observes the following behavior:
Turn 1: The customer requests cancellation of an existing subscription.
Turn 2: The agent correctly identifies the subscription and explains the cancellation procedure.
Turn 4: The customer specifies that cancellation must occur at the end of the current billing cycle.
Turn 6: The agent retrieves the relevant cancellation policy and correctly answers a billing question.
Turn 8: The agent generates a final confirmation stating that the subscription will be canceled immediately, overlooking the customer ' s earlier instruction to cancel at the end of the billing cycle.
Individual response evaluations show high correctness and relevance scores. However, the operations team suspects that the agent is failing to preserve important information across conversational turns.
The engineer needs an evaluation mechanism that detects failures involving information introduced earlier in a conversation, rather than scoring each response independently.
Which MLflow evaluation approach should the context engineer implement?
Submit