Black Friday Reward - Upto 75% off - Ends in 0d 00h 00m 00s - Coupon code: CFsave75

Databricks Certified Context Engineer Associate Context-Engineer-Associate Question # 1 Topic 1 Discussion

Databricks Certified Context Engineer Associate Context-Engineer-Associate Question # 1 Topic 1 Discussion

Context-Engineer-Associate Exam Topic 1 Question 1 Discussion:
Question #: 1
Topic #: 1

A context engineer is evaluating a Databricks customer support agent that handles complex multi-turn conversations involving subscription changes, billing adjustments, and service cancellation requests.

The agent uses Lakebase-backed session storage to maintain conversation history. MLflow 3 tracing is enabled, and each inference trace includes a session identifier.

During testing, the engineer observes the following behavior:

Turn 1: The customer requests cancellation of an existing subscription.

Turn 2: The agent correctly identifies the subscription and explains the cancellation procedure.

Turn 4: The customer specifies that cancellation must occur at the end of the current billing cycle.

Turn 6: The agent retrieves the relevant cancellation policy and correctly answers a billing question.

Turn 8: The agent generates a final confirmation stating that the subscription will be canceled immediately, overlooking the customer ' s earlier instruction to cancel at the end of the billing cycle.

Individual response evaluations show high correctness and relevance scores. However, the operations team suspects that the agent is failing to preserve important information across conversational turns.

The engineer needs an evaluation mechanism that detects failures involving information introduced earlier in a conversation, rather than scoring each response independently.

Which MLflow evaluation approach should the context engineer implement?


A.

Run the built-in Correctness scorer independently on each response trace, increasing the number of test questions to improve statistical confidence in the results.


B.

Evaluate only the final response against the latest retrieved policy document, since the agent ' s correctness should depend primarily on the most recent authoritative source.


C.

Increase the number of MLflow traces collected per conversation and compare average token usage, tool-call count, and latency between successful and unsuccessful sessions.


D.

Group traces by their conversation session identifiers and run MLflow ' s multi-turn KnowledgeRetention judge to evaluate whether information introduced in earlier turns is correctly maintained throughout the conversation.


Get Premium Context-Engineer-Associate Questions

Contribute your Thoughts:


Chosen Answer:
This is a voting comment (?). It is better to Upvote an existing comment if you don't have anything to add.