A context engineer is evaluating a Databricks customer support agent that handles complex multi-turn conversations involving subscription changes, billing adjustments, and service cancellation requests.
The agent uses Lakebase-backed session storage to maintain conversation history. MLflow 3 tracing is enabled, and each inference trace includes a session identifier.
During testing, the engineer observes the following behavior:
Turn 1: The customer requests cancellation of an existing subscription.
Turn 2: The agent correctly identifies the subscription and explains the cancellation procedure.
Turn 4: The customer specifies that cancellation must occur at the end of the current billing cycle.
Turn 6: The agent retrieves the relevant cancellation policy and correctly answers a billing question.
Turn 8: The agent generates a final confirmation stating that the subscription will be canceled immediately, overlooking the customer ' s earlier instruction to cancel at the end of the billing cycle.
Individual response evaluations show high correctness and relevance scores. However, the operations team suspects that the agent is failing to preserve important information across conversational turns.
The engineer needs an evaluation mechanism that detects failures involving information introduced earlier in a conversation, rather than scoring each response independently.
Which MLflow evaluation approach should the context engineer implement?
A context engineer is investigating an AI agent deployed on Databricks that handles customer support operations. The agent has two MCP tools registered: one described as “Searches customer support tickets and open case records for a given customer”, and another described as “Looks up customer issue history and unresolved case details by customer ID”.
When users ask whether a customer has any open issues, the agent randomly chooses between the two tools across repeated identical requests. When users ask about a customer’s past resolved issues, the agent similarly misfires between the two tools. The engineer has reviewed the tool call logs and confirmed that the agent cannot reliably distinguish which tool to invoke.
Which overlap between the two tool descriptions is causing the agent to select tools ambiguously?
A context engineer is troubleshooting a Databricks agent responsible for a multi-month clinical trial data processing pipeline. The agent uses a fixed sequential chaining strategy, running each processing stage in a predetermined order regardless of which stages have already completed successfully. When upstream data sources are delayed or partially available, the agent restarts the entire chain from the beginning, discarding all previously completed stage outputs. Logs show that stages with no dependency on the delayed source are being blocked unnecessarily, and total pipeline runtime has grown to an unacceptable length because of repeated full restarts on partial failures.
Which two changes should the context engineer make to resolve the observed failure pattern?
Choose two
A context engineer is debugging a multi-agent pipeline on Databricks in which a routing agent, a retrieval agent, and a summarization agent work together to answer complex insurance claims queries. Each agent maintains its own session state, and the agents communicate through a shared message bus. All agents retrieve documents from the same current, up-to-date Databricks AI Search index. The routing agent classifies a claim as high-priority, but the final output contains a low-priority summary that contradicts the routing agent’s classification. Claims adjusters report contradictory recommendations in final outputs, and pipeline reliability scores have dropped significantly over the past week.
Which underlying failure mode is most likely causing the conflicting decisions between agents?
A context engineer is building a customer support agent on Databricks. The agent’s session-initialization routine must populate a context object before turn 1 begins; the context object is then passed to every turn handler. The agent needs the user’s plan tier, region, and a few entitlement flags from the Lakebase customer_profile table in that context object. The values must be present before the first turn handler executes and may be referenced on any subsequent turn. The values do not change during a session, but could change across different sessions. The team wants to minimize per-turn Lakebase round-trips and the latency they add to turn responses.
Which retrieval approach should the engineer implement for this agent architecture?
A context engineer is configuring a Databricks agent for an automotive service center platform. The agent must serve service center staff who submit a vehicle identification number (VIN) and a reported symptom. The agent is expected to return a summary of the vehicle’s operating profile, identify the applicable SOP procedure code for the symptom, and retrieve the narrative step-by-step service instructions for that procedure. The engineer has access to these data sources:
Unity Catalog Delta table named vehicle_profiles with columns vin, engine_type, and operating_spec.
Unity Catalog Delta table named symptom_sop_map with columns symptom_code and sop_procedure_code.
Unity Catalog Delta table named service_history with columns vin, service_date, and technician_id.
Databricks AI Search index built from scanned OEM service manuals containing narrative SOP instructions and safety warnings.
Which combination of context elements correctly supports the agent’s task?
A context engineer is tuning a compaction prompt for a Databricks agent that summarizes customer support ticket histories before sending them to a downstream LLM. The trace being compacted spans 40 turns, and the compacted output must fit within the downstream LLM’s context window. During initial evaluation against a benchmark of escalated support cases, the engineer observes that the compaction output consistently omits several prior resolutions and key escalation events, even though it fits well within the downstream LLM’s context window.
What tuning step should the engineer take next?
A context engineer is developing a Databricks agent for a retail inventory management system. The system prompt and tool descriptions consume 1,800 tokens of a 4,096-token context window, and the engineer reserves 1,000 tokens for the user turn and model response. The remaining tokens are used for few-shot examples, each costing approximately 250 tokens, so the engineer includes exactly five examples.
The agent uses three tools: inventory_lookup, reorder_check, and supplier_lead_time.
Six candidate examples are available:
Example 1: A warehouse manager asks whether a depleted SKU needs reordering; the agent calls inventory_lookup, confirms depletion, calls reorder_check, and returns a binary reorder decision.
Example 2: A procurement officer asks when a low-stock item can be replenished; the agent calls reorder_check, calls supplier_lead_time, and returns a replenishment recommendation.
Example 3: A store associate asks for the current stock count of a single product; the agent calls inventory_lookup and returns a scalar quantity.
Example 4: A store associate asks for the current stock count of a single product; the agent calls inventory_lookup and returns a structured status object containing the quantity and a restocking flag.
Example 5: A store associate asks whether a forecasted shortfall requires immediate action; the agent calls inventory_lookup, compares against minimums, calls reorder_check, and returns a binary reorder decision.
Example 6: A logistics coordinator asks how long a specific supplier takes to fulfill an order; the agent calls supplier_lead_time and returns a scalar lead-time value.
Which example should the engineer omit to preserve the token budget while maximizing behavioral coverage?
A context engineer is developing an agent to support legal teams reviewing multi-part pharmaceutical trial protocols. Each protocol document includes a narrative background section, nested dosage tables with footnote qualifiers, cross-referenced section identifiers, and figure captions describing study design diagrams. The agent must answer precise questions such as which protocol section governs a specific dosage threshold and what footnote conditions apply to that threshold.
Available data sources include:
A PDF binary store in Unity Catalog Volumes.
A Delta table of plain-text chunks extracted from the protocols, each tagged with page number and document identifier, indexed in a Databricks AI Search index.
A Delta table of protocol structure metadata at the page level, containing table summaries, figure captions, and section hierarchy, accessible through Unity Catalog functions.
Which context elements should the context engineer include to answer structural and footnote-level questions accurately?
A context engineer is building a retail agent with Agent Bricks Custom Agents. The agent has two registered tools: a SQL query tool that queries a live inventory database, and a vector search tool that searches a static product catalog vector index for product descriptions, specifications, and categories. Both tools are configured correctly, function properly, and return valid results when called.
At runtime, when users ask questions such as “Is this item in stock?”, the agent consistently calls the product catalog tool and provides a plausible availability answer without checking the live inventory database. The product catalog tool is not intended to answer live stock availability questions.
Which context engineering issue explains this behavior?