The correct recommendation is to retrieve grounding data from knowledge_base and, at inference time, generate query embeddings and run a vector similarity search .

The chatbot currently answers some general HR questions but fails when policies change and when users ask about internal policy documents by title or category . That is exactly the kind of problem RAG is meant to solve: ground the LLM in the organization’s proprietary content instead of relying on the model’s training data or unrelated transactional tables. Microsoft’s RAG guidance states that RAG extends LLMs by grounding responses in your own content and that, for agentic retrieval, knowledge bases unify knowledge sources for retrieval.

So the grounding data should come from knowledge_base , because that table stores the HR policy documents and already includes fields like title, content, category, and embedding. Those are the fields directly tied to the missing and outdated policy answers. By contrast:

employee_profiles and benefits_enrollment are operational HR tables, not the authoritative store for policy-document grounding.

PDF exports of the policies would be inferior to querying the indexed/structured knowledge base already prepared for retrieval.

The LLM training data is specifically the wrong source when the issue is outdated internal content.

For the retrieval step, Microsoft’s guidance says to use embeddings for vector queries and notes that vector similarity search matches concepts, not exact terms . This is especially important because users ask about policy documents by title or category and also phrase questions in ways that might not exactly match document wording. Generating a query embedding and then running a vector similarity search is the appropriate retrieval step in a RAG pipeline.