Securing Enterprise Data for LLM-Powered Applications: A Reference Architecture for Inference-Time Data Protection

Abstract

Enterprises are rapidly adopting large language models (LLMs) by connecting them to internal data sources through approaches like retrieval-augmented generation (RAG), natural language database querying, and API based tool calling. In most enterprise deployments, the LLM itself is not hosted with in the company’s cloud tenant, but hosted by a third-party provider, which means the company data including customer records, financial information and proprietary documents must leave the organization’s, controlled environment every time a query is processed. This applies to the adoption of Saas products as well that leverage GenAI capabilities, where often times the data leaves the organization, the Saas platforms and are sent to the LLM vendors. Contractual agreements with the LLM providers help, but they cannot fully address risks like prompt injection, unintended data exposure, or regulatory non-compliance. This paper presents a practical reference architecture for securing enterprise data at inference time, organized around three main layers. The first layer addresses the data sources including unstructured content accessed through RAG, structured databases queried through SQL or graph languages, semi-structured records from SaaS APIs, and real-time event streams. The second layer covers the access mechanisms that connect these sources to LLMs, including function and tool calling, the Model Context Protocol, and code execution, along with agentic orchestration that combines multiple mechanisms autonomously. The third layer provides the governance controls such as data classification, role-based and context-aware access control, monitoring, audit trails, and regulatory compliance. The architecture draws on established industry standards including the NIST AI Risk Management Framework and the OWASP Top 10 for LLM Applications. This paper consolidates proven industry security practices into a coherent, practical, layered model that any organization can adapt when deploying LLM-powered applications against their enterprise data.