B5. Target Architecture for AI
A target picture for the AI architecture creates a shared understanding of how data, models, platforms, and processes work together. Without a reference picture, silo solutions, duplicated effort, and security gaps emerge. The architecture shows the necessary building blocks, their connections, and the governing principles, and sets security and data-protection guardrails. It is the basis for technical decisions, budget allocation, and prioritization.
What this chapter delivers: the answer to the question of which technical building blocks a company needs so that AI use cases do not emerge as silo solutions, and how those building blocks relate to each other. The reader (typically a CTO or architect) uses the reference building blocks as a target list: for each block, the architecture inventory records whether it exists in your company, is being built, or is deliberately not needed. The result is your own target picture including a gap list for the roadmap (chapter C2). Per use case, the architecture decision checklist makes the choices concrete. The chapter deliberately prescribes no products: it describes building blocks and principles; product and vendor selection follows in chapter B3.
The figure below shows the reference building blocks at a glance. It answers the question: which layers make up an AI-ready architecture? The six layers are arranged along the path of data, from the top one (raw data arrives) to the bottom one (results reach the business processes).
The orange arrows show this data path, and it is not the same for every use case. Predictive use cases forecast a value or a class, such as fraud detection (suspicious or not) or demand forecasting (how many units next month). They run through the feature layer because they need prepared features, for example the average revenue of the last 30 days. Generative use cases, by contrast, produce new content, such as an assistant that answers questions from your own documents. They skip the feature layer and access the data directly via the RAG pipeline.
The dashed arrow runs the other way, from the bottom back up. It stands for drift: over time, the data in live operations moves away from the data the model was trained on, and results quietly get worse. It can only be noticed in operations, but it can only be fixed at the data, which is why the arrow leads back there.
The security layer sits crosswise because it applies to all layers at once. Each layer states below what it receives from the previous one and what it delivers to the next; the sections explain each building block in detail.
Reference building blocks
Each layer opens with its goal and a Receives/Delivers line that shows where its inputs come from and where its results go. The table per layer names, for each building block, what concretely to put in place and what value it adds.
Data layer
The foundation of the architecture, since every model relies on trustworthy data. Goal of the layer: provide data from the source systems checked, classified, and findable, so that every project draws on the same inventory instead of creating its own copies. The underlying data strategy is developed in chapter B4; here are the technical building blocks that implement it.
Receives raw data from the source systems (ERP, CRM, sensors, documents). Delivers checked and classified data to the feature layer (predictive path) and directly to the RAG pipeline of the generative layer (generative path).
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Data-source catalog | Build a central directory that documents each source from two angles: the technical one (format, interface, currency) and the business one (owner, purpose, usage permission). | Every project sees at once which data exists and whom to ask. Without a catalog, every effort begins with weeks of searching. |
| Data ingestion | Set up a controlled ingestion path that, on every import, automatically checks (quality), classifies (sensitivity), and logs (provenance). | Errors and unchecked copies are caught at the entrance. Without it, bad data flows unchecked into every model. |
| Data platform | Provide a shared storage and processing home for structured and unstructured data, separated by domain but analyzable across domains. | Data sits in one place instead of scattered copies. Without it, every application creates its own, soon-diverging stores. |
| Governance | Enforce the rules from the data strategy technically: named data owners, approval processes, least-privilege access, GDPR-compliant logging. | Access is governed and auditable. Without enforcement, the governance rules stay ineffective. |
Feature layer
Prepares raw data into reusable features, e.g., “age” from date of birth or “average temperature of the last hour” from sensor data. Goal of the layer: make features computed once available across projects instead of rebuilding them in every project.
Receives checked data from the data layer. Delivers reusable features to the development layer. This layer belongs only to the predictive path; generative use cases do not use it and access the data layer directly via the RAG pipeline.
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Feature store | Build a central directory of reusable features with versioning, owners, and quality tests. | Features computed once are reused across projects. Without it, every project rebuilds the same feature slightly differently. |
| Guidelines | Set binding rules for naming, documentation, and reuse. | The feature inventory stays searchable. Without rules, it sprawls into an unmanageable stock. |
Development layer
Design, training, testing, and documentation of models, with traceability as a core requirement. Goal of the layer: build and package models so their creation stays reproducible and auditable.
Receives features from the feature layer (predictive) as well as data and generative building blocks for generative models. Delivers packaged, documented models to the operations layer and vetted generative building blocks to the generative layer.
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Reproducible, isolated environments | Run every experiment and every training run in a defined environment (containers, virtual environments). | Results are repeatable and experiments do not interfere. Without them, a result cannot be reproduced later. |
| Packaging of artifacts | Package finished models with everything that belongs to them: model and data cards and training logs. | Audits can trace what a model was built with and how. Without it, a model’s provenance cannot be evidenced. |
| Generative building blocks | Provide vetted prompt templates, retrieval components, and evaluation suites. | Generative applications are assembled from proven parts. Without them, every team rebuilds its prompts. |
Generative and agentic layer
A dedicated layer for LLMs, RAG, and agents. It is deliberately kept separate because it has its own building blocks, risks, and costs. Goal of the layer: control access to language models and steer multi-step agents safely.
Receives documents directly from the data layer (for the RAG pipeline) as well as models and prompt building blocks from the development layer. Delivers controlled model access and vetted agent flows to the operations layer.
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Model gateway | Route all model access through one controlled point that logs, enforces quotas, and allocates costs to the consumer, whether a proprietary model via API or an open model self-hosted. | Model access stays steerable and billable. Without a gateway, distributed API keys sprawl beyond anyone’s control. |
| RAG pipeline | Build the chain that makes internal documents usable: prepare, convert into embeddings, index, and at runtime return matching passages with source attribution (terms in chapter A2). | Models answer from your own, sourced documents. Without it, only the model’s general knowledge remains, without a source. |
| Prompt and context management | Treat prompts like code: version, test, approve. | Prompts are traceable and testable. Without it, they lie scattered across applications and notes. |
| Agent orchestration | Define a tool registry (which tools an agent may call), action chains, and intervention points with human confirmation (human-in-the-loop). For connecting tools and context, the Model Context Protocol (MCP) is emerging as an open standard; it decouples agents from individual vendors but does not replace access control (see chapter B9). | Multi-step agents act under control. Without it, an agent calls tools uncontrolled. |
| Protection layer | Add input and output filters against prompt injection and data exfiltration; why filters alone are not enough is explained in chapter B9. | Attacks and exfiltration via prompts are caught. Without it, the generative layer stands open. |
Operations layer
Stable, scalable, and reliable operation of production models. Goal of the layer: provide models as reliable services and monitor their state continuously.
Receives finished models from the development layer and services from the generative layer. Delivers stably operated services to the integration layer and reports detected drift back to the data layer, so data and models can be corrected.
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Delivery as a service | Offer models via clear interfaces, synchronously for interactive applications and asynchronously for bulk processing. | Models are reusable instead of hard-wired. Without it, every model is stuck in exactly one application. |
| Queues and job scheduling | Queue compute-intensive jobs such as training or bulk evaluation and process them in a controlled way. | Interactive use stays fast. Without it, large jobs crowd out running operation. |
| Observability | Collect metrics, logs, and traces and set up drift detection (metrics in chapter D1). | Failures and declining quality surface early. Without it, decay only shows in complaints. |
Integration layer
Embeds AI into existing business processes and systems instead of creating silo solutions. Goal of the layer: let AI results arrive where the work happens. Here the data path ends.
Receives stably operated services from the operations layer. Delivers AI results into the business processes and to every user channel.
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Connection to core systems | Connect AI services with ERP, CRM, DWH, and collaboration tools. | Results appear inside the work process. Without it, they land in a separate interface nobody opens. |
| Event-driven processing | Couple systems via defined contracts (APIs with committed service levels). | Changes on one side do not break the other. Without it, every system change tears the chain. |
| Access from products, portals, and mobile applications | Provide the same AI services to every channel instead of rebuilding per channel. | Every channel uses the same service. Without this block, duplicated effort arises per channel. |
Security layer
A cross-cutting concern across all layers, from data ingestion to integration, covered in depth in chapter B9. Goal of the layer: secure every other layer instead of adding security afterward.
Applies to all layers at once; that is why it sits crosswise in the figure and not as a separate step in the data path.
| Building block | What to put in place | Value, and what is missing without it |
|---|---|---|
| Management of keys, secrets, and certificates | Manage credentials centrally and rotate them regularly. | Compromised credentials can be replaced quickly and completely. Without it, a leaked key stays valid unnoticed. |
| Protection against data exfiltration | Add input filters and content controls, especially before prompts to external models. | Confidential data does not leave the company unnoticed. Without it, it flows out via prompts. |
| Separation of development, test, and production environments | Use no real customer data in development; separate test runs from production data. | Tests cannot accidentally damage production data. Without separation, every test endangers real data. |
Template: architecture inventory
The inventory brings all reference building blocks together in one list and turns it into your own target picture. How to use it: classify each block once as present, in progress, or deliberately not needed; for a gap, note the next step and a priority. Example: if the model gateway is missing but several generative use cases are planned, the entry reads “in progress” with priority high. The result is a gap list that feeds directly into the roadmap.
| Layer | Building block | Status (present / in progress / not needed) | Gap or next step | Priority |
|---|---|---|---|---|
| Data layer | Data-source catalog | |||
| Data layer | Data ingestion | |||
| Data layer | Data platform | |||
| Data layer | Governance | |||
| Feature layer | Feature store | |||
| Feature layer | Guidelines | |||
| Development layer | Reproducible, isolated environments | |||
| Development layer | Packaging of artifacts | |||
| Development layer | Generative building blocks | |||
| Generative and agentic layer | Model gateway | |||
| Generative and agentic layer | RAG pipeline | |||
| Generative and agentic layer | Prompt and context management | |||
| Generative and agentic layer | Agent orchestration | |||
| Generative and agentic layer | Protection layer | |||
| Operations layer | Delivery as a service | |||
| Operations layer | Queues and job scheduling | |||
| Operations layer | Observability | |||
| Integration layer | Connection to core systems | |||
| Integration layer | Event-driven processing | |||
| Integration layer | Access from products, portals, and mobile applications | |||
| Security layer | Management of keys, secrets, and certificates | |||
| Security layer | Protection against data exfiltration | |||
| Security layer | Separation of development, test, and production environments |
Deployment model: cloud, on-premises, or hybrid
Across all layers sits the question of where the building blocks run. It is decided per workload, not wholesale for the entire company, and depends on data sensitivity, regulation, costs, and existing operational competence.
| Model | Strengths | Limits | Typical use |
|---|---|---|---|
| Public cloud | Fast access to current models and GPU capacity, no hardware investment, elastic scaling | Running costs grow with usage; data residency and transfer questions (GDPR, sector supervision); vendor dependency | The standard case for piloting and most GenAI workloads with non-critical to confidential data |
| On-premises / private infrastructure | Full data sovereignty, plannable costs at high constant load, no external transfers | High investment (GPU hardware), own operational and security effort, slower access to model progress | Strictly confidential data, sovereignty requirements, very high constant volumes |
| Hybrid | Each workload runs where it belongs: sensitive inference internally, scaling load in the cloud | Two operational worlds; consistent security and governance needed across both | Regulated companies with a mixed portfolio; the most common target state |
Three practical rules:
- Decide per workload, not per creed: a company can source the internal assistant from the cloud and run the pricing model on its own infrastructure. The model gateway (above) keeps this mix steerable because all access runs through one controlled point.
- Think about the exit: connect cloud services so that a model or vendor change does not reinvent the application (open interfaces, swappable model connection, own prompts and evaluation data versioned in-house).
- Involve regulation early: for finance and insurance companies, outsourcing and third-party requirements apply (see chapter B8); the data location including sub-processors belongs in the vendor assessment.
Template: architecture decision checklist
For every use case before building. Each item prevents a typical, expensive rework later:
- Predictive or generative/agentic? Determines the layers used; a wrong classification leads to unsuitable metrics and missing protection measures.
- Knowledge path clarified: prompting, RAG, or fine-tuning? (see chapter A2) Prevents the most expensive misunderstanding: training where prompting would have sufficed.
- Model sourcing decided: proprietary via API or open, self-hosted? Determines cost model, data protection, and dependency.
- Data sovereignty and data protection clarified; no unwanted data exfiltration.
- Reusable building blocks used (feature store, prompt templates) instead of building new.
- Integration into core systems planned from the start, not as an afterthought.
- Observability planned from the start, including drift.
- Protection layer against prompt injection and data exfiltration planned.
- Cost allocation per use case possible (see chapter D2).
Architectural principles (guardrails)
Across individual use cases, a handful of principles keep the architecture consistent. They are the guardrails for every build-or-buy decision:
- Reuse before new build: shared platform building blocks reduce duplicated effort.
- Traceability by design: model and data cards as well as logs are mandatory, not optional.
- Data sovereignty first: differentiating use cases are not exposed via third-party models.
- Security as a cross-cutting concern: not a downstream layer, but designed into every layer.
Checklist: target architecture in place
While the architecture decision checklist above is kept per use case, this list checks the company’s target picture once:
- Architecture inventory completed across all reference building blocks, each classified as present, in progress, or deliberately not needed
- Gap list prioritized and carried into the roadmap (see chapter C2)
- Deployment model decided per workload, not blanket for the whole company
- Exit considered: open interfaces, swappable model connection, own prompts and evaluation data versioned in-house
- Outsourcing and third-party requirements reviewed, data location including sub-processors clarified
- Architectural principles adopted and made binding for build-or-buy decisions