Marc Winkelmann

Key-Value Pairs in Data Vault

Modeling Dynamic Key-Value Pairs in Data Vault

Handling highly dynamic, multi-valued key-value source structures presents a classic architectural challenge for enterprise modern data warehouses. Source platforms frequently capture customer attributes, preferences, telemetry, or metadata as sparse key-value pairs rather than structured, flat tabular records. For a single customer, dozens or even hundreds of distinct keys might exist, and each key can independently store multiple concurrent values. When data engineers attempt to map this fluid structure into an enterprise analytical store, traditional database design techniques quickly falter.

Architects faced with this scenario often consider creating a dedicated Satellite for every individual attribute key encountered in the source system. If a customer profile contains over a hundred distinct keys, this naive design path yields over a hundred individual Satellites hanging off a single Customer Hub. Managing, loading, and maintaining a hundred Satellites creates immense operational overhead, degrades schema stability, and forces continuous developer refactoring whenever source systems introduce, rename, or deprecate an attribute.

Adhering to core Data Vault design patterns provides a far more elegant, resilient solution. Instead of fragmenting dynamic attributes across dozens of physical tables, you can model these incoming structures using a single, consolidated Multi-Active Satellite within the Raw Data Vault layer. By extending the primary key definition of the Satellite, you retain absolute fidelity to source data without compromising downstream integration or auditability.

Understanding the mechanics of multi-layered multi-activity, designing robust Multi-Active Satellite primary keys, evaluating group delta-checking strategies, and structuring downstream delivery through the Business Vault ensures your architecture remains scalable, performant, and maintainable over the long term.

Understanding Multi-Layered Multi-Activity

To design an effective target schema, architects must first disassemble the two distinct layers of multi-activity inherent in sparse key-value data models. Standard Data Vault Satellites enforce a one-to-one relationship between a Business Key (or Hash Key) and its descriptive record at any single point in time, driven by the Load Date Timestamp. Multi-activity occurs when a single Business Key validly possesses multiple descriptive records within the exact same load batch.

In dynamic key-value structures, multi-activity operates on two simultaneous planes:

Attribute Multi-Activity: A single Business Object, such as a Customer, possesses multiple active descriptive keys simultaneously (e.g., home address, primary phone number, secondary phone number, communication preferences) delivered within the exact same extraction batch.
Value Multi-Activity: A single key for a single Business Object contains multiple active values concurrently (e.g., a customer listing three separate contact phone numbers or multiple shipping addresses under the same logical key attribute).

When combined, these two dimensions produce a complex, multi-layered data structure where a traditional flat table structure is completely absent at the source.

+------------------------------------------------------------------------------------------------------------+
|                                    CUSTOMER BUSINESS OBJECT (Hub Customer)                                 |
+------------------------------------------------------------------------------------------------------------+
                                                      |
                                                      v
+------------------------------------------------------------------------------------------------------------+
|                                  LAYER 1: ATTRIBUTE MULTI-ACTIVITY                                         |
|                               (Multiple Keys Active in the Same Batch)                                     |
|                                                                                                            |
|      +--------------------------------------+              +--------------------------------------+        |
|      | Key: "Phone Number"                  |              | Key: "Address"                       |        |
|      +--------------------------------------+              +--------------------------------------+        |
+------------------------------------------------------------------------------------------------------------+
                           |                                                    |
                           v                                                    v
+------------------------------------------------------------------------------------------------------------+
|                                   LAYER 2: VALUE MULTI-ACTIVITY                                            |
|                              (Multiple Values Active per Single Key)                                       |
|                                                                                                            |
|      +--------------------------------------+              +--------------------------------------+        |
|      | Value 1: "+1-555-0100"               |              | Value 1: "123 Main St, Anytown"       |        |
|      | Value 2: "+1-555-0199"               |              |                                      |        |
|      +--------------------------------------+              +--------------------------------------+        |
+------------------------------------------------------------------------------------------------------------+

Rather than forcing this fluid source data into artificial columns, the optimal Raw Data Vault pattern ingests the payload intact, leveraging a specialized primary key design inside a unified Multi-Active Satellite.

Primary Key Design for Multi-Active Key-Value Satellites

In a standard Data Vault Satellite, the composite primary key consists of the parent Hash Key (or Business Key) and the Load Date Timestamp (LDTS). Because a Multi-Active Satellite allows multiple records for a given Hash Key and LDTS combination, the primary key must be expanded to maintain physical record uniqueness without losing attribute granular details.

To handle dynamic key-value pairs without splitting tables by attribute, you incorporate the attribute key itself alongside a generated Subsequence Number directly into the Satellite’s primary key signature.

Column Name	Primary Key Role	Description	Sample Data
`HK_Customer`	`PK`	Hash key linking to the parent Customer Hub	`a8f5c...`
`LoadDate`	`PK`	Load Date Timestamp representing ingestion time	`2026-10-15 10:00:00`
`AttributeKey`	`PK`	Name of the source key attribute	`PhoneNumber`
`Subsequence`	`PK`	Sequence identifier resolving value-level multi-activity	`1`
`AttributeValue`	Payload	Actual string or contextual payload value	`+1-555-0100`

If a customer submits two phone numbers and one address during a single batch load, the resulting Multi-Active Satellite contains four primary key attributes: HK_Customer, LoadDate, AttributeKey, and Subsequence.

When the staging layer ingests two separate phone numbers under the key PhoneNumber, it assigns a Subsequence value of 1 to the first number and 2 to the second number. This guarantees physical key uniqueness across both layers of multi-activity while keeping all attribute variations consolidated within a single, highly durable table schema.

Evaluating Satellite Splitting vs. Unified Storage

A common question among solution architects pursuing Data Vault certification is whether storing all incoming key-value pairs inside one Multi-Active Satellite violates standards or degrades performance.

In general Data Vault modeling practice, Satellite splitting is performed for specific, business-driven reasons:

Differing Rates of Change: Separating rapidly updating attributes from static descriptive data to optimize storage and change tracking.
Security and Access Control: Isolating Restricted, Confidential, or Personally Identifiable Information (PII) to apply stringent column- or table-level access controls.
Source System Boundaries: Grouping descriptive attributes by their distinct physical source systems or operational domains.

Applying Satellite splits strictly because an attribute key name differs (i.e., “Key-Based Splitting”) leads to severe schema fragility. If a source system introduces a new customer tracking tag tomorrow, a key-split architecture forces immediate schema modifications, pipeline deployments, and model refactoring.

By contrast, preserving key-value data inside a single Multi-Active Satellite keeps the Raw Data Vault layer entirely resilient to upstream changes. New keys flow into the existing Satellite structure as standard data rows without requiring a single DDL statement or software deployment.

Specialized splits should only be applied to this structure when explicit governance rules require it—such as carving off PII keys (like Social Security Numbers or national tax identifiers) into a dedicated, encrypted Satellite.

Raw Data Vault Loading Patterns for Multi-Active Data

Loading Multi-Active Satellites requires careful consideration of delta processing. Because multi-active records represent dynamic sets, traditional single-row change detection using a standard Hash Diff column cannot reliably determine whether a multi-active set has changed between batch loads.

Data Vault practitioners typically choose between two primary ingestion strategies: Full Set Dump Loading and Group Hash Diff Delta Detection.

Strategy 1: Full Set Dump Ingestion (Snapshot Loading)

The simplest and most resilient loading mechanism for multi-active data is inserting the complete incoming attribute set into the Multi-Active Satellite during every execution cycle, bypassing row-level delta checks.

Because modern cloud data warehouses leverage columnar storage, micro-partitioning, and cheap blob storage, holding complete snapshots per load cycle is highly cost-effective. Storing redundant records avoids complex loading algorithms, maximizes ingestion speed, and eliminates edge-case bugs caused by unstable record orderings in source systems.

Query engines bypass redundant data during downstream reads by filtering against partition boundaries or scanning targeted snapshot timestamps, ensuring high compute efficiency despite higher physical storage volume.

Strategy 2: Group Hash Diff Delta Detection

When organizational constraints require keeping the Raw Data Vault lean by suppressing duplicate loads, data teams implement a Group Hash Diff loading pattern.

Under this pattern, the staging pipeline calculates a single, deterministic Hash Diff across the entire multi-active group for a given Business Key rather than computing a distinct Hash Diff per row.

+------------------------------------------------------------------------------------------------------------+
|                                      GROUP HASH DIFF CALCULATION                                           |
+------------------------------------------------------------------------------------------------------------+
|                                                                                                            |
|   Source Group Records (Customer A8F, Batch Day 1):                                                       |
|   - PhoneNumber | 1 | +1-555-0100                                                                         |
|   - PhoneNumber | 2 | +1-555-0199                                                                         |
|   - Address     | 1 | 123 Main St                                                                           |
|                                                                                                            |
|                                                   │                                                        |
|                                                   ▼                                                        |
|   1. Sort group attributes deterministically (e.g., ORDER BY AttributeKey, AttributeValue)                 |
|   2. Concatenate sorted attribute strings into a unified group string                                      |
|   3. Apply cryptographic hash function (MD5 / SHA-256)                                                     |
|                                                                                                            |
|                                                   │                                                        |
|                                                   ▼                                                        |
|   RESULTING GROUP HASH DIFF: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855               |
+------------------------------------------------------------------------------------------------------------+

To execute a Group Hash Diff check:

Deterministic Ordering: The pipeline sorts all active attributes for a given Business Key deterministically (e.g., ordering alphabetically by AttributeKey and AttributeValue).
String Aggregation: The sorted values are concatenated into a single combined string representation.
Group Hashing: A cryptographic hash (e.g., MD5 or SHA-256) is calculated over the aggregated string to produce the Group Hash Diff.

During subsequent batch loads, the pipeline compares the incoming Group Hash Diff against the most recent Group Hash Diff stored in the Satellite for that Business Key:

Unchanged Group: If the incoming Group Hash Diff matches the previous load’s Group Hash Diff, the data is skipped entirely.
Changed Set or Sequence: If any attribute value changes, a key is added, or an existing key is removed, the Group Hash Diff changes. The pipeline inserts the entire updated attribute set as a new multi-active group under the new Load Date Timestamp.

Inserting the complete multi-active set whenever a change occurs—rather than performing partial row updates—is critical for downstream analytical consistency.

Downstream Integration and Information Mart Query Patterns

Maintaining dynamic key-value pairs in a single Raw Data Vault Satellite prioritizes long-term storage stability and automated ingestion. However, business users and analytical tools usually expect traditional columnar layouts (e.g., separate columns for Phone_Number and Primary_Address).

The Data Vault architecture handles this transformation by separating the data integration layer from the data delivery layer.

+---------------------+      +---------------------------------+      +---------------------------------+
|   STAGING LAYER     | ---> |      RAW DATA VAULT LAYER       | ---> |  BUSINESS VAULT / INFORMATION   |
|                     |      |                                 |      |           MARTS LAYER           |
+---------------------+      +---------------------------------+      +---------------------------------+
| Ingest key-value    |      | Multi-Active Satellite          |      | Point in Time (PIT) Tables      |
| payloads; calculate |      | stores raw dynamic pairs using  |      | and Business Vault Views pivot  |
| Subsequence keys.   |      | composite primary keys.         |      | attributes into flat columns.   |
+---------------------+      +---------------------------------+      +---------------------------------+

Business Vault Pivoting

In the Business Vault or Information Mart layer, data engineers build use-case-driven views or tables that pivot sparse key-value rows into structured analytical attributes.

Using SQL conditional aggregation or PIVOT operations, specific keys are extracted from the underlying Multi-Active Satellite to populate modeled presentation structures:

SQL

SELECT 
    HK_Customer,
    LoadDate,
    MAX(CASE WHEN AttributeKey = 'PhoneNumber' AND Subsequence = 1 THEN AttributeValue END) AS Primary_Phone,
    MAX(CASE WHEN AttributeKey = 'PhoneNumber' AND Subsequence = 2 THEN AttributeValue END) AS Secondary_Phone,
    MAX(CASE WHEN AttributeKey = 'Address' AND Subsequence = 1 THEN AttributeValue END) AS Primary_Address
FROM 
    SAT_CUST_ATTR_MA
WHERE 
    LoadDate = :target_snapshot_timestamp
GROUP BY 
    HK_Customer, 
    LoadDate;

By isolating pivoting logic to the Business Vault and Data Mart layers, you insulate your core ingestion models from changing business requirements. If business analysts require a newly added attribute key, engineers simply update or add a Business Vault view without modifying the underlying Raw Data Vault structures.

Simplifying Point-in-Time (PIT) Architectures

Loading multi-active records as unified sets provides significant advantages when constructing downstream Point in Time (PIT) performance tables.

When multi-active records are kept together under a single Load Date Timestamp, a standard PIT table structure can easily track the active state of the entire multi-active group using simple HK_Customer and LoadDate references.

Conversely, if individual key-value rows are inserted independently with different timestamps, standard PIT structures break down. Analysts are forced to construct specialized “Multi-Active PIT” tables that track timestamps at the granular key level (HK_Customer + AttributeKey), dramatically increasing query complexity and join costs across large datasets.

Preserving multi-active set integrity during ingestion ensures PIT tables remain simple, performant, and easy to maintain across complex analytical pipelines.

Establishing a Sustainable Data Vault Architecture

Handling dynamic key-value data models within an enterprise data warehouse requires a balance between source system adaptability and operational simplicity. Attempting to build rigid, column-oriented Satellites for highly dynamic key-value sources creates maintenance debt, pipeline fragility, and endless schema refactoring.

By adopting a unified Multi-Active Satellite pattern in the Raw Data Vault, enterprise data teams achieve:

Schema Stability: Ingesting dynamic, sparse key-value sources without requiring ongoing DDL changes or structural modifications.
Auditability and Lineage: Preserving raw source attributes intact alongside precise load timestamps and sequence keys.
Loading Efficiency: Utilizing Group Hash Diffs or snapshot loads to handle complex multi-activity reliably.
Query Performance: Streamlining downstream analytical queries through structured Business Vault views and simplified Point in Time performance tables.

Leveraging standard Data Vault design principles empowers organizations to integrate complex, fast-changing source structures into a scalable enterprise data warehouse that stays durable over time.

Watch the Video

Marc Winkelmann In Data Vault Friday

Branching and Deployment Strategy

Data Vault for DBT Environment Filtering Strategies

Managing high-volume data transformation platforms like Data Vault 2.0 requires balancing rapid development cycles with cloud compute expenses. When using Data Vault for DBT (formerly datavault4dbt), executing full historical data refreshes across non-production environments—such as Development, Pull Request (PR) preview environments, and Pre-Production—drastically increases Snowflake, Databricks, or BigQuery warehouse costs while extending build times.

Because package macros like datavault4dbt generate standardized Data Vault entities (Hubs, Links, and Satellites), teams frequently face a architectural dilemma: how do you restrict data processing volumes in non-production environments without modifying vendor package code or breaking upstream data dependencies?

Modifying package macros directly by overriding them locally in your DBT project breaks upstream package updates. As soon as a local macro duplicates a package macro name, DBT uses the local version, severing the link to future releases, bug fixes, and feature enhancements provided by package maintainers. Maintaining an upgrade-safe Data Vault deployment strategy requires implementing flexible architectural patterns, native DBT capabilities, and modern data warehouse features.

Architectural Extension Patterns: The Pre-Staging Layer

The primary recommended pattern for limiting data volumes without modifying Data Vault package code is introducing an environment-aware pre-staging layer.

In a standard Data Vault 2.0 architecture, raw source data enters a staging layer where hash keys, payload columns, and metadata tracking fields (such as the Load Date Timestamp, or LDTS, and Record Source, or rsrc) are generated. Attempting to filter data inside the staging layer or directly within the Data Vault entities can invalidate hash calculations or leave downstream Satellites orphaned.

By creating a pre-staging model (or view) directly upstream of your staging layer, you isolate volume-limiting logic to standard SQL statements using DBT target configurations.

SQL

-- models/pre_stage/stg_orders_prefiltered.sql

SELECT 
    order_id,
    customer_id,
    order_date,
    order_status,
    total_amount,
    ldts
FROM {{ source('raw_data', 'orders') }}

{% if target.name in ['dev', 'pr', 'preprod'] %}
WHERE ldts >= DATEADD(day, -2, CURRENT_TIMESTAMP())
{% endif %}

In this setup, the downstream datavault4dbt staging and entity macros process the pre-filtered dataset seamlessly. The Data Vault package macros remain untouched, fully upgrade-safe, and completely agnostic to the environment filtering rules.

Managing Referentially Intact Subsets in Non-Production

While pre-staging filters effectively reduce processing time, applying naive time-window filters (e.g., WHERE ldts >= CURRENT_DATE - 2) introduces a critical data architecture challenge: referential integrity decay.

In transactional data systems, transactional entities (like orders or bank transfers) change continuously, whereas master data entities (like customers, accounts, or products) update infrequently. If a pre-staging filter limits data to the last 48 hours, you will process incoming order transactions but miss historical customer attribute changes that occurred weeks earlier. When loading Data Vault entities:

Hubs and Satellites for transactional data process normally.
Hubs and Satellites for master data receive no updates, leaving key relationships unpopulated in point-in-time (PIT) structures or dimensional Link/Satellite joins.
Downstream Information Marts (like dimensional Star Schemas or Bridge tables) return empty or incomplete datasets, causing automated integration tests in PR environments to fail false-negatively.

To maintain referential integrity without loading full historical datasets, organizations utilize two core strategies: static reference seeds or referentially intact subset generation.

For smaller environments, static reference datasets can be seeded or maintained in a persistent staging area for dimensions, ensuring parent business keys always exist when child transactions process. For enterprise systems, PR pipelines can query a distinct list of business keys present in the filtered fact window and dynamically pull all corresponding historical master records from production.

Modern Branching and Deployment Strategies for Data Vault

Beyond SQL-level data filtering, optimizing non-production run times requires re-evaluating branching models and database environment management.

Traditional enterprise data engineering setups often rely on long-living Git branches corresponding to static database environments (e.g., a dev branch deploying to a DEV database, a preprod branch deploying to a PREPROD database, and a main branch deploying to PROD).

This static environment architecture creates several long-term operational challenges:

Environment Drift: Long-living branches drift apart over time, causing complex merge conflicts and unpredictable deployment behavior when code finally merges into production.
Selective Deployment Friction (Cherry-Picking): When multiple developers push features into a shared dev or preprod environment, deploying a single validated feature to production requires complex Git cherry-picking to exclude unverified features sharing the same branch.
Persistent Storage and Compute Costs: Maintaining fully populated, static pre-production databases incurs continuous cloud storage and compute overhead.

Adopting GitHub Flow and Trunk-Based Development

To eliminate these bottlenecks, modern analytics engineering teams adopt Trunk-Based Development or GitHub Flow.

In a trunk-based architecture, the main branch represents the single source of truth and deploys directly to Production. Developers create short-lived feature branches directly from main. These branches exist only for the duration of a specific feature or bug fix (typically hours or days) and are merged back into main immediately upon code review and continuous integration (CI) validation.

By eliminating intermediate long-living branches like dev or preprod, teams eliminate merge drift and remove the need to cherry-pick commits. Every feature branch is tested in isolation, validated against production-grade data, and merged straight to the main code path.

Zero-Copy Cloning and Dynamic PR Environments

Implementing short-lived feature branches requires dynamic, on-demand database environments. Cloud data platforms decouple storage from compute, allowing data teams to spin up completely isolated, zero-cost database environments instantly during CI/CD execution.

When a developer opens a Pull Request in GitHub or GitLab, an automated pipeline triggers to validate the Data Vault model changes. Instead of maintaining a persistent, static PR database, the pipeline dynamically creates an isolated target environment for that specific PR using cloud-native cloning features:

Snowflake: CREATE TRANSIENT DATABASE PR_1234 CLONE PROD;
Databricks: CREATE TABLE pr_schema.sat_customer SHALLOW CLONE prod_schema.sat_customer;
BigQuery: Metadata-only table clones or copy partitions.

Zero-copy cloning and shallow cloning create immediate metadata pointers to production underlying files without physically copying data files. This process completes in seconds, regardless of whether the production Data Vault contains gigabytes or petabytes of data.

                    ┌─────────────────────────┐
                    │     Production Data     │
                    │   (PROD.raw_vault)      │
                    └────────────┬────────────┘
                                 │
                 Zero-Copy / Shallow Metadata Clone
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Dynamic PR Environment                       │
│                     (PR_BUILD_1234)                             │
│                                                                 │
│  ┌───────────────────────┐             ┌─────────────────────┐  │
│  │ Unchanged Entities    │             │ Modified Models     │  │
│  │ (Read-only Clone)     │             │ (Isolated Rebuild)  │  │
│  │                       │             │                     │  │
│  │  hub_customer         │             │  sat_customer_detail│  │
│  │  link_order_customer  │             │  (Rebuilt via PR)   │  │
│  └───────────────────────┘             └─────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

The PR pipeline builds only the specific Data Vault entities or models that were modified in the feature branch. Because unmodified tables in the PR environment point directly to production storage, modified models write their new test outputs to isolated storage locations without impacting production pipelines or incurring duplicate storage charges. Once the PR is merged or closed, a teardown step drops the temporary PR database or schema automatically.

Organizations seeking end-to-end guidance on architecting scalable pipelines and modern release models can engage with Data Vault consulting teams to establish standardized CI/CD blueprints tailored to their cloud stack.

Advanced DBT Features: Deferral and State Tracking

Combining cloud-native zero-copy cloning with DBT’s native execution flags creates an upgrade-safe, highly performant deployment pipeline.

When developing locally or running CI pipelines, two native DBT capabilities eliminate the need to run full model builds:

1. DBT State Tracking (`--state`)

DBT can compare current project code against a saved production manifest (manifest.json). By running DBT with the state modifier, DBT identifies exactly which models, macros, or sources have changed:

Bash

dbt run --select state:modified+ --state ./production_artifacts

The state:modified+ selection syntax instructs DBT to execute only the models that were directly changed in the current Git branch, along with all downstream dependencies (such as downstream Information Marts, PITs, or Bridge tables). Unchanged upstream Hubs and Links are skipped entirely.

2. DBT Model Deferral (`--defer`)

When building only modified models in a fresh development schema, downstream or sibling models might fail because upstream parent models do not exist in the temporary target schema.

The --defer flag resolves this automatically. When DBT encounters a reference ({{ ref('hub_customer') }}) to a model that does not exist in the current development schema, it defers the reference to the environment defined in the production manifest.

Bash

dbt run --select state:modified+ --state ./production_artifacts --defer --prod-env-manifest ./production_artifacts/manifest.json

During a PR build where only a Satellite model (sat_customer_details) was updated:

DBT skips execution of hub_customer because it was not modified.
DBT builds sat_customer_details inside the dynamic PR schema.
When sat_customer_details queries its parent hub_customer, DBT defers the query execution directly to PROD.raw_vault.hub_customer.

This mechanism completely removes the requirement to manually inject environment-specific WHERE clauses into core Data Vault models or package macros.

To ensure developers and architects maintain consistency across complex modeling environments, enterprise teams routinely complete official Data Vault training to align on standardized pattern designs and operational workflows.

Summary Comparison of Filtering and Deployment Strategies

Choosing the right combination of data limiting and branching strategies depends on organizational maturity, data volume, and cloud platform capabilities.

Strategy Pattern	Implementation Effort	Upgrade Safety	Data Integrity Risk	Cost Savings Potential
Pre-Staging Layer Filtering	Low	High (Package intact)	Medium (Referential breakage if unmanaged)	High
Local Macro Overrides	Low	Low (Breaks upstream package updates)	Low	High
Zero-Copy / Shallow Cloning	Medium	High	None (Full Prod dataset referenced)	Very High (No storage copy)
DBT Deferral & State Selection	Medium	High	None	Maximum (Computes changed models only)

By pairing an upstream pre-staging layer for local sandbox testing with Trunk-Based Development, zero-copy environment cloning, and DBT state deferral for CI pipelines, data engineering teams achieve rapid execution, strict cost control, and full alignment with future datavault4dbt package releases.

Watch the Video

Marc Winkelmann In Data Vault

How Data Vault Supports AI and ML Readiness in the Modern Data Platform

Hologram of Padlock on sunset panoramic cityscape of Bangkok, Southeast Asia. The concept of cyber security intelligence. Multi exposure.

How Data Vault Supports AI and ML Readiness

Most organisations today are not failing at AI because they chose the wrong model. They are failing because they built on the wrong foundation. The model is rarely the bottleneck — the data underneath it is.

At Scalefree, working with clients across Europe, we see this consistently: AI workflows will never succeed at scale without proper data support. Not eventually. Always.

This article explains why, and what a mature, AI-ready data architecture actually looks like — with Data Vault 2 at its core.

In this article:

Where Most Companies Are Right Now
Reasons That Stop Companies From Scaling with AI
Why AI Workflows Fail Without a Data Foundation
The Enabling Data Platform: What It Looks Like
How This Architecture Directly Solves the AI Scaling Problem
Data Vault 2 as the Foundation for AI Readiness
Starting the Journey: Three Things to Do in Parallel
Want to Go Deeper?

Where Most Companies Are Right Now

The journey most organisations follow with AI looks roughly the same. It starts with discovery — the first time someone opens a chatbot, enters a prompt, and gets a result that genuinely surprises them. That moment creates momentum.

What follows is an extended period of experimentation. Prototypes are built. Some use cases work. Others fail — not because AI is incapable, but because the setup was wrong, or the use case was not worth the complexity it introduced. This phase is characterised by learning, and by a growing realisation that surfaces quickly: the same problems data engineers have been solving for 20 or 30 years have not gone away. They have simply reappeared under a new name.

Structured processes are needed. Governance is needed. Data integration is needed. The foundation matters — and for many organisations, that foundation is not ready.

The companies that move beyond experimentation into genuine AI maturity are not the ones that found a better model. They are the ones who built a better platform first.

Reasons That Stop Companies From Scaling with AI

Two patterns emerge consistently when AI projects reach the limits of their initial setup.

The first starts innocently. A workflow tool is connected to a data source and a language model. It works. Results are impressive. Then a second data source is added, then a third. A quality control step is introduced. A loop is needed. A second agent handles edge cases. What started as a clean prototype becomes a tangled, fragile system that is expensive to maintain and nearly impossible to debug. When errors appear — and they will — fixing them means untangling months of accumulated complexity. Multiply this across ten, twenty, or forty processes in an organisation, and the maintenance burden alone consumes any efficiency the AI was supposed to create.

The second pattern emerges from urgency. Business users want to move quickly, and internal IT is often a bottleneck. The response is shadow AI: tools adopted outside governance controls, company data uploaded to external platforms without authorisation, processes built that bypass audit trails, GDPR compliance, and data ownership rules. It produces results fast. It also creates legal exposure, data leakage risk, and a complete loss of organisational visibility into what AI is actually doing with company data. For many organisations today, if asked honestly what AI processes are running and what data they are using, the honest answer is: we do not know.

Both patterns are understandable in how they start. Both become critical problems at scale.

Why AI Workflows Fail Without a Data Foundation

The root cause in both cases is the same: attempting to solve data problems inside an AI workflow rather than before it.

When an AI agent needs to access 20 different types of information — personal contact data, past purchase history, product catalogue, past email correspondence, website behaviour, and company context — and that data lives in disconnected source systems with no integration layer, every new data point added to the workflow increases complexity. Quality issues compound. Costs rise with every additional token consumed. And because AI systems produce different results every time they process inconsistent input, errors are unpredictable and difficult to reproduce.

Clean, integrated, well-described data does not make AI smarter in a general sense. It makes AI consistently useful — which is what actually matters when deploying at enterprise scale.

The Enabling Data Platform: What It Looks Like

The architecture Scalefree recommends for AI-ready data platforms follows a clear logical structure, regardless of which specific tools an organisation uses.

Source systems feed into a Persistent Staging Area, where raw data is collected and preserved as-is. This is not a transformation layer — it is a historical record of everything that arrived, in the form it arrived in.

From there, data moves into an integration layer. This is where a Data Vault 2 modeling standard sits. The role of this layer is to integrate data from different source systems, resolve business key conflicts, clean data, and build a single, auditable, historically complete view of the business. It can grow and evolve as new sources are added, without restructuring what already exists. It is also built piece by piece, use case by use case, which means an organisation does not need to build the entire platform before deriving value from it.

Above that, the platform builds Feature Marts. Feature Marts are the direct interface between the data platform and AI agents. They are optimised for AI consumption — sometimes flat and wide, sometimes in a dimensional modeling style, with rich semantic descriptions that help an AI agent understand not just what the data is, but what it means. A Feature Mart might contain all prospect activity data in a single unified view, ready for an agent to consume without having to navigate joins, resolve conflicts, or interpret raw table structures.

AI agents plug into Feature Marts. They do not go directly to the predecessor layers, and they certainly do not go directly to source systems. The Feature Mart is the clean, governed, role-restricted interface that makes agents reliable.

How This Architecture Directly Solves the AI Scaling Problem

Security and access control. When an AI agent connects to a Feature Mart rather than a raw database, access can be scoped precisely. The agent sees only the data it needs for its specific function. If the agent is compromised, the blast radius is limited. This is the same principle applied to any employee or system — give it exactly the access it needs, nothing more.

Data integration. An organisation’s data engineers have already done the work of cleaning, integrating, and resolving data quality issues across source systems. AI engineers do not need to rebuild this. They need to collaborate with data engineers to shape Feature Marts from data assets that already exist. This is a fundamental shift in how AI teams and data teams should work together — and it accelerates the time from idea to deployed AI use cases.

Full audit trail. With a proper data platform, every piece of data that flows into an AI decision can be traced back to its source, with a timestamp. The output can also be stored back in the platform. This means that when a compliance question arises — which data informed this decision, on which date, processed by which agent — the answer is available.

GDPR and compliance. If data deletion rules are already implemented in the platform, they extend automatically to AI agents. Data that has been deleted from the platform under GDPR rules will not be served to an AI agent via the Feature Mart. Compliance is inherited, not rebuilt for each use case.

Cost control. Providing an agent with a clean, pre-integrated Feature Mart means it processes the right data once, rather than consuming tokens navigating raw, inconsistent, or duplicated data sources. Token costs are a real and growing concern for organisations running AI at scale. A well-structured data foundation is also a cost optimisation strategy.

Semantic layers. AI agents make mistakes when they do not understand the data they are working with. A data catalog and semantic layer that provides meaningful descriptions of every data asset — what a field means, how a metric is calculated, what business context surrounds a particular entity — reduces AI hallucinations significantly. This is especially important as organisations move toward conversational interfaces that allow business users to query data in natural language.

Data Vault 2 as the Foundation for AI Readiness

Data Vault 2 is not simply a modeling technique. It is a complete methodology covering architecture, modeling, and implementation standards. Its particular strengths make it well-suited as the integration layer in an AI-ready platform.

The insert-only, historically complete nature of Data Vault means that every version of every piece of data is preserved. An AI agent working with a Feature Mart derived from a Data Vault has access to the full history of a business entity — not just its current state. This matters significantly for ML models that rely on historical patterns, and for audit requirements that demand a complete record of what was known at any point in time.

Data Vault’s business-key-centred approach means that data from multiple source systems can be integrated without losing the original context of each source. An AI agent drawing on customer data that has been properly integrated through a Data Vault model is working with a single, coherent view of that customer across every system in the organisation — rather than multiple conflicting records from disconnected databases.

Data Vault 2 also extends the original methodology to address real-time and semi-structured data patterns — JSON structures, streaming sources, event-driven loading — which are precisely the data types that AI-driven workflows increasingly depend on.

Starting the Journey: Three Things to Do in Parallel

Organisations do not need a complete data platform before they begin building AI use cases. What they need is a plan to build both in parallel, with the right teams working together.

The first priority is extending the existing data platform to serve AI applications — identifying which data assets already exist, which Feature Marts can be built from them quickly, and which source systems need to be connected next.

The second priority is identifying the right processes to automate. The most valuable targets are not tasks unique to one person, but processes that many people across the organisation perform repeatedly — the same steps, executed at scale, across departments. These are the processes where AI creates compounding returns.

The third priority is building cross-functional teams. AI engineers, data engineers, and business users need to work together from the start. Business users understand the processes. Data engineers have already solved the data integration and quality problems. AI engineers know which models fit which constraints and how to optimise for cost and performance. No single group has all three — and organisations that try to run AI projects without all three perspectives will hit the limits of that approach quickly.

Want to Go Deeper?

If your organisation is evaluating its data platform readiness for AI, or if you are already building AI workflows and hitting the limitations described in this article, Scalefree offers a free Data Vault Handbook — 60 pages covering the fundamentals of Data Vault and where it fits in a modern data architecture. Available to order to your door, free of charge within Europe.

For teams ready to take the next step, the Data Vault 2.1 Training & Certification equips data engineers and architects with the methodology, modeling skills, and CDVP2.1 credential to build and govern Data Vault implementations at enterprise scale.

To discuss how Scalefree can support your data platform or AI readiness journey, get in touch directly.

Marc Winkelmann In Data Vault

Data Vault and Data Mesh: Not Versus, But Together

Best Practices for Data Mesh Implementation

Data Vault and Data Mesh

Few topics generate more confusion in enterprise data architecture than the relationship between Data Vault, Data Mesh, and Data Fabric. Online discussions often frame these as competing approaches — as if an organisation must choose one and abandon the others. This framing is wrong, and understanding why matters for anyone building a serious data platform in 2025 and beyond.

These three concepts operate at different levels. They address different problems. And when combined correctly, they complement each other in ways that none of them can achieve alone.

In this article:

Data Vault, Data Mesh, and Data Fabric Are Not Competing
Why Data Vault Is the Foundation Data Mesh Needs
The Problem With Going "Full Data Mesh"
How This Architecture Works in Practice
What Makes This Combination Work
Data Catalogs and Governance as Connective Tissue
Starting the Journey

Data Vault, Data Mesh, and Data Fabric Are Not Competing

The confusion starts because all three terms appear in similar conversations — data platform architecture, enterprise data strategy, scalability. But they are not alternatives to each other.

Data Fabric is a technical approach. It defines the architecture of a data platform — how data moves from source systems through integration layers to delivery, how metadata drives automation, how access is governed, and how the platform scales. It is the engineering blueprint.

Data Mesh is an organisational approach. It defines who is responsible for data, how teams are structured, how data products are owned and maintained, and how to avoid the bottlenecks that emerge when a single central IT team is responsible for everything. It is the operating model.

Data Vault is the methodology that sits between the two. It provides the modeling technique, the reference architecture, the implementation standards, and the agile delivery framework that make both a high-quality Data Fabric and a functional Data Mesh possible. It is the glue.

None of these replaces the other. An organisation can have a Data Fabric architecture without Data Vault — but it will lack the standardisation and automation that make the platform scalable. It can adopt Data Mesh principles without Data Vault — but the integration layer will be fragile and inconsistent. Data Vault without a clear architectural vision and organisational operating model delivers solid modeling but leaves the surrounding platform undefined. For a deeper look at how these three approaches fit together architecturally, Scalefree’s guide on Data Vault, Data Mesh, and Data Fabric covers the full modern architecture picture.

Why Data Vault Is the Foundation Data Mesh Needs

Data Mesh’s central argument is that centralised IT teams become bottlenecks as data platform requirements grow. The solution is to distribute ownership — giving domain teams (sales, finance, operations, logistics) responsibility for their own data products rather than routing every requirement through a central team.

This is a sound organisational idea. But it creates a serious technical problem: if every domain team builds its own data pipelines from scratch, organisations end up with redundant data, conflicting definitions of the same business objects, and a proliferation of unmaintained pipelines. The very inefficiency Data Mesh was designed to solve reappears in a different form.

Data Vault solves this problem at the architecture level. The key insight is that not all layers of a data platform benefit equally from decentralisation.

The Raw Data Vault — the layer that absorbs raw data from source systems and integrates it using business keys — should remain centralised. This layer contains no business logic. It simply records what arrived, when, and from where. Because it is standardised and highly automatable, a small central team can maintain it with minimal overhead. And because it is centralised, every domain team draws from the same single source of facts — the same customer records, the same product data, the same account structures — rather than each team pulling its own version from disconnected pipelines.

The Business Vault and Information Marts, by contrast, are exactly where domain knowledge matters. Business rules, calculated metrics, KPI definitions, and data product shaping all require the kind of deep domain understanding that lives in business teams, not in central IT. This is where decentralisation makes sense — and where Data Mesh principles directly apply.

The result is a practical middle ground: centralise the Raw Vault where standardisation creates efficiency, decentralise the Business Vault and Information Marts where domain knowledge creates value. This is not a theoretical compromise — it is the architecture that enterprise Data Vault implementations demonstrate works at scale.

The Problem With Going “Full Data Mesh”

Fully decentralised Data Mesh — where every domain team manages its own data end to end, from source ingestion to delivery — sounds attractive in theory. In practice, it replicates the problems of the pre-data-warehouse era, where every department ran its own shadow IT, built its own pipelines, and maintained its own version of business objects that nobody else could join reliably.

When domain teams each ingest their own version of a source system, the same Salesforce data gets pulled ten times by ten different teams with ten slightly different transformation approaches. The same customer appears as ten different records with ten different definitions of “active.” Joining across domains becomes a project in itself. The governance that Data Mesh promises — through federated standards and data contracts — is extremely difficult to enforce when no shared foundation exists.

Full centralisation has its own problems, of course. A single IT team responsible for all data products across a large organisation will always struggle with prioritisation, domain knowledge gaps, and delivery speed. The bottleneck that Data Mesh identifies is real.

The architecture that resolves this tension uses Data Vault’s layered structure as the boundary between centralised and decentralised work. Central team: source ingestion, Raw Vault, automation infrastructure, platform governance. Domain teams: Business Vault logic, Information Marts, data product definition, and ownership.

How This Architecture Works in Practice

Organisations that implement this combined approach typically follow an evolution rather than a big-bang restructuring. Teams begin working together on a centralised platform, building the Raw Vault and establishing the automation patterns and tooling. As the platform matures and team members develop deep platform knowledge, those people move into domain teams — bringing their technical expertise with them, closer to the business knowledge that domain work requires.

The central team shrinks to a small core — often just two or three people — responsible for maintaining the Raw Vault, the automation infrastructure, and the platform governance layer. Domain teams handle everything from the Business Vault outward, working with full autonomy on their data products while drawing on the shared, integrated foundation beneath them.

This approach has a specific advantage when organisations grow through acquisition. When a company absorbs another — bringing new source systems, new customer records, new business objects — the Raw Vault absorbs the new data without restructuring what already exists. New Hubs, new Satellites, and new integration logic are added incrementally. Existing data products continue to function. The integration project that would take a traditional data warehouse months or years can be completed in weeks.

The same scalability applies to organic growth. New business units, new products, new markets — each can be onboarded as a new domain team, drawing from existing Raw Vault entities where overlap exists, adding new entities where it does not. The platform grows with the organisation rather than requiring periodic rebuilds.

What Makes This Combination Work

Three characteristics of Data Vault make it particularly well suited as the foundation for a Data Fabric and Data Mesh architecture.

Standardisation enables automation. Data Vault’s Hub-Link-Satellite structure is highly consistent. Once the patterns are established and the metadata is defined, the loading of Raw Vault entities can be generated automatically rather than hand-coded. This is precisely what Data Fabric requires — and precisely what makes the central layer maintainable by a small team even as the number of source systems grows. For a detailed look at how datavault4dbt implements this automation approach, Scalefree’s tooling is specifically designed around this principle.

Historical completeness supports data products. Data Mesh’s concept of the data product — a trusted, documented, governed dataset that domain teams can consume and build upon — requires a reliable foundation. A data product built on a Raw Vault entity inherits complete historical data, a full audit trail, and provable lineage back to source. These are the properties that make data products trustworthy enough to use in downstream analytics, AI applications, and regulatory reporting.

The layered architecture maps naturally to organisational boundaries. Raw Vault and Business Vault are not just technical distinctions — they correspond to a meaningful organisational divide between technical data engineering work and business knowledge work. The architecture makes the organisational model explicit rather than leaving it implicit, which reduces friction when defining team responsibilities and data product ownership.

Data Catalogs and Governance as Connective Tissue

A combined Data Vault, Data Mesh, and Data Fabric architecture only delivers its full value when metadata is managed seriously. Domain teams need to be able to discover what data products already exist, understand their lineage, know how fresh the data is, and assess whether an existing product meets their needs before building a new one.

Without a well-maintained data catalog, teams rebuild work that already exists, queries return answers that conflict with other answers, and the governance that Data Mesh requires to function collapses into informal agreements and institutional knowledge held by a few individuals.

With a proper catalog — one where every data product is documented, every entity has clear ownership, every metric definition is visible, and lineage traces from source to delivery — the platform becomes genuinely self-service. Non-technical users can find and use data products without IT support. AI use cases that require querying data in natural language become feasible. And the platform can scale to serve a large organisation without proportional growth in support overhead.

Data Vault contributes directly to catalog quality. The Raw Vault’s record source and load date on every entity provide automatic lineage. The standardised naming conventions make entities discoverable. The separation of Raw Vault from Business Vault makes the application of business rules explicit and auditable rather than buried in opaque transformation logic.

Starting the Journey

For organisations considering this architectural direction, the starting point is almost always the same: begin with the data platform foundation before attempting to distribute ownership. Teams need to understand the platform — how the Raw Vault works, how automation is configured, how the Business Vault extends the raw layer — before they can work effectively within domain structures.

The Data Vault 2.1 Training & Certification equips data engineers and architects with the complete methodology — from Raw Vault design through Business Vault patterns to Information Mart delivery — so they can build and govern this kind of platform with confidence. For teams evaluating their current architecture and planning the move toward a more scalable and governed platform, Scalefree’s Data Platform Review provides an expert assessment and a clear recommended path forward.

The question is not whether to choose Data Vault, Data Mesh, or Data Fabric. The question is how to combine them in the sequence and proportion that fits your organisation’s current maturity and growth trajectory. The answer, in almost every case, starts with the same foundation: a clean, standardised, automated Raw Vault that every domain team can trust.

For further reading on how Scalefree approaches enterprise data platform architecture, the free Data Vault Handbook covers the core methodology, and the Data Vault consulting practice works with organisations at every stage of the implementation journey.

Marc Winkelmann In Data Vault Friday

Handling Contradicting Facts About a Single Business Key in Data Vault

A frequent dilemma facing data warehouse architects and BI engineers during batch processing is determining how to manage contradicting descriptive data associated with a single business key. In modern enterprise environments, data ingested into the staging area often presents conflicting values within the exact same load cycle. These conflicts typically manifest under two primary operational scenarios: multi-source data feeds providing divergent descriptive attributes for a common entity, or Change Data Capture (CDC) streams containing rapid successive updates packed into a single batch delivery.

Resolving these conflicting data points correctly without compromising core enterprise data modeling standards requires a clear distinction between the Raw Data Vault layer and downstream Business Vault components. Teams embarking on enterprise data platform modernizations often enroll in formal Data Vault training to master the separation of concerns necessary for handling complex multi-source consolidation and high-frequency stream integration.

Understanding the structural mechanics behind source prioritization and CDC sequence ordering ensures that auditability remains entirely intact while downstream reporting tools receive unified, accurate golden records.

In this article:

Scenario 1: Resolving Multi-Source Contradictions via Source Prioritization
Scenario 2: Handling High-Frequency Change Data Capture (CDC) Batches
Architectural Integrity in Enterprise Data Platforms
Watch the Video

Scenario 1: Resolving Multi-Source Contradictions via Source Prioritization

The first scenario occurs when multiple source systems deliver descriptive attributes regarding the exact same business key within the same processing window. For example, an organization’s HR system and CRM platform may both store employee records. The HR system serves as the primary master system for internal workforce data, recording an employee name as “Marc Winkelman” with a “c”. Simultaneously, the CRM platform captures customer-facing account managers and registers the same employee key with the name “Mark Winkelman” using a “k”.

When both source extracts are staged concurrently, both records share the exact same business key and receive the exact same Load Date Timestamp (LDTS). How the data repository ingests and downstream views present this conflicting descriptive data determines the ultimate stability of the data warehouse.

Raw Data Vault Satellites: Single Source per Satellite

A common design mistake is attempting to consolidate conflicting records from disparate source systems into a single Raw Data Vault Satellite. Inserting conflicting attributes from multiple systems into one Satellite breaks best practices, leads to primary key collisions, and introduces undesirable source coupling.

In a properly modeled Data Vault 2.0 architecture, the golden rule remains strict: one source table feeds one target table in the Raw Data Vault. The Hub table holds the distinct business key once, acting as the passive integration point. However, two separate Satellites are constructed beneath that Hub: an HR Employee Satellite and a CRM Employee Satellite. Each Satellite captures the raw state delivered by its respective operational system, complete with its own load metadata and descriptive payload.

This architecture completely preserves auditability. If the CRM source pipeline experiences a delay or failure at 3:00 AM while the HR extract completes successfully at 1:00 AM, the HR pipeline executes independently without waiting for or depending on CRM data availability.

Business Vault Alignment and Point-in-Time (PIT) Tables

Because the raw data remains split across system-specific Satellites, resolving which value takes precedence is deferred to the Business Vault and Information Mart layers. This approach treats data prioritization as an agile, adaptable business rule rather than a rigid physical constraint imposed during raw ingestion.

To unite these disparate timelines on a common axis, data engineers construct Point-in-Time (PIT) tables. A PIT table creates regular snapshot dates across all Satellites attached to a parent Hub or Link object. The PIT table contains the business key, the Snapshot Date Timestamp (SDTS), and key pointers—specifically the combination of business key and load date timestamp—for each corresponding Satellite record active at that snapshot interval.

Formulating the Golden Record using COALESCE Logic

Once the PIT table establishes the alignment pointers across all source Satellites for a given snapshot, an Information Mart view (such as an Employee Dimension) can construct a single unified record. Priority rules are applied using standard database expressions like COALESCE within the selection logic.

To implement this prioritization pattern, the view performs an INNER JOIN from the central PIT table into the HR Satellite based on the HR pointer keys, alongside another INNER JOIN into the CRM Satellite using the CRM pointer keys. Because the PIT table brings both pointers onto a single row, the SELECT clause can evaluate the attributes conditionally.

Using COALESCE(hr_sat.employee_name, crm_sat.employee_name), the query engine attempts to extract the employee name from the authoritative HR Satellite first. If the attribute is null or absent for that snapshot, the expression falls back gracefully to the CRM Satellite’s attribute value. This pattern ensures deterministic, rule-based cherry-picking of attributes while maintaining full traceability back to the individual underlying Satellites.

Scenario 2: Handling High-Frequency Change Data Capture (CDC) Batches

The second scenario involves receiving multiple state changes for a single business key from a single source system inside a single micro-batch or Change Data Capture (CDC) pipeline extract.

In high-volume operational engines, an entity attribute might undergo several rapid updates within seconds. For instance, a user might enter a name as “Marc”, update it to “Mark”, and subsequently correct it back to “Marc”. When the CDC ingestion pipeline processes these events in a nightly or hourly batch, all three state changes arrive bundled in a single file or stream package.

The Pitfall of Primary Key Violations

If an ETL process attempts to attach a single batch-level Load Date Timestamp to all three CDC change rows, inserting them directly into a standard Satellite results in a primary key violation. In Data Vault modeling, the primary key of a standard Satellite is defined as the composite of the parent key (e.g., hash keys in the Data Vault) and the Load Date Timestamp.

Assigning an identical timestamp to three distinct state changes for the same business key destroys uniqueness and prevents standard delta tracking. Furthermore, treating these rapid sequential changes as a multi-active payload is an anti-pattern. Multi-active Satellites are explicitly designed for instances where multiple child attributes are concurrently valid at the exact same moment in time (such as multiple active phone numbers delivered in a flattened JSON array), not for sequential historical updates.

Owning the Load Date Timestamp vs. Source Dates

A critical architectural principle in Data Vault design is that the data warehouse team must retain complete ownership over the generation and management of the Load Date Timestamp. Relying on source-supplied timestamps or operational system last-modified dates as the physical primary key of a Satellite introduces significant risk.

Operational business dates supplied by source systems are frequently subject to quality issues. Source timestamps can be null, formatted inconsistently across tables, adjusted retrospectively by database administrators, or corrupted during system migrations. Using a source date as a Satellite primary key key breaks the load dependency chain and impedes deterministic incremental loading. The Load Date Timestamp must strictly represent the point in time when the record entered the data warehouse ecosystem.

Resolving Sequential Order with Technical Sequence Numbers

To load multiple intra-batch CDC updates into a Satellite without violating primary key constraints or relying on unstable source dates, data engineering pipelines utilize technical sequence numbers or change log row IDs.

CDC source systems almost universally output a technical sequence ID, incrementing transaction counter, or precise log sequence number (LSN) alongside each captured record. When processing the staged CDC batch, the loading mechanism evaluates this sequence order and adjusts the applied Load Date Timestamp slightly for each consecutive record within the batch—typically adding fractional sub-second increments (such as microseconds) per row sequence.

By adding microsecond adjustments based on the technical stream sequence, each consecutive record inside the CDC batch receives a unique, monotonically increasing Load Date Timestamp. This preserves the precise order in which changes occurred in the operational source system while satisfying the primary key requirements of the standard Satellite structure.

Architectural Integrity in Enterprise Data Platforms

Maintaining clean separation between source ingestion, business rules, and presentation layers is essential for long-term scalability. Combining multiple source systems into a single raw Satellite to avoid duplicate records introduces severe maintenance liabilities, forces pipeline dependencies, and undermines the modularity of the data platform.

Keeping raw data isolated by source system within the Raw Data Vault layer guarantees that operational failures in one source do not disrupt the availability of independent domain pipelines. The Business Vault layer, empowered by Point-in-Time tables and rule-driven Information Mart views, handles source harmonization, prioritization logic, and deduplication dynamically. Reviewing your platform architecture through comprehensive data platform review services ensures that your ingestion pipelines balance high throughput with total auditability.

By applying deterministic sequence ordering to intra-batch CDC records and leveraging PIT-backed COALESCE selection across multi-source Satellites, data teams effectively resolve all forms of contradicting facts—ensuring high query performance and absolute governance across the entire enterprise data asset.

Watch the Video

Marc Winkelmann In Data Vault

How Long Does The Data Vault Certification Take? Timeline Explained

Person standing above the clouds looking toward a bright horizon — Data Vault certification journey

How Long Does Data Vault Certification Take?

It is one of the first questions anyone asks before committing to a professional certification: how much time does this actually take? For Data Vault certification, the answer is straightforward — and the structure is designed to fit around the reality of working professionals. Here is the complete timeline, from first login to certified practitioner.

In this article:

The Full Timeline at a Glance
Phase 1 — Pre-Course Self-Paced Videos (Approximately 15 Hours)
Phase 2 — Live Instructor-Led Training (3 Days)
Phase 3 — Exam Window (8 Weeks, 2 Attempts Included)
Phase 4 — Post-Certification Platform Access (6 Months)
Complete Timeline Summary
Is There a Faster Path?
Who Is the Certification For?
Next Training Dates

The Full Timeline at a Glance

The CDVP2.1^® (Certified Data Vault 2.1 Practitioner) certification follows a defined sequence. There is pre-course preparation, a live instructor-led training block, an exam window, and post-certification access to continued learning resources. Each phase has a clear duration.

In total, from starting your preparation to sitting the exam, most candidates complete the process within 10 to 11 weeks.

Phase 1 — Pre-Course Self-Paced Videos (Approximately 15 Hours)

Before the live training begins, candidates work through a set of self-paced video modules covering the foundational concepts of Data Vault 2.1. These cover the reference architecture, the core modeling components (Hub, Link, Satellite), the agile data methodology, and the layered structure of a Data Vault platform.

The self-paced content runs approximately 15 hours in total. You work through the material at your own pace and must complete it before the start of the instructor-led live training. Most candidates spread this over two to three weeks, fitting it around their regular work schedule.

Completing the pre-course material before the live training is important. The three days of instructor-led sessions build directly on these foundations, and arriving prepared means you get significantly more value from the live discussions, Q&A, and hands-on exercises.

Phase 2 — Live Instructor-Led Training (3 Days)

The instructor-led component of the Data Vault 2.1 Training & Certification runs for three consecutive days and is conducted online or on-site by a Scalefree-certified instructor, depending on the training format you choose. Sessions cover Data Vault modeling in depth, architecture patterns, implementation approaches, automation concepts, and the methodology for managing Data Vault projects in a real enterprise environment.

The live format matters. This is not a recorded walkthrough — it is an interactive session where participants bring real questions from their own projects, and the instructor works through edge cases, design decisions, and common mistakes in real time. Attendees regularly join from across Europe and beyond, which means the discussion reflects a broad range of industries and tool stacks.

Three days is intensive. By the end of day three, candidates have covered the full scope of the CDVP2.1^® examination and have had the opportunity to clarify anything that needs it before the exam window opens.

Phase 3 — Exam Window (8 Weeks, 2 Attempts Included)

After the live training concludes, the exam window opens. Candidates have eight weeks to sit the CDVP2.1^® examination, and two attempts are included in the certification package.

The exam is proctored and taken online, so there is no need to travel to a testing centre. The eight-week window gives candidates flexibility to review the material, consolidate what they learned during the live sessions, and choose their own moment to sit the exam — whether that is the week after training or closer to the deadline.

Two attempts are a meaningful safety net. Most candidates who arrive prepared from the pre-course material and engage actively during the live training pass on their first attempt. The second attempt is there if you need more time to solidify a particular area before trying again.

Phase 4 — Post-Certification Platform Access (6 Months)

Passing the CDVP2.1^® exam is not the end of the learning journey. Certified practitioners receive six months of access to the Data Vault Alliance platform, which includes extended reference material, community resources, and continued learning content.

This post-certification access is particularly valuable for practitioners who are actively implementing Data Vault on a project during or after the training period. Having a structured reference resource available as real implementation questions arise — rather than only during the training itself — is a practical advantage that experienced candidates consistently highlight.

Complete Timeline Summary

Phase	Duration	Format
Pre-course self-paced videos	~15 hours (self-paced, typically 2–3 weeks)	Online, on-demand
Live instructor-led training	3 days	Online, on-site
Exam window	8 weeks (2 attempts included)	Online, proctored
Post-certification platform access	6 months	Data Vault Alliance platform

Is There a Faster Path?

The structure above is designed to be as efficient as it is thorough. Three days of instructor-led training is a concentrated format — the same content would typically be spread over a much longer period in a self-paced programme. For data engineers and architects who are already working with data warehousing concepts, the pre-course videos are often faster than the 15-hour estimate because much of the foundational context is already familiar.

There is no shortcut through the exam itself — the CDVP2.1^® is a rigorous, proctored assessment — but the preparation path is as streamlined as it can reasonably be while still producing practitioners who can apply the methodology on real projects.

Who Is the Certification For?

The CDVP2.1^® certification is designed for data engineers, data architects, BI developers, and technical team leads who are building or planning to build a Data Vault-based data platform. Prerequisites include solid SQL knowledge, experience with data warehousing or BI development, and a working understanding of data modeling fundamentals. The pre-course videos are designed to bring everyone to a consistent baseline before the instructor-led training begins.

For teams considering Data Vault at an organisational level, Scalefree also offers in-house training — the same certification program delivered privately for your team, at a date and format that fits your schedule.

The free Data Vault Handbook is a good starting point if you want to understand the methodology before committing to training. You can also contact Scalefree directly to discuss in-house options or upcoming public training dates.

Next Training Dates

Public Data Vault training runs on a regular schedule throughout the year. Check the current calendar on the training page for upcoming dates, and register early — cohort sizes are intentionally kept small to maintain the quality of the instructor-led sessions.

Marc Winkelmann In Data Vault Friday

How to Deal With Late Arriving Data

Late Arriving Data

Late arriving or backdated data is a common challenge in data warehousing. In Data Vault, it is important to distinguish between the technical timeline used for loading data and the business timeline representing when events actually occurred in the real world.

In this article:

1. Technical Timeline vs Business Timeline
2. Capturing the Business Timeline
3. Timeline Corrections Without an Extended Tracking Satellite
4. Practical Guidelines
Summary
Watch the Video

1. Technical Timeline vs Business Timeline

When loading data into the Raw Vault, always use a Load Date Timestamp (LDTS):

Set when the record first arrives in your target system (landing zone, data lake, or Raw Vault).
Never backdate this timestamp—it should always move forward.
Used for incremental loading, delta detection, and reproducibility of snapshots.

This timestamp does not reflect the real-world timing of the data. It is purely a technical artifact to track ingestion order.

2. Capturing the Business Timeline

To handle late arriving or backdated data, use descriptive business dates stored in your satellites, such as:

Apply Date / Effective Date: When the data became valid in the source system or real world.
Last Modified Date: When the record was last changed in the source system.

These business timestamps allow you to create snapshots or temporal views that reflect the true order of events.

3. Timeline Corrections Without an Extended Tracking Satellite

You can correct timelines without adding additional satellites by leveraging the business timestamps stored in your existing satellites:

Create temporal PIT tables or snapshots based on the business timeline, not the load date.
When late-arriving data is detected:
- Option 1: Rebuild the affected snapshots to include the late data.
- Option 2: Apply counter transactions to reverse previous measures and apply the updated values.
Always keep the load date unchanged—it only tracks ingestion, not validity.

This approach ensures that your historical reports reflect the correct business sequence without complicating the Raw Vault model.

4. Practical Guidelines

Do not order or aggregate data using the load date when interpreting or reporting; always use business dates.
Maintain separate timelines:
- Load Date: Technical, for data ingestion and reproducibility.
- Business Date: For interpretation, analysis, and handling late arrivals.
Rebuild snapshots or use counter transactions as necessary when late data affects measures or aggregates.

Summary

Late arriving data can be handled in Data Vault without adding extra tracking satellites by clearly separating technical and business timelines. Load Date timestamps remain forward-only, while satellites store business dates to drive temporal snapshots and corrections. Using temporal PIT tables, counter transactions, or snapshot rebuilding ensures your analytics reflect the real-world timeline accurately.

Watch the Video

Marc Winkelmann In Data Vault

What’s New in Data Vault 2.1 and Why It Matters for Modern Data Warehousing

What’s New in the Data Vault 2.1 Training

In the world of data warehousing and business intelligence, the Data Vault methodology has long been a trusted foundation for scalable and agile data architectures. With the release of Data Vault 2.1, the methodology has evolved to address new challenges in modern data environments — from handling semi-structured data to aligning with concepts like Data Mesh and Data Lakehouse.

CDVP2.0 and CDVP2.1 are separate courses, separate assessments, and separate professional certifications. The CDVP2.1 course materials were substantially revised and expanded, and the CDVP2.1 certification assessment questions were completely rewritten. A CDVP2.0 credential does not automatically confer CDVP2.1. To earn CDVP2.1, a professional must complete the separate CDVP2.1 course and certification.

In this article, we summarize what’s new in Data Vault 2.1 compared to 2.0, what these changes mean for practitioners, and how you can take advantage of the new training materials to become officially certified.

In this article:

1. A Major Expansion in Content and Learning Resources
2. Enhanced Instructor-Led Training Experience
3. Better Preparation for Certification
4. Dealing with JSON and Semi-Structured Data
5. Stronger Differentiation Between Logical and Physical Modeling
6. Introducing Ontologies and Taxonomies
7. Extended Business Key Collision Code Concept
8. Merging Satellites Without PIT Tables
9. Alignment with Modern Industry Terminology
10. Unlock the Full Potential with Instructor-Led Training
Final Thoughts
Watch the Video

1. A Major Expansion in Content and Learning Resources

One of the most visible improvements in Data Vault 2.1 is the expansion of the official training content. The separate CDVP2.1 course and certification now includes extensive video material featuring Dan Linstedt himself, who explains and demonstrates key Data Vault principles in depth.

Participants can now spend several hours watching recorded theoretical sessions and hands-on demonstrations. The new format combines the benefits of self-paced learning with the engagement of instructor-led sessions. You can also download the official SQL loading patterns for all Data Vault entities from the Data Vault Alliance (DVA) training portal.

Another highlight is access to the Data Vault Alliance community — a global network of Data Vault practitioners where members exchange best practices, discuss implementations, and share insights from real-world projects.

2. Enhanced Instructor-Led Training Experience

The well-known three-day instructor-led training remains a cornerstone of the certification path, but it has been optimized to deliver even more value. Trainers now dedicate more time to practical case studies, group discussions, and collaborative modeling workshops.

Instead of spending large portions of class time on theory, participants focus on applying concepts to real-world data challenges. Trainers provide direct feedback on Data Vault models, encourage peer review, and help attendees explore different architectural scenarios.

This redesign creates a more interactive, productive learning experience — especially valuable for consultants, data architects, and engineers who want to strengthen their practical Data Vault expertise.

3. Better Preparation for Certification

Preparing for the official Certified Data Vault 2.1 Practitioner (CDVP2.1) exam is now easier and more structured. The course includes integrated live quizzes during training sessions, allowing participants to test their understanding and interact directly with the instructor.

In addition, a practice exam has been introduced to help you assess your readiness before attempting the final certification. This makes it easier to identify knowledge gaps and feel confident on exam day.

4. Dealing with JSON and Semi-Structured Data

One of the most exciting changes in Data Vault 2.1 is the new module on handling JSON data and other semi-structured sources. As modern data platforms increasingly deal with variable data structures, the methodology now provides clear guidance for integrating such data efficiently.

The course introduces a set of rules and best practices for balancing performance, flexibility, and complexity. You’ll learn when to apply a schema-on-read approach instead of schema-on-write, how to maintain stability as source structures evolve, and how to preserve governance and traceability in semi-structured environments.

Dan Linstedt often refers to this as the “JSON Dilemma” — the challenge of maximizing flexibility without sacrificing performance or clarity. Data Vault 2.1 equips you with the methodology and patterns to solve that dilemma effectively.

5. Stronger Differentiation Between Logical and Physical Modeling

Another core enhancement in Data Vault 2.1 is the clearer separation between logical and physical modeling. While Data Vault 2.0 touched on this concept, version 2.1 makes it explicit: the logical model represents the business concept, while the physical model depends on the underlying technology and performance needs.

For example, on some platforms normalization works best, while on others (such as document-oriented databases), denormalization might be more efficient. The physical implementation should adapt to these realities — but the logical model remains consistent as the blueprint for the business layer.

This separation provides greater flexibility to evolve with technology without compromising the integrity of the business model. It also helps teams align architecture decisions with specific database or cloud platform requirements.

6. Introducing Ontologies and Taxonomies

In line with the growing emphasis on semantic data integration, Data Vault 2.1 introduces the use of ontologies and taxonomies as essential tools for business modeling. These concepts allow organizations to connect business terms, hierarchies, and relationships in a way that supports consistent data integration across departments and systems.

By embedding ontologies and taxonomies into the modeling process, organizations can improve data understanding, reduce ambiguity, and strengthen the link between data structures and business meaning.

7. Extended Business Key Collision Code Concept

The Business Key Collision Code concept has been extended in Data Vault 2.1 to better support cross-system integration. This improvement helps resolve conflicts that arise when business keys overlap or differ across systems — a common challenge in enterprise data integration.

With enhanced rules and examples, the training now guides you through best practices for identifying, classifying, and merging business keys, ensuring a consistent, high-quality data foundation.

8. Merging Satellites Without PIT Tables

Data Vault 2.1 introduces new approaches for handling historical data when traditional Point-In-Time (PIT) tables or snapshot techniques are not required. In cases where you need to maintain very long data histories or join multiple satellites describing the same business object, version 2.1 outlines methods for merging satellites without relying on PIT tables.

This allows for greater flexibility in data retrieval strategies and helps optimize performance in long-term historical scenarios.

9. Alignment with Modern Industry Terminology

To stay relevant with the evolving data landscape, Data Vault 2.1 integrates current industry concepts such as Data Mesh, Data Fabric, and Data Lakehouse. These paradigms are mapped to the Data Vault framework, demonstrating how the methodology fits within modern data architectures.

This changes ensures that Data Vault practitioners can easily connect the methodology to the broader trends and technologies shaping the data industry today.

10. Unlock the Full Potential with Instructor-Led Training

If you’re ready to deepen your knowledge and apply these changes in practice, the instructor-led Data Vault 2.1 training offered by Scalefree is the next step. This hands-on training combines theoretical knowledge, real-world exercises, and guided discussions to help you implement Data Vault successfully in your organization.

Visit the training page to find more information, view upcoming training dates, and begin your journey toward CDVP2.1 certification.

Final Thoughts

Data Vault 2.1 represents a significant step forward for data professionals seeking a future-proof methodology. With improved training content, better integration of semi-structured data, a sharper focus on modeling concepts, and alignment with modern architectural trends, Data Vault continues to be a robust choice for building scalable, flexible, and business-aligned data warehouses.

Professionals who previously completed CDVP2.0 may choose to complete the separate CDVP2.1 course and certification, that provides the tools, knowledge, and community support to take your data architecture to the next level.

Watch the Video

Marc Winkelmann In Data Vault Friday

Data Vault on dbt Snapshots

In recent years, dbt has become one of the most popular tools in modern data stacks. At the same time, Data Vault continues to be a proven methodology for building scalable, auditable, and historically complete data warehouses.

It is therefore no surprise that questions arise at the intersection of both worlds. One question we recently received perfectly captures this:

“Can you build a Data Vault view downstream off of dbt snapshots?
I feel dbt snapshots are safer because they capture data ‘as is’, and a Data Vault might be designed wrong.”

This is a great question—and one that touches architecture, performance, data modeling, and risk management at the same time. In this article, we’ll unpack the topic step by step and give a clear, practical answer.

In this article:

First Things First: What Are dbt Snapshots?
dbt Snapshots vs. Data Vault Satellites
- Insert-Only vs. Update-Based Modeling
Why Insert-Only Matters in the Cloud
Where dbt Snapshots Shine
Back to the Core Question
The Real Challenge: Performance and Cost
Does a “Wrong” Data Vault Design Mean Data Loss?
A Pragmatic Recommendation
Final Thoughts
Watch the Video

First Things First: What Are dbt Snapshots?

Before we compare dbt snapshots with Data Vault concepts, let’s align on what dbt snapshots actually are.

According to dbt’s own documentation, snapshots are used to implement Type 2 Slowly Changing Dimensions (SCD Type 2) on mutable source tables.

If you’re familiar with dimensional modeling, this should sound very familiar. SCD Type 2 means:

Whenever a record changes, a new row is inserted.
The old version of the record is kept for historical analysis.
Validity timestamps define from when to when a record version was valid.

In a typical example, a source table might only store the current state of an order:

January 1st: Order status = pending
January 2nd: Order status = shipped

The source system overwrites the status, so you only ever see the latest value. But in analytics and data warehousing, we usually want to know how the data looked at a specific point in time.

That’s where dbt snapshots come in. They store multiple versions of the same business key and enrich the data with technical columns such as:

dbt_valid_from
dbt_valid_to

Whenever dbt detects a change, it:

Inserts a new row for the new version.
Updates the dbt_valid_to of the previous version.

From a functional perspective, this is classic SCD Type 2 behavior.

dbt Snapshots vs. Data Vault Satellites

Now let’s compare dbt snapshots with Data Vault modeling. This is where things get interesting.

In Data Vault, Satellites are responsible for storing descriptive attributes and tracking changes over time. In other words:

Satellites are also SCD Type 2 structures.
They store full history.
They insert a new row for every detected change.

So at first glance, dbt snapshots and Data Vault satellites look almost identical. And conceptually, they are very close.

However, there is one important difference.

Insert-Only vs. Update-Based Modeling

Modern Data Vault implementations follow a strict insert-only approach. That means:

No updates to existing records.
No physical valid_to column updates.
History is reconstructed using window functions or Point-in-Time (PIT) tables.

dbt snapshots, on the other hand, do update the previous record to set the dbt_valid_to timestamp.

From a pure modeling perspective, both approaches are valid. But from a platform and performance perspective, insert-only has some strong advantages—
especially in cloud data warehouses like Snowflake, BigQuery, or Redshift.

Why Insert-Only Matters in the Cloud

Cloud-native data warehouses are optimized for append-heavy workloads.

For example:

Snowflake uses micro-partitions that are immutable.
Updates often result in copy-on-write operations.
Insert-only workloads scale better and are cheaper.

This is one of the reasons why Data Vault adopted insert-only patterns years ago. It’s not just about modeling philosophy—it’s about performance and scalability.

That doesn’t mean dbt snapshots are “wrong”. It just means they were designed with a slightly different use case in mind.

Where dbt Snapshots Shine

From a practical standpoint, dbt snapshots are extremely useful in specific scenarios.

One very common use case is a persistent staging area.

Imagine you receive:

Full data extracts every day from a source system.
No CDC (Change Data Capture).
Large tables where storing daily full loads would be wasteful.

In this case, dbt snapshots allow you to:

Store only the changes between loads.
Keep historical versions.
Reduce storage and processing overhead.

From this perspective, dbt snapshots act like a slim persistent staging layer. They capture the source data “as is” and preserve history.

If you already receive proper CDC data from upstream systems, then dbt snapshots are often unnecessary. The change tracking has already been done for you.

Back to the Core Question

So let’s return to the original question:

Can you build a Data Vault view downstream of dbt snapshots?

Technically and conceptually, the answer is:

Yes, you can.

If your dbt snapshots contain all source changes, you have everything you need to:

Identify business keys.
Track attribute changes.
Build hubs, links, and satellites.

In theory, you could build a fully virtualized Data Vault layer on top of snapshots:

Virtual hubs
Virtual links
Virtual satellites

From a data completeness perspective, nothing is missing.

The Real Challenge: Performance and Cost

Unfortunately, theory and reality often diverge.

While a fully virtualized Data Vault sounds elegant, it usually doesn’t work well in practice—at least not today.

Why?

Large historical datasets require heavy joins and window functions.
Virtualization pushes computation to query time.
Cloud compute costs increase rapidly.

In most real-world environments, fully virtualizing the Data Vault on top of snapshots leads to:

Slow queries
High compute bills
Poor user experience

That’s why most architectures still materialize the Data Vault at some point.

Does a “Wrong” Data Vault Design Mean Data Loss?

Another concern in the question is the fear of designing the Data Vault “wrong”.

This fear is understandable—but largely unfounded.

One of the core promises of Data Vault is:

You do not lose data due to modeling decisions.

Even if:

You split satellites too much.
You group attributes differently than you would today.
You later realize a better modeling pattern.

You can always:

Refactor satellites.
Split or merge them.
Reload data from existing Data Vault tables.

This is possible because Data Vault stores raw, historized data—not business logic.

So while a persistent staging area can be helpful, it is not a safety net you absolutely must have. A properly loaded Data Vault already is that safety net.

A Pragmatic Recommendation

So what does a pragmatic architecture look like today?

Use dbt snapshots if you need a persistent staging layer and don’t have CDC.
Materialize the Raw Data Vault for performance and scalability.
Virtualize downstream layers (Business Vault, Information Marts) where possible.

This approach balances:

Data safety
Performance
Cost efficiency

As data volumes grow and histories span years or decades, full virtualization simply becomes inefficient. Materialization at the Raw Vault level is still the sweet spot in most projects.

Final Thoughts

dbt snapshots are a powerful feature and fit nicely into modern data stacks. They can absolutely support Data Vault architectures—especially as a persistent staging layer.

However, they don’t eliminate the need for a materialized Data Vault. Nor do they replace the robustness and flexibility that Data Vault modeling provides.

Used together, dbt and Data Vault can form a strong, future-proof foundation for enterprise analytics—when each tool is applied where it makes the most sense.

Watch the Video

Marc Winkelmann In Agile Data

Agile Development in Data Warehousing with Data Vault 2.0

Agile Development in Data Warehousing: Initial Situation

Agile methodologies bring flexibility and adaptability to data warehousing, making them a natural fit for modern approaches like Data Vault 2.0. A common issue in data warehousing projects is that a scope is often missing and many of the processes such as controlled access, GDPR handling, auditability, documentation and infrastructure are not optimized. Additionally, data warehouse projects that have a scope often begin without a real focus on business value. This is mostly due to the fact that the use cases are not clearly communicated and the data architects do not know where to start. The consequence of this means no business value can be delivered.

In this article:

Data Vault 2.0 Methodology
The Scope of a Sprint
Define the project
Agile Development
Agile Development Review
Conclusion

Data Vault 2.0 Methodology

It is often assumed that Data Vault 2.0 is only a modeling technique, but this is not correct. Data Vault 2.0 includes the modeling technique, a reference architecture and the methodology. The methodology introduces project management tools such as CMMI, Six Sigma and Scrum to solve the problems described. While CMMI and Six Sigma deal with general project management issues, Scrum is mostly used specifically in the development team and provides the framework for a continuously improving development process. The use of agile development in Data Vault 2.0 projects will be described in more detail below.

The Scope of a Sprint

The first step in setting up a data warehouse project in an agile way is defining the objective of the project with just one or two pages. Unlike waterfall projects, the goal is to produce working pieces of usable outputs, could be reports or dashboards, in continuous iterations, otherwise called sprints. This means that we don’t need to plan the entire project in detail but instead can build around a general idea or goal for the final data warehouse before then focusing on planning the first sprints. In order to address the aforementioned problems, the focus of the sprints needs to be centered around business value. For this reason, it is important to receive constant feedback from the business users for a continuous improvement process.

Define the project

Both the scope of a sprint and the architecture follow a business value driven approach built vertically and not horizontally. This means they are not built layer by layer but instead feature by feature. A common approach for this is the Tracer Bullet approach. Based on business value, which is defined by a report, a dashboard or an information mart, the source data will be identified and modeled through all layers and loaded.

As shown in Figure 1, the entire staging area layer is not initially built but rather a small part of the respective layer is built based on data in the scope, in this case the SalesReport.

Agile Development

Before new functionality can be implemented in a sprint, it needs to be defined.
This task lies with the product owner as they are the ones to write and prioritize user stories.
As already explained, the goal of a sprint is to produce working pieces of usable outputs called features.
In addition, there are tech topics that need to be considered. There are various methods to support Sprint Planning, such as planning poker or Function Point Analysis, which are discussed in more detail in another article.

Another good indicator is to evaluate the sprint itself while the sprint is still ongoing. If the development team does not manage to implement a feature in a sprint, this can often be seen as a good indicator that the scope is too large.

To avoid this, all work packages that are not relevant for the feature should be removed. Though, what is often the case these work packages are not completely removed out of fear from the business user.

To address this fear it is important to educate the business user that they will be delivered but only in a later sprint and temporarily moved into the backlog.

Figure1 : Data Vault 2.0 Architecture

Due to the flexible and scalable Data Vault model, these layers can be extended with the next feature with little to no re-engineering. This is possible due to the fact Data Vault consists of a Raw Data Vault and a Business Vault model which means it contains the logical architecture as well as the data modeling perspective. The Raw Data Vault is modeled in a data-driven way by integrating the data by business keys. Only hard business rules like data type conversions or hash key calculations are applied. All other soft business rules are only applied in the Business Vault.

Here, we turn data into information. For this reason, the Raw Data Vault requires less refactoring and can be extended limitlessly.

Agile Development Review

Another important success factor for agile projects is proper review and improvement. Even before the next sprint starts, two meetings must be held by the team:

The sprint review meeting: This meeting is about reviewing the delivered features. Usually the development team, the product owner, the Scrum Master and the end-users participate in this meeting.
Retrospective meeting: This meeting usually takes place directly after the review meeting and focuses on identifying activities that need to be improved.
Backlog refinement for prioritizing the user stories and to make sure that the team understands what to do
Sprint planning to plan which user stories fit into the next sprint based on estimating the effort.

It is important that these meetings are held so that toe source errors can be found. In this way, the outcome of a project can be improved and the development processes optimized in an iterative way.

Conclusion

Data Vault 2.0 is not only a scalable and flexible modeling technique, but a complete methodology to accomplish enterprise vision in Data Warehousing and Information Delivery by following an agile approach and focusing on business value. By using agile methods in data warehousing, the focus in projects can be on the business value and delivering useful products to the customer.

Marc Winkelmann In Data Vault Friday

Differences between Data Vault 2.0 and Data Vault 2.1

Data Vault 2.0 vs Data Vault 2.1

As organizations continue to grapple with rapidly evolving data landscapes, Data Vault remains a leading methodology for building scalable, auditable, and flexible data warehouses. With the release of Data Vault 2.1, practitioners and architects often ask: “What’s changed since 2.0?” In this article, we’ll dive into the differences across three core areas—design principles, ETL patterns, and modeling best practices—and show you how 2.1 enhances your ability to tackle modern data challenges like data lakehouses, data mesh, and nested JSON feeds.

In this article:

1. Design Principles: Staying True but Embracing Modern Architectures
2. ETL Patterns: From Batch to Streaming and JSON
3. Modeling Best Practices: Updated Patterns for a Distributed World
4. Educational & Organizational Enhancements
- Rich Video & Quiz Content
- Certification & Community
Choosing When to Adopt 2.1
Conclusion
Watch the Video
Meet the Speaker

1. Design Principles: Staying True but Embracing Modern Architectures

Core Continuity

At its heart, Data Vault 2.1 retains all the foundational tenets of 2.0: separation of concerns (Hubs, Links, Satellites), immutable history, and decoupling of raw data capture from business transformations. If you already have a robust 2.0 implementation, there’s no need for a forklift upgrade—2.1 is evolutionary, not revolutionary.

Lakehouses, Mesh, and Fabric

Where Data Vault 2.1 shines is in explicitly addressing emerging architectures. You’ll find guidance on integrating Vaults within data lakehouses (e.g., Delta Lake, Apache Iceberg), as well as how Vault concepts align with data mesh domains and data fabric overlays. Instead of an “Enterprise Data Warehouse” monolith, 2.1 helps you embed Vault patterns into cloud-native, distributed environments.

Logical vs. Physical Modeling

With the proliferation of diverse storage engines—relational, columnar, NoSQL document stores, and graph databases—2.1 distinguishes your logical Vault model (Hubs, Links, Satellites) from its physical implementation. You now have clear guidelines on:

Keeping the logical model technology-agnostic
Adapting physical denormalization or document embedding strategies per platform capabilities
Optimizing storage formats (e.g., Parquet, Delta, or JSONB) while preserving auditability

This separation equips data engineers to leverage the strengths of their chosen database without sacrificing Vault integrity.

2. ETL Patterns: From Batch to Streaming and JSON

Expanded CDC Strategies

Data Vault 2.1 deepens its coverage of Change Data Capture (CDC) patterns. You’ll find refined techniques for:

Transactional order guarantees: Ensuring raw Vault loads adhere to source system timestamps to preserve lineage.
Handling late-arriving or out-of-order events: Techniques to backfill or correct Satellites without breaking immutability.
Parallel loading: Avoiding cross-system dependencies by pre-joining keys within each source’s staging area.

Informal “Pre-Join” Denormalization

2.1 codifies the practice of pre-joining business keys in staging or external views—a pattern previously covered only in practitioner forums. This denormalization step enriches payload tables with true business keys upfront, eliminating repetitive lookups during Link loads and simplifying ETL script maintenance.

JSON and Nested Structures

Perhaps the most visible ETL addition is 2.1’s JSON processing module. With more sources emitting nested, semi-structured payloads, new patterns include:

Flatten-first loading: Initial extraction of atomic fields into raw Satellites before storing full payloads.
Schema evolution handling: Capturing structural changes (added arrays or nested objects) as metadata in Vault artifacts.
Selective shredding: Automating transformation of common sub-documents into separate Hubs/Links/Satellites.

3. Modeling Best Practices: Updated Patterns for a Distributed World

Managed Self-Service BI

Data Vault 2.1 recognizes the shift toward self-service analytics within federated teams. Best practices now recommend:

Role-based access controls at the raw & business Vault layers, ensuring data stewards can grant fine-grained permissions.
Row- and column-level security patterns that can be implemented natively in cloud warehouses (Snowflake masking policies, SQL Server RLS, etc.).
Embedding governance metadata in Vault tables, enabling automated lineage and impact analysis for downstream consumers.

Expanded Satellite Strategies

While 2.0 introduced Point-in-Time (PIT) and Bridge tables for performance, 2.1 adds:

Snapshot Satellites: Prebuilt structures for frequented combinations of Hubs & Satellites—ideal for dimensional views.
Behavioural Satellites: Grouping event-driven attributes (e.g., clickstreams) separately from master-data Satellites.

Cross-Domain Linkage

Data Vault 2.1 extends guidance on managing relationships across micro-warehouse domains—a nod to data mesh. It clarifies when to use:

Cross-domain Links: For relationships spanning autonomous teams with separate Hubs.
Reference Hubs: Capturing shared code lists (e.g., currency, country) that multiple domains consume.

4. Educational & Organizational Enhancements

Rich Video & Quiz Content

Training for 2.1 now includes extensive pre-recorded modules by Dan, focusing on conceptual foundations—freeing up live classroom time for interactive labs and advanced case studies. Over 40 quizzes interspersed throughout the curriculum reinforce learning and feed directly into certification exams.

Certification & Community

Becoming a Data Vault 2.1 certified practitioner involves:

5 days of combined video and onsite training (versus one day of video + three days live in 2.0).
An updated exam covering new ETL patterns, JSON handling, and modern architecture integration.
Access to an expanded Slack community and biweekly “Vault Clinics.”

Choosing When to Adopt 2.1

Given the backwards-compatible design, migration from 2.0 to 2.1 can be phased:

Retain existing Hub/Link/Satellite structures in the Raw Vault.
Gradually introduce new ETL patterns (JSON shredding, snapshot Satellites) in parallel.
Implement enhanced governance and self-service controls in the Business Vault.
Leverage certification resources to upskill architects and engineers on updated best practices.

Conclusion

Data Vault 2.1 advances the methodology by weaving in lessons from cloud-native architectures, self-service analytics, and semi-structured data sources—without discarding the proven foundation of 2.0. Whether you’re standardizing a data mesh deployment or optimizing your JSON pipelines, 2.1 provides the patterns and guardrails needed to build a modern, auditable, and flexible data platform.

Watch the Video

Meet the Speaker

Marc Winkelmann In Data Warehouse

10 Essential Skills Your Team Needs to Build an Analytical Data Platform

Build an Analytical Data Platform

Building a modern analytical data platform is more than just choosing the right database or ETL tool. It requires a blend of business insight, data expertise, architecture design, and automation savvy. In this article, we’ll explore ten essential skills your team needs to design, develop, and maintain a robust, scalable, and high-value data platform.

In this article:

1. Business Understanding
2. Objective Setting & ROI Focus
3. Data Understanding & Modeling
4. Data Acquisition Techniques
5. Structured Architecture: The Medallion Approach
6. Data Integration & Modeling in the Silver Layer
7. Temporality & Historical Tracking
8. Code Generation & Automation Tools
9. Agile Development & Traceability
10. DevOps & Cost Management
Watch the Video
Conclusion

1. Business Understanding

Before diving into any technical work, your team must understand the business domain and the data itself. This doesn’t mean every engineer needs to be a data analyst, but they should know:

Which source systems hold the data (CRM, ERP, marketing platforms, etc.)
Key business objects (customers, contracts, opportunities) and how they relate
Business processes behind the data, like a customer’s lifecycle or sales funnel

By grounding the team in real-world outcomes—such as improving customer retention or reducing churn—engineers stay focused on delivering measurable ROI.

2. Objective Setting & ROI Focus

Clear objectives guide every stage of your platform’s development. Whether your goal is to accelerate financial reporting or enable real-time marketing analytics, defining the desired outcomes:

Helps prioritize features and data sources
Aligns stakeholders around common metrics
Boosts motivation by tying work to tangible business value

Teams that regularly track ROI milestones can adjust scope and resources proactively, ensuring the platform grows in step with organizational goals.

3. Data Understanding & Modeling

A deep dive into your source systems reveals hundreds—even thousands—of tables. Your engineers need to know:

Primary and foreign keys connecting entities
Relationship cardinalities (one-to-one, one-to-many, many-to-many)
Data quality quirks and domain-specific rules

This understanding informs the modeling approach—be it third normal form, star schemas, or Data Vault—ensuring downstream analytics are consistent and reliable.

4. Data Acquisition Techniques

Extracting data from source systems can take many forms:

Full daily extracts via CSV or JSON files
API calls for near-real-time data feeds
Change Data Capture (CDC) for incremental updates

Knowing when to use each approach minimizes data latency, reduces load times, and optimizes storage. CDC, in particular, slashes the volume of data transferred, but requires robust handling to maintain consistency.

5. Structured Architecture: The Medallion Approach

Dumping raw data into a single database is a recipe for chaos. Instead, adopt a layered “medallion” architecture:

Bronze Layer (Staging/Landing): Raw data as ingested
Silver Layer (Cleansed, Integrated): Unified and harmonized data across systems
Gold Layer (Presentation): Curated tables/views for business users and BI tools

Medallion Architecture in an analytical data platform

This separation of concerns simplifies debugging, improves performance, and clarifies responsibilities for each team member.

6. Data Integration & Modeling in the Silver Layer

The silver layer is where the “magic” happens:

Integrating disparate systems into a unified view
Applying your chosen modeling technique (e.g., star schema, Data Vault)
Ensuring referential integrity and consistent business definitions

Investing in a proven modeling framework not only scales with additional data sources but also enables automation and accelerates the onboarding of new subject areas.

7. Temporality & Historical Tracking

Beyond technical timestamps (extract load times), your data has business timelines:

Contract start/end dates
Customer sign-up and churn events
Promotion or campaign effective periods

Implementing snapshot tables, slowly changing dimensions, or time-aware modeling ensures accurate trend analysis, historical comparisons, and auditability.

8. Code Generation & Automation Tools

Hand-coding every pipeline is time-consuming and error-prone. Leverage tools that:

Automatically generate ETL/ELT code based on templates
Orchestrate complex workflows and dependencies
Enforce consistency through standard patterns and conventions

Automation not only speeds up development but also improves data quality by reducing manual interventions.

9. Agile Development & Traceability

Adopting an agile mindset means delivering small, working increments quickly. Apply traceability by:

Defining clear targets (e.g., monthly revenue report)
Mapping those targets back to specific source tables
Focusing on data that directly supports your objectives

This approach prevents “scope creep” and ensures that every pipeline built serves an immediate analytical need.

10. DevOps & Cost Management

Once pipelines are automated, you need:

Orchestration frameworks (e.g., Airflow, Dagster) to schedule and monitor jobs
CI/CD for data code, including version control and automated testing
Cost monitoring tools to track cloud resource usage and optimize performance

Effective DevOps practices guarantee reliability, while cost-awareness keeps your platform sustainable in the cloud era.

Data Profiling: DataHub continuously monitors and analyzes data quality, automatically generating profiling metrics that reveal data distributions, identify anomalies, and help maintain high data quality standards. It provides key statistics such as row and column counts, query frequency, top users, and last update timestamps, along with detailed attribute profiling, including value ranges, central tendencies, null and distinct values. The table below shows some examples of these profiling metrics.

Watch the Video

Conclusion

Building an analytical data platform is a multifaceted endeavor. By equipping your team with these ten skills—spanning business understanding, data modeling, architecture design, automation, and DevOps—you’ll lay the foundation for a platform that delivers consistent insights, scales gracefully, and drives real business value.

Modeling Dynamic Key-Value Pairs in Data Vault

Understanding Multi-Layered Multi-Activity

Primary Key Design for Multi-Active Key-Value Satellites

Evaluating Satellite Splitting vs. Unified Storage

Raw Data Vault Loading Patterns for Multi-Active Data

Strategy 1: Full Set Dump Ingestion (Snapshot Loading)

Strategy 2: Group Hash Diff Delta Detection

Downstream Integration and Information Mart Query Patterns

Business Vault Pivoting

Simplifying Point-in-Time (PIT) Architectures

Establishing a Sustainable Data Vault Architecture

Watch the Video

Architectural Extension Patterns: The Pre-Staging Layer

Managing Referentially Intact Subsets in Non-Production

Modern Branching and Deployment Strategies for Data Vault

Adopting GitHub Flow and Trunk-Based Development

Zero-Copy Cloning and Dynamic PR Environments

Advanced DBT Features: Deferral and State Tracking

1. DBT State Tracking (--state)

2. DBT Model Deferral (--defer)

Summary Comparison of Filtering and Deployment Strategies

Watch the Video

How Data Vault Supports AI and ML Readiness

Where Most Companies Are Right Now

Reasons That Stop Companies From Scaling with AI

Why AI Workflows Fail Without a Data Foundation

The Enabling Data Platform: What It Looks Like

How This Architecture Directly Solves the AI Scaling Problem

Data Vault 2 as the Foundation for AI Readiness

Starting the Journey: Three Things to Do in Parallel

Want to Go Deeper?

Data Vault and Data Mesh

Data Vault, Data Mesh, and Data Fabric Are Not Competing

Why Data Vault Is the Foundation Data Mesh Needs

The Problem With Going “Full Data Mesh”

How This Architecture Works in Practice

What Makes This Combination Work

Data Catalogs and Governance as Connective Tissue

Starting the Journey

Handling Contradicting Facts About a Single Business Key in Data Vault

Scenario 1: Resolving Multi-Source Contradictions via Source Prioritization

Raw Data Vault Satellites: Single Source per Satellite

Business Vault Alignment and Point-in-Time (PIT) Tables

Formulating the Golden Record using COALESCE Logic

Scenario 2: Handling High-Frequency Change Data Capture (CDC) Batches

The Pitfall of Primary Key Violations

Owning the Load Date Timestamp vs. Source Dates

Resolving Sequential Order with Technical Sequence Numbers

Architectural Integrity in Enterprise Data Platforms

Watch the Video

How Long Does Data Vault Certification Take?

The Full Timeline at a Glance

Phase 1 — Pre-Course Self-Paced Videos (Approximately 15 Hours)

Phase 2 — Live Instructor-Led Training (3 Days)

Phase 3 — Exam Window (8 Weeks, 2 Attempts Included)

Phase 4 — Post-Certification Platform Access (6 Months)

Complete Timeline Summary

Is There a Faster Path?

Who Is the Certification For?

Next Training Dates

Late Arriving Data

1. Technical Timeline vs Business Timeline

2. Capturing the Business Timeline

3. Timeline Corrections Without an Extended Tracking Satellite

4. Practical Guidelines

Summary

Watch the Video

What’s New in the Data Vault 2.1 Training

1. A Major Expansion in Content and Learning Resources

2. Enhanced Instructor-Led Training Experience

3. Better Preparation for Certification

4. Dealing with JSON and Semi-Structured Data

5. Stronger Differentiation Between Logical and Physical Modeling

6. Introducing Ontologies and Taxonomies

7. Extended Business Key Collision Code Concept

8. Merging Satellites Without PIT Tables

9. Alignment with Modern Industry Terminology

10. Unlock the Full Potential with Instructor-Led Training

Final Thoughts

1. DBT State Tracking (`--state`)

2. DBT Model Deferral (`--defer`)