Most organizations don’t have a data problem. They have a metadata problem. The information about what data exists, where it comes from, who owns it, and whether it can be trusted is scattered across dozens of systems, and often across spreadsheets, chat threads, and people’s heads.
In our joint webinar with DataHub, Lorenz Kindling and Tim Schlitzer looked at what it takes to close that gap: a strong, auditable core in the data platform, and a context layer on top that makes technical metadata usable for the business.
From Metadata to Data Discovery: Building a Metadata Layer with Data Vault and DataHub
Modern data platforms generate valuable metadata across tools such as dbt, data warehouses, and orchestration systems, but this information often remains fragmented.
Join Scalefree and DataHub to explore how Data Vault provides a stable platform core, how data catalogs have evolved into broader metadata platforms, and how technical and business metadata can be connected to improve discovery, governance, and trust.
The session will conclude with a practical DataHub demo.
Four symptoms of fragmented metadata
Lorenz opened with the failure modes he sees repeatedly in data platform and governance projects:
- Discovery. Teams build genuinely good data products, and users never find out they exist, or how to get to them.
- Trust. Without visible ownership, quality results, and lineage, a technically excellent dashboard still gets ignored. Trust is lost quickly and earned back slowly.
- Usage. Plenty of data, and plenty of metadata, sits unused simply because nobody knows it’s there or what to do with it.
- Access. Discovery is only half the story. Users also need a clear path to request access, ideally with the data owner named right next to the asset.
From data warehouse to data platform
The architectural context has shifted, too. Lorenz traced the path from the classic centralized data warehouse, through the data lake era (early lakes, he noted, often turned into “data dumpsters” when built without intention), through hybrid architectures and lakehouses, to what we now simply call the data platform: the whole ecosystem around the data, including AI and governance.
That shift raises the bar on capabilities. A modern platform is expected to deliver standardization and automation, auditability, flexibility, scalability, cost control, security, regulatory compliance, and support for AI applications.
Data Vault covers a substantial share of that list by design, an insert-only architecture gives you auditability and the ability to reproduce a report you handed out months ago, and the methodology brings standardization, automation, flexibility, and scalability with it.
But Lorenz was clear about the limits. Full regulatory compliance usually needs the business side to supply context; an auditor wants more than SQL. AI applications need business context, not just a well-modeled core, and in his experience, projects that go badly are frequently the ones missing exactly that. Cost control needs to be centralized across the platform rather than solved inside the model.
His framing: a strong core is necessary but not sufficient. If your model isn’t auditable, there’s no point building governance on top of it. And if it is, you still need something to translate it for the business.
Build Better Data Platforms
Practical architecture insights for modern data teams. Join 8,000+ data professionals.
Get Free InsightsThe four generations of data catalogs
Tim placed today’s tooling on a maturity curve:
- No metadata management, knowledge lives in spreadsheets, Slack, and tribal memory; data teams spend most of their time hunting rather than building.
- Traditional catalogs, built for humans to search and browse. Useful, but inherently manual, and AI systems don’t click through a UI.
- AI data catalogs, automated, observable, machine-readable context for both humans and agents. Decentralized setups like data mesh or data products need automated governance to work at all.
- The context platform, real-time, accurate context for agentic workflows that make autonomous decisions.
Most customers he sees sit between stages two and three.
Where DataHub fits
DataHub is organized around three pillars:
- Discovery and lineage, “Ask DataHub” answers plain-language questions like which tables feed our recommendation model, backed by cross-platform, column-level lineage from source through every transformation to the final report. Auto-documentation (with a human in the loop) covers the thousands of tables nobody wants to describe by hand, and an MCP server plus Teams and Slack bots bring the context to where people and agents already work.
- Observability, freshness, volume, column, schema, and custom SQL checks; AI-assisted anomaly detection; impact analysis before you deploy a breaking change; and incident workflows that record how something was resolved.
- Governance, define criteria once (sensitive data tagged, ownership assigned, quality checks passing) and let assets get certified programmatically.
Tim’s live demo walked all of this from a business user’s perspective: asking a question in plain English, finding the owning team, seeing which Power BI dashboards already depend on the table, tracing a single column from PostgreSQL through Spark and Snowflake into a dbt model, running an impact analysis on a column change, and completing a customizable GDPR assessment as the data owner.
The context layer runs through the whole platform
Lorenz closed the loop on architecture. The catalog is no longer a box at the bottom of the diagram. It talks to source systems, the staging area, the lakehouse, the Data Vault, the information marts, and the BI and analytics tools, pulling metadata in and pushing context back out.
That includes the intelligence layer. AI systems need context to be useful, but not all of it: over-contextualizing burns tokens and money. The goal is the right context, plus access to the data.
The value, in other words, comes from translation. Technical context, lineage, transformations, tests, dependencies, runtime metadata, largely already exists in tools like dbt. A context layer turns it into definitions, ownership, business rules, policies, and an understandable data product.
