Skip to main content
search
0
CartClose Cart
In Data Vault Friday

Modeling Invoices in Data Vault

Watch the Video

As part of our ongoing Data Vault Friday series, our CEO, Michael Olschimke, engages with a relevant and practical question from our audience about Data Vault modeling.

“What are the best practices for modeling Data Vault table structure to store invoice data?”

In this concise yet informative video, Michael shares valuable insights into the best practices for designing Data Vault table structures specifically tailored for storing invoice data. Recognizing the importance of effectively modeling this type of data, Michael addresses key considerations, and potential challenges, and recommends optimal approaches to ensure a robust and scalable solution.

For those seeking guidance on structuring Data Vault tables for invoice data, this video serves as a quick and insightful resource.

In Data Vault Friday

Real-Time Loading of CDC Packages in Data Vault – PART 2

Watch the Video

As part of our ongoing Data Vault Friday series, our CEO Michael Olschimke delves into a pertinent question posed by a member of our audience.

“I would be interested in some ideas about how to load data from Apache Kafka. In our case, we receive CDC data from DB servers over Apache Kafka.

One specific concern raised is about maintaining the correct sequence of data in Raw Vault when dealing with different partitions in a Kafka topic. This becomes particularly crucial in scenarios involving Change Data Capture (CDC) from database servers.”

In this enlightening video, Michael provides insightful ideas and strategies for effectively loading data from Apache Kafka while ensuring the integrity of the sequence in the Raw Vault. He tackles the nuances of handling different partitions within a Kafka topic, offering practical guidance to address challenges associated with maintaining data order.

For those navigating the intricacies of data loading from Apache Kafka, this video provides valuable insights and solutions.

In Beginner

Data Warehouse Automation – Build or Buy?

Watch the Webinar

Data Warehouse Automation – Build or Buy?

In this webinar, we take a sneak peek into one of the hot topics of modern data warehousing, namely Data Warehouse Automation. We would break down the basics of DW automation & how it has brought about a cultural shift in the realm of a modern Data Warehouse & its architecture.

In this regard, we will also touch upon the often-asked question “Build or buy” along with sharing our experience working with several customers who have benefited immensely from automation and the key lessons we have learned as part of our overall DW automation journey.

This webinar is for anyone who loves Data!

Watch Webinar Recording

Webinar Agenda

1. Understanding Data Warehouse Automation
2. Drivers for Decision ING
3. Automation in Data Vault
4. Anti-Patterns in DV Automation
5. Best Practices

In Data Vault Friday

Logical Industry Models in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke addresses a thought-provoking question raised by a member of our audience.

“In our data warehouse architecture, we have integrated a Data Vault 2.0 approach with a logical industry model: once the raw data has been loaded into the Raw Data Vault, the data undergoes a transformation into the Business Vault. This follows the logical design, maintaining the Data Vault style but derived from the industry model. Subsequently, it is further transformed into the business access layer, forming an information mart.

However, despite the initial intent of adopting a best-of-breed approach, there’s a realization that somewhere along the way, agility was compromised.”

In this enlightening video, Michael delves into the challenges faced in a data warehouse architecture that combines a Data Vault 2.0 approach with a logical industry model. Specifically, he addresses the placement of logical vendor models within this framework, exploring ways to maintain agility in the process.

In Data Vault Friday

Real-Time Loading of CDC Packages in Data Vault – PART 1

Watch the Video

In the latest installment of our Data Vault Friday series, our CEO Michael Olschimke addresses a pertinent question posed by an audience member.

“I would be interested in some ideas about how to load data from Apache Kafka. In our case, we receive CDC data from DB servers over Apache Kafka.

Should the data be converted from AVRO/JSON format to database format in Staging / Raw Vault? Or should it be loaded directly in an unchanged format? What is the Best Practice here?”

In this insightful video, Michael provides practical guidance on loading data from Apache Kafka, specifically when dealing with CDC (Change Data Capture) information from database servers. He explores the options of converting data from AVRO/JSON format to a database format within Staging/Raw Vault versus loading it directly in its original unchanged format.

For those navigating the complexities of data loading from Apache Kafka, this video offers valuable insights and best practices to inform decision-making in data architecture.

In Data Tools

Data Vault 2.0 with DBT – Part 1

Data Vault 2.0 with dbt

This article focuses on the benefits of using Data Vault 2.0 with dbt, and the importance of choosing the correct implementation tools. Data is an important asset in the decision-making process. As we have previously discussed in another post, Data Vault 2.0 is the right choice when the goal of an enterprise data warehouse is to have fully historized and integrated data. Additionally, it is also better suited to instances in which data from many source systems needs to be combined. You can find the previous blog post here.

While Data Vault 2.0 focuses on the “what”, there are many options for the “how” of technically translating a Data Vault model into physical tables and views in the enterprise data warehouse, as well as for orchestrating and loading/processing the procedures. And this is where Data Vault 2.0 with dbt creates an effective solution.

In this article:

About dbt

This data build tool transforms your data directly into your data warehouse. For reference, dbt is the “T” in ELT. Therefore, dbt assumes that data is already loaded into a database from which the current database can query. In contrast, ETL extracts data then transforms it and before loading it into the target. With ELT, the data is not transformed as it moves into the data warehouse. (with transform we mean transformation by soft business rules which would change the meaning of the data. Of course, we have to make sure that the data fits into the target table (data type adjustments, etc.). Here we are talking about “hard rules”.)
Dbt is particularly compatible and useful in cloud DWH solutions such as Snowflake, Azure Synapse Analytics, BigQuery, and Redshift and performs transformations and modeling directly on the database in order to take advantage of the performance of these highly scalable platforms.

How dbt works

Models and SQL statements can be easily created, tested, and managed in dbt itself. A powerful combination of the scripting language, Jinja2, and the all-time classic SQL allows users to build models. The simple interface enables data analysts without engineering know-how to initiate appropriate transformations. Data team workflows become more efficient and cost-effective because of this. Behind this tool sits an open-source community that is constantly and passionately developing the tool. As such, dbt is available both as a free, reduced core version and as a comprehensive and flexible cloud version.

How does Data Vault 2.0 with dbt work?

Part of the Data Vault 2.0 methodology is the model that focuses on how to design the core data warehouse with a scalable solution. The core Data Vault entities are hubs, links ,and satellites. That said, using Data Vault 2.0 with dbt provides the ability to generate Data Vault models and also allows you to write your data transformations using SQL and code-reusable macros powered by Jinja2 to run your data pipelines in a clean and efficient way.

Data Vault 2.0 with dbt

Conclusion

Dbt does not reinvent the wheel, but when it comes to building a new EDW – especially if it is in the clouds – it provides a very helpful basic framework with many important functions for Continuous Integration and Deployment already defined. dbt brings the standards of software development into the world of data transformation. This allows developers to concentrate on the core tasks of data modeling and business logic. Especially, not only for smaller projects, this tool offers a lightweight and extremely affordable alternative to other data warehouse automation solutions.

In Data Vault Friday

NULL Business Keys in Data Vault

Watch the Video

In our continuous Data Vault Friday series, our CEO, Michael Olschimke, takes a moment to delve into a thought-provoking question raised by our audience.

“Does the Business Key should be a not null column at the source?”

This succinct yet critical query is the focus of this brief but insightful video. Michael engages with the nuances of the business key, exploring whether it should be a mandatory, not-null column at the source. As he unpacks the considerations, the audience gains valuable insights into the implications and potential advantages of enforcing the not-null constraint on the business key.

For those seeking clarity on best practices surrounding business key management, this video provides concise guidance.

In Data Vault Friday

Managed Self-Service Industrialization in Data Vault

Watch the Video

As part of our continuing Data Vault Friday series, our CEO, Michael Olschimke, engages with a pertinent question posed by our audience.

“How does the industrialization work in Managed Self-Service BI?”

In this succinct yet informative video, Michael delves into the intricacies of the industrialization process within the realm of Managed Self-Service Business Intelligence (BI). The audience is treated to a valuable discussion on the methodologies and practices involved in streamlining and scaling BI processes for efficient and consistent outcomes.

Michael sheds light on the significance of industrialization in the context of Self-Service BI scenarios, providing insights that are relevant for both beginners and seasoned professionals in the field.

In Data Vault Friday

Sharding in Data Vault 2.0

Watch the Video

In our continuing Data Vault Friday series, our CEO, Michael Olschimke, engages with an intriguing question posed by our audience.

“How does sharding work in Data Vault 2.0?”

In this illuminating video, Michael takes us on a journey to explore the intricacies of sharding within the context of Data Vault 2.0. Delving into the technical aspects, he provides insights into the process of laying out data on a Massively Parallel Processing (MPP) cluster. Interestingly, Michael shares his expertise from the comfort of his personal MPP cluster located in his home basement, adding a unique and practical dimension to the discussion.

For those seeking a deeper understanding of sharding techniques and their implementation in Data Vault 2.0, this video serves as a valuable resource.

In Intermediate

Utilizing Potentials of Data Vault 2.0 – Overcoming Bad Practices – Part 1

Watch the Webinar

Utilizing Potentials of Data Vault 2.0 – Overcoming Bad Practices – Part 2

What are common mistakes when applying Data Vault 2.0 in enterprise data warehouse projects? Do you have questions regarding Data Vault modeling  and the realization of GDPR causes you great difficulties or is your project stuck because you are delivering no business value?

This webinar describes common Anti-patterns of Data Vault 2.0, their consequences, and the solution to eliminate them from your current or in your future projects.

Tune in and learn more to avoid Data Vault 2.0 bad practices and apply simple solutions.

Watch Webinar Recording

In this article:

Webinar Agenda

1. How to use Data Vault 2.0 for modeling business information
2. How to avoid the pitfalls of being unable to deliver business value
3. How to mask Business Keys from Hubs for privacy

In Data Vault Friday

Aggregate Measures Using Bridge Tables in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our knowledgeable trainer, Marc Finger, dives into a pertinent question raised by our audience regarding aggregate measures in Data Vault:

“How and where to aggregate measures from transactional system on a business date considering late arriving messages?”

In this informative video, Marc addresses the intricacies of aggregating measure values within bridge tables. The audience is treated to a comprehensive discussion on the techniques and considerations involved in efficiently handling aggregate measures in the context of bridge tables.

Marc provides valuable insights and practical tips, offering clarity on how to navigate the challenges associated with aggregating measures while utilizing bridge tables in your data architecture.

In Data Vault Friday

Lambda Architecture vs. Data Vault 2.0 Architecture

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke engages with a question that delves into the comparison between the Data Vault 2.0 architecture and the Lambda architecture for real-time systems.

“We are currently comparing the Data Vault 2.0 architecture with the Lambda architecture for real-time systems. Can you elaborate on the similarities and differences?”

In this enlightening video, Michael provides a comprehensive exploration of the distinctions and commonalities between the Lambda architecture and the Data Vault 2.0 architecture. The audience gains valuable insights into the considerations, strengths, and potential use cases of each approach, aiding in informed decision-making for real-time system implementations.

Close Menu