Skip to main content
search
0
CartClose Cart
In Data Vault Friday

Data Vault on Databricks

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke addresses a pertinent question raised by the audience, unraveling the discourse around the compatibility of Data Vault 2.0 (DV2.0) with Databricks.

“There has been hype going on on LinkedIn about whether or not DV2.0 is suited to exist on Databricks. Many people disagree that it is. The most significant comments are ‘lots of joins,’ ‘performance getting data out,’ and ‘not suited for modern automation.’ The latter ties to tools creating generated code per object VS. parameterized pipelines.”

In this illuminating video, Michael delves into the discussions surrounding the suitability of Data Vault 2.0 in the Databricks environment. He provides insights into the concerns raised, such as the perceived challenges related to joint operations, data retrieval performance, and the alignment with modern automation practices.

Michael offers a balanced perspective, exploring the nuances of utilizing DV2.0 on Databricks and addressing the key considerations raised in the LinkedIn discussions.

In Beginner, Salesforce

Warum Eigentlich Salesforce?

Watch the Webinar

Warum Eigentlich Salesforce?

In diesem Webinar geht es um die Frage “Warum eigentlich Salesforce?”. Wir werden uns die 3 Hauptvorteile des CRM-Systems von Salesforce ansehen: Integrierbarkeit, Erweiterbarkeit und Anpassbarkeit. Erfahren Sie, wie Salesforce Ihnen dabei helfen kann, Ihr Unternehmen zu transformieren und Ihre digitalen Prozesse zu optimieren.

Sie werden verstehen, wie Salesforce Ihre Arbeitsabläufe nahtlos integrieren und automatisieren kann, um Ihnen Zeit und Ressourcen zu sparen. Wir zeigen Ihnen, wie einfach und schnell es ist, Salesforce zu erweitern und anzupassen, um den spezifischen Bedürfnissen Ihres Unternehmens gerecht zu werden.

Melden Sie sich jetzt an und erfahren Sie, warum Salesforce die beste Wahl für die digitale Transformation Ihres Unternehmens ist.

Watch Webinar Recording

Webinar Agenda

1. CRM Systeme in der digitalen Transformation → Shared Customer Insight (nach Jeanne Ross)
2. Warum Salesforce so gut passt. (Salesforce order MS Dynamics in Spitzengruppe)
3. Grund 1 Integrierbarkeit
4. Grund 2 Erweiterbarkeit
5. Grund 3 Anpassbarkeit

In Data Vault Friday

Multi-temporal Source Data (Sap Hrms) in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke explores a valuable question from the audience, shedding light on the intricacies of modeling an SAP HRMS source with SCD type 2 data and dealing with time-dependent information in Data Vault 2.0.

“Could you please guide us on how to model an SAP HRMS Source that holds the data in SCD type 2 in the source itself with an effectivity start date and end date for each change? What will be the best way to deal with time-dependent data in Data Vault 2.0?”

In this enlightening video, Michael provides practical guidance on modeling strategies for incorporating SAP HRMS source data with Slowly Changing Dimension (SCD) type 2 attributes directly in the source. He addresses the complexities of handling time-dependent data within the Data Vault 2.0 framework, offering insights into the best practices for managing effectivity start and end dates for each change.

Michael shares valuable considerations and recommendations, providing a clear roadmap for efficiently handling time-dependent data scenarios in Data Vault 2.0 projects.

In Beginner, Data Vault

Quick Guide of a Data Vault 2.0 Implementation

Data Vault 2.0 Architecture

Data Vault 2.0 Implementation

Data Vault 2.0 is often assumed to be only a modeling technique, but it encompasses much more than that. Not only that, but it is a whole BI solution composed of agile methodology, architecture, implementation, and modeling.

So why start using Data Vault?

  • Data Vault 2.0 allows you to build automated loading processes/patterns and generate models very easily
  • Platform independence
  • Auditability 
  • Scalability
  • Supports ELT instead of ETL processes

Now that we answered the why, you may be wondering what steps are needed to implement Data Vault 2.0 in your project.

It depends on a lot of factors like your business case, the architecture you want to have in place, how your sources are loaded, the sprint timeline of your project, etc.

Walk-through of a Data Vault 2.0 Implementation

It can be a bit overwhelming for beginners to start using Data Vault 2.0 and how and where to implement it. In this webinar, a very basic guide will be provided showing the steps needed for making a Data Vault 2.0 implementation based on a business requirement from scratch. This will be done with a demonstrated example, and it starts from the gathering of some sample requirements to the finished delivered product.

Watch Webinar Part 1Watch Webinar Part 2

In this article:

Data Vault 2.0 feature by feature architecture

One thing is for sure: the architecture should be built vertically, not horizontally. This means not layer by layer but feature by feature. 

A common approach here is the Tracer Bullet approach. Based on business value, which is defined by a report, a dashboard, or an information mart, the source data needs to be identified, modeled, and loaded through all layers of the architecture. 

For example, let’s say the business request was to build a dashboard to analyze the company’s sales:

1. Extract

First thing, we need to extract the data from the source systems and load the data as it is somewhere. In this example, we put it in a Transient Staging Area but you could choose a persistent one in a Data Lake as well.

2. Transform

Next, you should apply some hard rules if necessary, be careful with this as you do not want to make business calculations here, using a transformation tool. There are a lot of different data warehouse automation tools that you can choose from: dbt, Coalesce, WhereScape, etc.

Data Vault 2.0 Architecture

3. Load

Load your Raw Stage into the Raw Vault.

4. Model Business requirements

Model the Data Vault entities needed for the business requirement to be fulfilled. If we have some Sales transactions and customers data, for example, we will model a Non-historized Link, also known as Transactional Link, and a Customer Hub, along with any additional Satellites for holding the Customer descriptive data that we want to see in the Sales Dashboard in the end.

5. Apply Business Logic

Next, we need some calculations and aggregations to be performed, so we will build some business logic on top of the raw entities, loading it into the business vault.

6. Build an Information Mart

Now, we could directly use the data stored in the Raw and Business Vault into charts/dashboards, but we want to structure the data, so it can be easily read and fetched by business users, so we will build an information mart with a star schema model with a fact table and dimensions.

7. Visualize Data

To build the Sales Dashboard in a BI visualization tool like PowerBI or Tableau, we now fetch directly from the star schema in the information mart, which has all the information we need, using a connection to my data warehouse in our database.

Data Vault 2.0 offers an agile, scalable, and flexible approach to Data Warehousing Automation. As demonstrated in the example, we only modeled the Data Vault tables that were necessary for accomplishing the handed task of building a Sales dashboard. This way you can scale up your business by demand, so you don’t have to figure out and map out the whole enterprise in one go. 

The answer to how to implement Data Vault 2.0 can be translated into a simple phrase: Focus on business value!

If you would like to see an explanation of this step-by-step implementation with some demonstration of actual data using dbt as the chosen transformation tool, check out the webinar recording.

Conclusion

Implementing Data Vault 2.0 involves a structured approach that begins with extracting data from source systems into a staging area, followed by minimal necessary transformations, and loading into the Raw Vault. Subsequently, business requirements guide the modeling of Data Vault entities, application of business logic, construction of information marts, and data visualization. This feature-by-feature methodology ensures scalability and flexibility, allowing organizations to focus on delivering business value incrementally. By aligning development efforts with specific business needs, enterprises can efficiently build and expand their data warehousing solutions.

In Data Vault Friday

EDW Environments in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke addresses a crucial question from the audience that highlights a common challenge in data projects.

“I’m currently working on a project where the ‘environments’ (Dev, Prod, Test) are not well administrated. This topic is not mentioned at all in the DV2.0 methodology. Could you please elaborate on the roles of these environments and how to correctly use and manage them? As context, the problem faced at the moment by the company is that they’re not being able to test correctly and then implement. Also, the environments don’t necessarily count with the same information.”

In this insightful video, Michael provides a comprehensive discussion on the roles and importance of environments (Development, Production, Test) in the context of Data Vault 2.0 methodology. He addresses the challenges faced by the company, emphasizing the critical role that well-administered environments play in testing, implementing, and ensuring data consistency across different stages.

Michael shares practical insights into the correct utilization and management of environments, offering guidance on establishing a robust environment strategy within the Data Vault framework.

In Intermediate

Automating Business Logic

Watch the Webinar

Automating Business Logic

In this webinar, you’ll learn that Data Vault automation is not restricted to loading data, but can also be applied to the presentation layer.

There’s always some repeatable business logic – think of calculations such as currency conversion, Lifetime Value (LTV), or Net Present Value (NPV) – to feed different reports, even if all of them contain different information.

We’ll explain how you can create custom business templates and add additional layers in the information marts, to apply calculations repeatedly and even interdependently, thereby extending the scope of Data Vault automation from integration to presentation.

This webinar focuses on practical solutions.

Watch Webinar Recording

Webinar Agenda

1. How to get data out of a Data Vault.
2. What’s a PIT, what’s a bridge?
3. What’s meant by virtualization?
4. How to identify low-hanging fruits, i.e. the repeatable business logic in your solution.
5. How to automate those business rules using VaultSpeed.

In Data Vault Friday

Loading Historical Data in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our BI Consultant Julian Brunner delves into a question from the audience that addresses a common challenge.

“One of our sources delivers all the historical data in one batch. So all the records have the same load date. How can I load the data into the EDW properly?”

In this insightful video, Julian shares practical solutions and strategies for loading historical data into an Enterprise Data Warehouse (EDW) when faced with the unique scenario of receiving all records with the same load date. The question prompts a discussion on best practices to ensure proper handling and integration of historical data within the Data Vault framework.

Julian provides valuable insights into the considerations and steps involved in effectively managing historical data loads, offering guidance on maintaining data integrity and accuracy within the EDW.

In Data Vault Friday

Data Vault 2.0 Source System Disaster Recovery

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke engages with a challenging question from our audience, aiming to find an elegant solution to a complex scenario.

“I’m trying to find an elegant way of addressing the following problem.

You have a DV2.0 Insert Only BI deployment fed by multiple OLTP systems. One of these OLTP systems will be subject to a disaster and associated recovery process. This will be done with a loss of 3h worth of data from the OLTP in question. During the 3 hours, multiple loads into the DV were completed.

I’m trying to avoid an effectivity satellite for each hub.”

In this insightful video, Michael explores strategies for handling data from multiple source systems with disaster considerations in a Data Vault 2.0 Insert Only BI deployment. The question prompts a discussion on avoiding the use of an effectivity satellite for each hub, offering alternative approaches to address the challenges posed by data loss during disaster recovery.

Michael shares practical insights and considerations for designing resilient solutions within the Data Vault framework while optimizing the balance between complexity and efficiency.

In Data Vault Friday

Data Vault 2.0 Project Tracking

Watch the Video

In our continuous Data Vault Friday series, our CEO Michael Olschimke addresses a pertinent question from our audience regarding the application of Scrum in Data Vault 2.0.

“We are struggling with the application of Scrum in Data Vault 2.0: the Kanban board is overloaded with technical user stories. However, in theory, the user stories should be oriented towards the business and user needs.”

In this insightful video, Michael delves into the challenges faced when integrating Scrum methodologies into Data Vault 2.0 projects, particularly the issue of an overloaded Kanban board with technical user stories. The question prompts a discussion on the alignment of user stories with business and user needs, emphasizing the importance of maintaining a business-centric focus.

Michael shares practical insights and recommendations for optimizing the use of Kanban boards in Data Vault 2.0 projects, ensuring a balance between technical requirements and business-oriented user stories.

In Data Tools, Intermediate

Bring Your Data Vault Automation to the Next Level with DataVault4coalesce

SALESFORCE SOLUTIONS

Data Vault Automation with DataVault4coalesce

A cooperation created DataVault4coalesce, an open source extension package for coalesce.io. In a previous webinar, we explored coalesce.io, a new competitor in the highly contested market of data warehouse automation tools.

coalesce

Level up your Data Vault automation – with DataVault4coalesce

Coalesce is a modern, column-aware data warehouse automation tool. In this webinar, you will learn how Scalefree’s latest publication brings best practices out of the Data Vault world into your coalesce.io experience. This includes data loading patterns, data vault related features, and more! All embedded into easy-to-use UI options to make use of Coalesce’s unique configurable user interface. Tune in to see DataVault4coalesce in action!

Watch Webinar Part 1Watch Webinar Part 2

In this article:

And everyone who watched that webinar might remember that at the end, we announced an even closer relationship between coalesce.io and Scalefree and a commitment to bring Scalefree’s best practices into coalesce.io!

For those who didn’t watch the webinar, you can find it here.

Recap: What is Coalesce?

coalesce.io is a Data Transformation solution made for Snowflake. When working with Coalescse, you build directed acyclic graphs (DAG) which contain nodes. A node represents any physical database object, like tables or views, or even stages and external tables.

Coalesce itself is built around metadata that stores table and column-level information, which describes the structure inside your data warehouse. This metadata-focused design enables a couple of features that strongly drive towards scalability and manageability. 

All the metadata allows a team to track past, current, and desired states of the data warehouse by deeply integrating it and all the workflows that it brings. Additionally, users can define standardized patterns and templates on column- and table-level.

How can Data Vault jump in here?

These mentioned templates open up the gate to implement Data Vault 2.0 related patterns and best practices. Especially on the table level, it might quickly come to mind that you could try to build a template for a Hub or a Link.

On column level, this could be a repeated transformation which is then managed in only one so-called macro, which makes it very easy to implement changes with low to zero impact. You could think of hash key calculation or virtual load-end-dating here.

And that is exactly what we at Scalefree have done since the webinar last year. Lead developers from coalesce.io sat together with Data Vault experts and developers from Scalefree with one goal: Create something amazing that helps users to automate their Data Vault 2.0 implementation!

Datavault4Coalesce

How fast can I build a Data Vault? Yes!

This cooperation created DataVault4coalesce, an open-source extension package for coalesce.io, which will be available on March 16th! Let’s have a sneak peek at a selection of what users can do with DataVault4coalesce.

The first release of DataVault4coalesce will feature a basic set of Data Vault 2.0 entities:

While providing DDL and DML templates for the entity types mentioned above, DataVault4coalesce makes use of Coalesce’s ability to define the UI for each node type. For stages, this means that users can decide if they want DataVault4coalesce to generate ghost records automatically or not, as shown in the screenshot below:

Datavault4Coalesce

This Data Vault related interface can be found across all node types and allows users to adjust DataVault4coalesce to fit their requirements conveniently!

Conclusion

First of all a bummer, DataVault4coalesce will only be available starting from the 16th of March. But there is no reason to wait that long to dive into coalesce.io itself! Since it is now part of Snowflake Partner Connect, it’s never been easier to get your hands on a fresh coalesce.io environment!

Just sign up for a free Snowflake trial here and initialize your coalesce.io experience within seconds by accessing the Partner Connect portal! Then, you just have to load any desired data into it, and you can start building your data pipelines with coalesce.io. And when the 16th of March finally arrives, you just have to add DataVault4coalesce to your coalesce.io environment – and now you can start to build Data Vault faster than ever!

Also, don’t miss out on this recording, where we will show you DataVault4coalesce in action. Watch it here!

In Data Vault Friday

(Logical) Information Marts in Data Vault

Watch the Video

In our continuous Data Vault Friday series, our CEO Michael Olschimke addresses a question from our audience that delves into the intricacies of the CDVP2 training.

“We are having trouble understanding the attached slide 28 of the CDVP2 training.

– What is the difference between Business DV Pits & Bridges and Pits & Bridges?
– We are confused about why Business Vault and Info Mart are put into one logical wrapper. Why does physical and logical wrapper differentiate?”

In this elucidating video, Michael provides clarification on the distinctions between “raw” and “business” Point-in-Time (PIT) and bridge tables. The question prompts a discussion on understanding the nuances of these components within the Data Vault methodology.

Michael shares insights into the reasoning behind grouping Business Vault and Info Mart into one logical wrapper while emphasizing the differentiation between physical and logical wrappers. The discussion provides valuable context for participants seeking clarity on the CDVP2 training material.

In Intermediate

Why Data Vault 2.0 Is the Best Data Model for Automation

Watch the Webinar

Why Data Vault 2.0 Is the Best Data Model for Automation

Many data teams worry that automation won’t work on their specific data and technology stack. They’ve learned the hard way that automation doesn’t always stand up to the complexity of different source data models, taxonomies, and tech stack components.
Join this webinar to understand how Data Vault 2.0 is designed to focus on models and logic, not complex code so that it’s rapidly becoming the DWH standard.

We’ll explain how Data Vault has taken the best of the more traditional modeling
approaches, such as Inmon or Kimball, to provide the level of abstraction, quality, and agility that automation requires.

You’ll learn how the Data Vault model and its methodology and architecture leverage
automation. And how we use integration templates based on Data Vault standards to pave the way to fully automated data loading.

This webinar takes you from theory to practice.

Watch Webinar Recording

Webinar Agenda

1. The pros and cons of different data modeling techniques.
2. The prerequisites for automation.
3. Why Data Vault works best.
4. How to create abstractions in data warehousing.
5. Demo: how it’s applied in VaultSpeed.

Close Menu