Skip to main content
search
0
CartClose Cart
In Beginner, Salesforce

Unlock Success: Dive into the Salesforce Summer Release ‘24!

Watch the Webinar

Unlock Success: Dive into the Salesforce Summer Release ‘24!

Gear up for success as we dive into the highly anticipated Salesforce Summer Release ’24 in our exclusive webinar, “Unlock Success: Dive into the Salesforce Summer Release ’24!” Gain a competitive edge by getting ahead of the curve with a sneak peek into the upcoming updates that are set to revolutionize your business. Join us for an insightful exploration of the latest features and enhancements before they’re even released, and ensure you stay steps ahead of the competition.

In this dynamic webinar, we’ll provide you with an insider’s look into what the Salesforce Summer Release ’24 has in store. From game-changing functionalities to transformative enhancements, you’ll discover how these updates can propel your business forward and drive greater efficiency, productivity, and success. Whether you’re a seasoned Salesforce user or new to the platform, this webinar offers invaluable insights to help you maximize the potential of Salesforce and stay at the forefront of innovation.

Don’t miss this exclusive opportunity to gain early access to the groundbreaking features of the Salesforce Summer Release ’24. Register now to secure your spot and embark on a journey towards unlocking success with Salesforce. Stay ahead of the competition and position your business for growth and prosperity in the ever-evolving digital landscape.

Watch Webinar Recording
In Data Vault Friday

How to Track Soft Deletes in an Insert Only Data Vault 2.0 Architecture

Watch the Video

In our ongoing series, our BI Consultant Lorenz Kindling addresses a question from the audience about managing soft deletes in an insert-only data environment. This topic is particularly relevant for those in the field of data warehousing, where maintaining historical data integrity and accuracy is paramount.

The question posed was, “How to track soft deletes with insert only?” Lorenz’s response explores the complexities and best practices for implementing soft deletes within an insert-only framework. He explains that soft deletes involve marking records as inactive rather than physically removing them from the database. This approach is crucial for maintaining a comprehensive historical record and ensuring that data integrity is not compromised. Lorenz suggests using a specific status indicator or a flag within the data model to denote records that are logically deleted. This allows for efficient querying and reporting without the risk of losing historical data.

Lorenz, who has been advising renowned companies since 2021 at Scalefree International, draws on his extensive experience in Business Intelligence and Enterprise Data Warehousing to provide practical insights. He emphasizes that by carefully planning and implementing a robust soft delete mechanism, organizations can achieve a balance between data retention and performance. Lorenz’s approach ensures that data warehouses remain both scalable and efficient, even as they grow and evolve over time.

In conclusion, Lorenz highlights the importance of adopting best practices in data warehouse automation and Data Vault modeling to manage soft deletes effectively. By using insert-only methods with proper indicators for soft deletes, organizations can maintain the integrity and usability of their data warehouses, thereby supporting long-term business intelligence and analytics goals. This strategy not only addresses common data warehousing challenges but also aligns with modern data management principles.

In Data Tools, dbt webinar, Intermediate

Scale Up your Data Vault Project – with dbt Mesh

dbt Mesh - data mesh solution

dbt Mesh

Learn how dbt Mesh enhances Data Vault projects within dbt Cloud by facilitating a more efficient data mesh architecture. The larger a data warehouse project grows, the more people begin to rely and work with the data provided. This work could be consuming the data, applying business rules, modeling facts and dimensions, or other typical tasks in a data environment. In a large organization, all these users might be scattered across different divisions, and the data they are working with might belong to different business domains. At some point, the entire organization faces the challenge of data sharing and governance guidelines, which might prohibit users of the sales department from accessing data from the finance department. A data mesh offers a solution that helps organizations to deal with these challenges. If you want to learn more about the data mesh, check our recent blog article about Data Vault and data mesh here!

We also have a webinar on exactly this specific subject. Don’t miss it and watch the recording for free!

Data Mesh Support bei dbt Cloud

Many organizations struggle with introducing a Data Mesh approach into the Data Vault landscape. In this webinar, we will dive into dbt Mesh, and how to leverage it in a Data Vault project.

Watch Webinar Recording

In this article:

What is dbt Mesh?

Dbt Mesh is a recently added feature that makes dbt Cloud work more efficiently with a data mesh approach. The already familiar {{ ref() }} function is no longer limited to models within one dbt project, instead it can refer to models of other dbt projects.

Why would I want to refer to other dbt projects?

Imagine a big organization that uses dbt Cloud for their Data Vault implementation. The project might have 400 sources defined, 2000 models implemented, and is used actively by 30 developers. Out of these 30 developers, there might be 5 people specifically working on the Business Data Vault and Information Mart layer for finance-related objects. Another 5 developers are working on the same layers but for sales-related objects.

At some point, you might want to avoid finance people messing around with the sales-related dbt models, so a data mesh architecture is to be implemented. This would allow the organization to define policies regarding data sharing, data ownership, and other governance measures.

With dbt Mesh, both the Sales and the Finance team would get their own dbt project. Since both should be based on the same Raw Data Vault, an additional foundational dbt project is created exclusively for staging and Raw Data Vault objects. Both domain-specific dbt projects, sales and finance, can now refer to Raw Vault objects inside the foundational dbt project, avoiding actually physically replicating the data.

dbt Mesh - data mesh solution

How can I leverage dbt Mesh in a Data Vault powered Data Mesh?

Define Data Contracts

Dbt models, or groups of models, can now be configured to have data contracts. Inside the already familiar .yml files, models can now be set to be publicly available (within an organization), data owners can be enforced, and table schemas can be locked.

Create a Foundational dbt project

In a Data Mesh architecture, the most common way to implement Data Vault 2.0, is to have a commonly shared Raw Vault as a foundation, and both Business Vault and Information Marts are divided by business domains. In dbt Mesh, this would reflect in a foundational dbt project, that includes all staging and Raw Data Vault objects. Only the Raw Data Vault objects would be configured to be accessible by other dbt projects, since the staging models should not be used outside of Raw Data Vault models.

Add domain-level dbt projects

Based on the foundational Raw Vault dbt project, each domain team can now work in their own dbt project. They access the Raw Data Vault via the (extended) {{ ref() }} function and don’t have to worry about maintaining these Raw Vault objects. Additionally, they can define which of their artifacts might be useful for other domains, these can be shared via their own data contracts.

Distribute Responsibilities

Typically, a power user does not create Hubs, Links, and Satellites. And it’s not their responsibility to ensure a reliable Raw Data Vault to build transformations on. Therefore, it is important to define responsibilities within each dbt project. Especially objects that are shared outside of one project should always have data contracts and defined owners. This ensures that users of these shared objects can rely on it.

Conclusion

All in all, dbt Mesh offers a fantastic way to properly implement a true data mesh approach. It is especially relevant, when different business domains of one organization are working together in dbt to create trustable deliverables. In most scenarios, it makes sense to already start using dbt Mesh, although your project might not be too big yet. Having clear responsibilities and data contracts always helps maintain trust and transparency for your data!

In Data Vault Friday

Modelling Exchange Rates

Watch the Video

In our ongoing series, our CEO Michael Olschimke addresses a question from the audience about modelling daily exchange rates within the Data Vault framework for a non-banking industry. The query highlights a common challenge faced by many organizations: integrating and managing exchange rate data effectively.

The question posed was, “How would you model daily exchange rates in Data Vault 2.0 for a non-banking industry? We are already using a reference table for the list of currencies (I guess we would have currency as a hub in the banking industry, but that is not our case). Now we also need daily exchange rates for currency conversions in the datamart layer. I would start with a Link for exchange rates, but do we need to create a hub for currencies? How about existing references to currency in the existing model (currently in SAT, because we have currency as a reference table)?”

Michael’s response delves into the intricacies of data modelling in such scenarios. He suggests that even though your industry is non-banking, establishing a structured and scalable way to manage exchange rate data is crucial. Using a Link for exchange rates is a good starting point, but creating a hub for currencies could provide additional benefits. This hub would act as a central repository for all currency-related information, ensuring consistency and ease of access across different layers of the data architecture. Additionally, integrating existing references to currencies within the model can streamline operations and enhance the accuracy of financial data analytics.

In conclusion, Michael emphasizes the importance of a well-thought-out data architecture. By creating a dedicated hub for currencies and effectively linking exchange rate data, organizations can ensure more accurate and efficient currency conversions in their datamart layer. This approach not only aligns with best practices in data vault modeling but also supports the broader goal of maintaining data integrity and usability across the enterprise.

In Data Vault Friday

How to Implement Data Quality Techniques

Watch the Video

In our latest video, BI Consultant Julian Brunner tackles a pressing query: “Where to implement data quality techniques? Is it possible to clean dirty data at the entry point into the raw data vault?” Data quality is foundational for informed decision-making, and Julian’s expertise shines as he navigates this critical terrain.

Julian highlights the importance of a holistic approach to data quality management, emphasizing the need for robust frameworks spanning the entire data lifecycle. Whether it’s validation rules, data profiling, or cleansing algorithms, proactive measures at every stage can fortify data integrity.

In Beginner

Automating a Scalable Data Warehouse with Data Vault Builder

Watch the Webinar

Automating a Scalable Data Warehouse with Datavault Builder

Unlock the power of automation in your data warehouse with Data Vault Builder in our upcoming webinar. Dive into the intricacies of Data Vault 2.0 and discover why it’s tailor-made for automation, promising efficiency and scalability like never before. Whether you’re a seasoned data professional or just embarking on your data warehousing journey, this webinar offers invaluable insights into streamlining your processes and accelerating implementation.

During this joint webinar, you’ll delve into the core principles of Data Vault 2.0 and witness firsthand how Data Vault Builder revolutionizes the implementation process. Through a live demonstration, gain practical knowledge and actionable tips to optimize your data warehouse architecture. From overcoming common challenges to kickstarting your project with confidence, this session equips you with the tools and techniques needed to succeed in the world of data warehousing.

Don’t miss this opportunity to elevate your data warehousing game and leverage the full potential of automation with Data Vault Builder. Join us and discover how to transform your data infrastructure into a dynamic, scalable powerhouse. Whether you’re a data architect, analyst, or IT professional, this webinar promises to be a game-changer for your organization’s data strategy. Register now to secure your spot!

Watch Webinar Recording
In Data Vault Friday

Data Vault 2.0 Pre-Analysis aka Automation for the Poor

Watch the Video

In our ongoing series, our CEO Michael Olschimke discusses a question from the audience:

“In recent training from ScaleFree I saw a glimpse of an excel sheet that basically annotated data sources with data vault specific metadata. It had like plenty of Salesforce attributes in it together with annotations like: Business key, Link business key, Satellite descriptive attribute, etc.

Can you talk a bit about this metamodel? Can it be used to drive automated creation of the data vault structures?”

Michael stresses the irreplaceable role of pre-analysis in establishing a successful Data Vault 2.0 framework. Michael underscores the crucial nature of this stage, aligning our work with each client’s aspirations while staying true to the guiding wisdom of Dan Linstedt.

In Data Tools

Exploring datavault4dbt: A Practical Series on the dbt Package for Data Vault 2.0 – Vol. 2: Standard Entities in the Raw Vault

Exploring datavault4dbt

In our initial post of this series, we delved into the creation of our staging layer using DataVault4dbt, an open-source package designed for Data Vault 2.0 within dbt. In this installment, we embark on the journey to construct our first standard Data Vault 2.0 model entities in the Raw Vault, including Hubs, Links, and Satellites. As in our previous post, we recommend staying up-to-date with the latest changes and adaptations in the DataVault4dbt package by referring to the project’s GitHub repository Wiki.

In this article:

Before We Start

Before we get started, ensure that you have the DataVault4dbt package correctly installed in your packages.yml file and that you’ve executed dbt deps.

For this tutorial, we’ll be using the TPCH Snowflake Sample Data. Moreover, we assume you’ve already established your staging model, which includes the calculation of hashkeys and hashdiffs. Here’s a snippet from our staging model‘s configuration, which we’ll need later when creating the Raw Vault entities:

stg_orders
Staging layer in datavault4dbt

A. Standard Hub with datavault4dbt

Hubs are constructed based on a unique list of business keys, making their configuration relatively straightforward. In this example, we’ll be creating the Hub for orders:

order_h
Hub in datavault4dbt
  • hashkey: the hashkey name in the staging model
  • business_keys: name of the business key used as input for the previously mentioned hashkey
  • source_models: name of our staging model

B. Standard Link with datavault4dbt

Link models establish connections between business keys. In our case, we’ll create a connection between the previously formed Order Hub and the Customer Hub:

order_customer_1
Link in datavault4dbt
  • link_hashkey: hashkey of the Link, generated using the foreign keys from the Hubs in the staging layer
  • foreign_haskeys: a list of foreign hashkeys to be included in our link
  • source_models: name of our staging model

C. Standard Satellite Version 0 with datavault4dbt

Following Data Vault 2.0 standards, Version 0 Satellites are created as incremental tables. In our example, the Satellite will be connected to the previously generated Order Hub:

order_0s
satellite version 0 in datavault4dbt
  • parent_hashkey: name of the parent entity’s hashkey, in our case, the Order Hub
  • src_hashdiff: hashdiff already calculated on the staging model
  • src_payload: original columns used in the hashdiff calculation
  • source_model: name of our staging model

D. Standard Satellite Version 1 with datavault4dbt

Additionally, the Version 1 Satellite is a virtually generated entity created on top of our Version 0 Satellite. Beyond the materialization type, the main difference with the V0 Satellite is the introduction of a new column for calculating the load end date. The load end date will be useful for us downstream when dealing with PIT tables in the Business Vault.

order_s
satellite version 1 in datavault4dbt
  • sat_v0: name of the related Version 0 Satellite
  • hashkey: hashkey name of the parent entity, in our case, the order Hub
  • hashdiff: hashdiff already calculated on the staging model
  • ledts_alias: name of the load end date column to be generated
  • add_is_current_flag: when true, it generates a new column flagging the last loaded rows based on the load end date

Conclusion

In this journey through the creation of Raw Vault standard entities, we’ve established a strong foundation for our Data Vault 2.0 architecture. By utilizing DataVault4dbt within dbt, we’ve simplified the development of Hubs, Links, and Satellites. These fundamental building blocks are the cornerstone of a robust and scalable data warehousing solution. As we progress in this series, we’ll continue to explore advanced concepts and delve into the intricacies of Data Vault modeling, preparing us to unlock the full potential of our data.

In Data Vault Friday

Data Vault PITs in PowerBI – Joining Type 2 Dimensions

Watch the Video

In our ongoing series, our CEO Michael Olschimke discusses a question from the audience:

“We would like either build a semantic model or let end users build it themselves as star schemas.

In Infomarts we expose facts and dimensions.

Dimensions are based on pits and expose all contextual attributes valid to a given snapshot data.

Now the problem is that facts and dimenions need to be joined not only by main keys of dimension, but also a snapshot data. However, PBI allows only joins with 1 attribute.

What is a recommended way to tackle this?

I thought of introducing sequence numbers in PITs and exposing them in virtualized fact views, additionally exposing separate snapshot dimension that synchronizes snapshots of all the dims (otherwise we end up in cartesian join). However this defeats partitioning in the PITs (join over sequence number and not hashKey + SnapshotDate blocks partition pruning).”

Michael delves into an in-depth discussion on leveraging Data Vault’s Point-in-Time (PIT) tables within PowerBI, exploring how this integration enhances analytical capabilities and supports dynamic reporting in the realm of Big Data.

In Beginner, Data Vault

Designing the Business Vault: Key Strategies for Effective Data Organization

DV2.0 Architecture with Business Vault

Designing the Business Vault

Data Vault 2.0 has emerged as a comprehensive framework, offering agility, scalability, and adaptability. At the heart of this framework lies the Business Vault, a critical component for effective data organization and analysis in modern enterprises.

In this article, we will check the key principles and strategies for designing a robust Business Vault within the context of Data Vault 2.0.

Designing the Business Vault: Key Strategies for Effective Data Organization

Join us for an insightful webinar on “How to design the Business Vault?” as we explore the critical role of the Business Vault within the Data Vault 2.0 framework. Discover how the Business Vault serves as a pivotal component for translating raw data into actionable insights, applying soft business rules to streamline end-user structure creation, and ensuring an efficient population of Information Marts.

Watch webinar recording

In this article:

Understanding Data Vault 2.0

Data Vault 2.0 represents a paradigm shift in data architecture, distinguishing itself from traditional warehousing methods. Its flexibility and scalability make it ideal for organizations navigating the complexities of modern data ecosystems.

Data Vault 2.0 architecture follows a multi-layer approach, consisting of the Staging Layer, the Enterprise Data Warehouse Layer, and the Information Marts Layer. By dividing our data architecture into multiple layers, we can respond to both the needs of the technical teams (i.e., historization, auditability, and data integration) and the requirements of the business users (i.e., quick access to relevant, well-organized information). This integrated approach ensures a harmonious synergy between technical and business objectives.

DV2.0 Architecture with Business Vault

To achieve all these goals, Data Vault 2.0 proposes a subdivision inside the Enterprise Data Warehouse Layer: the Raw Vault and the Business Vault. The Raw Vault will receive and integrate the unaltered data from the source, while the Business Vault will translate the raw data into meaningful insights for informed decision-making.

Importance of Business Vault

The Business Vault serves as a middle ground between the Raw Vault and the Information Mart layers. It is an optional vault, sparsely generated on top of the Raw Vault and normally it is virtualized. Differently from the Raw Vault, in the Business Vault, we will be applying soft business rules, i.e., those rules that change the data.

This layer will be created to serve the business in different ways, such as the generation of query assistance entities or by precomputing calculated fields that later will be used on downstream layers. In other words, the Business Vault will host business-rule changed data and its purpose is to ease the creation of end-user structures.

Key Concepts of a Business Vault

A Business Vault will be modeled following the Data Vault 2.0 design principles. Nevertheless, it won’t necessarily follow the strict auditability requirements of the Raw Vault, as we can drop and recreate the Business Vault entities at any time. With the purpose of serving to populate the Information Mart more easily and efficiently, the entities will be created only if they are necessary for the business. This is also why the Business Vault usually keeps reusable business logic.

The types of entities we can typically find inside a Business Vault could be Point-In-Time (PIT) and Bridge Tables, for query assistance; Computed Satellites or Links, for storing computed data; and Exploration Links, for connecting Hubs that were not previously connected to the Raw Vault. Besides, any other entities that are created on top of the Raw Vault, using business logic and queried by the Information Marts layer, would belong to the Business Vault. For instance, we might need business logic to map instances of the same thing, thus creating a Business Same-as Link.

Conclusion

In data management, Data Vault 2.0 encompasses different aspects such as data modeling, methodology, and architecture. Distinguished by its versatility, this framework places a significant emphasis on agility and adaptability. In this sense, at the core of Data Vault 2.0 architecture lies a pivotal concept, the Business Vault, a key player for efficient data organization and analysis in modern enterprises.

The Business Vault, a flexible optional layer, interprets raw data into actionable insights, applying soft business rules. Its purpose is to streamline end-user structure creation by hosting processed data. Entities are created selectively, keeping reusable business logic. In essence, the Business Vault ensures the efficient population of Information Marts by focusing on business-critical data.

Interested in more? Watch the webinar recording here for free!

In Data Vault Friday

Multiple PIT Tables for Different Business Scenarios

Watch the Video

In our ongoing series, CEO Michael Olschimke addresses an audience question:

“We have a need for PIT tables to be used in different business scenarios and would like to use different SQL statements to load a number of PIT tables, one for each scenario.

What is your take on it?”

PIT tables help track the historical state of records over time for analysis, and customized SQL statements cater to the specific data needs of each business context.

Michael will cover best practices, performance optimization, and data integrity when managing numerous PIT tables, offering insights that can enhance organizational data strategy.

In Salesforce Services

Salesforce Marketing Cloud Overview

Impactful Marketing

Summary

In this 12-minute video, we provided a concise overview of Salesforce Marketing Cloud’s features, demonstrated the process of requesting a test environment, and highlighted certification opportunities.

The remainder of the video showcased live demonstrations of the platform’s features within a marketing environment.


Salesforce Marketing Cloud

Salesforce Marketing Cloud is a comprehensive marketing automation platform offering diverse features.

It enables users to efficiently manage and analyze customer interactions across multiple channels.

The platform allows for personalized customer journeys, leveraging data-driven insights for targeted marketing campaigns.

Users can request a test environment to explore and familiarize themselves with the platform’s capabilities.

Additionally, Salesforce Marketing Cloud offers certification options for users to validate their expertise in utilizing its powerful tools.


Why It Matters

Salesforce Marketing Cloud is crucial for businesses aiming to enhance customer engagement and drive successful marketing campaigns.

Its robust features empower organizations to create personalized and data-driven strategies, optimizing customer journeys for maximum impact.

The ability to request a test environment ensures seamless integration and adaptation to individual business needs, fostering efficient utilization of the platform’s capabilities.

Furthermore, certifications offered by Salesforce Marketing Cloud validate and elevate professionals’ skills, contributing to a skilled workforce capable of leveraging the platform for strategic marketing success.

Embracing Salesforce Marketing Cloud is a strategic move towards achieving targeted, impactful, and results-oriented marketing initiatives.


Target Audience

It is especially relevant for digital marketers, CRM managers, and marketing teams aiming to streamline and optimize their multi-channel marketing efforts.

The platform’s versatility makes it suitable for both beginners exploring marketing automation and experienced professionals looking to elevate their strategies with data-driven insights.

Watch the Video

Close Menu