Skip to main content
search
0
CartClose Cart
In Data Vault Friday

Extending Existing Data Vault Model by GDPR-Identified Data

Watch the Video

In our ongoing Data Vault Friday series, our esteemed CEO, Michael Olschimke, tackles a compelling question raised by an engaged member of our audience.

“Let’s assume that DWH is fed from many source systems and one of them (some minor one, called ‘XYZ’) exports customer data identified by PERSONAL_ID (no other identifier available). We already have HUB_CUSTOMER based on some other customer identifier, and the PERSONAL_ID attribute is stored in SAT_CUSTOMER_PD. But there is one important thing regarding customer data, there are cases where multiple rows in HUB_CUSTOMER have the same PERSONAL_ID in mentioned satellite (which means, that some of the customers have been registered multiple times in our core systems).”

In this illuminating episode, Michael delves into the intricate scenario of integrating customer data from diverse sources, emphasizing the challenges posed by the absence of a unique identifier and the existence of duplicate entries. He articulates a strategic approach to address this nuanced issue within the Data Vault framework, providing practical insights and recommendations for achieving a coherent and accurate representation of customer information.

This discussion proves invaluable for data professionals navigating the complexities of consolidating diverse customer data sets with varying identifier structures.

In Data Vault Friday

Processing CDC Data in Data Vault

Watch the Video

As part of our ongoing Data Vault Friday series, our knowledgeable BI Consultant, Julian Brunner, delves into a question presented by an audience member.

“One of our sources delivers CDC data. Are there any DV standards/best practices on how to model and process this kind of data?”

In this insightful episode, Julian addresses the specific challenges associated with Change Data Capture (CDC) data within the realm of Data Vault methodology. He provides a comprehensive overview of the best practices and standards that should be considered when modeling and processing CDC data in a Data Vault environment.

Julian’s expertise shines through as he navigates the intricacies of incorporating CDC data seamlessly into the Data Vault model. Viewers gain valuable insights into the recommended approaches, key considerations, and potential pitfalls to be aware of when dealing with CDC data sources.

By the end of the discussion, Julian equips the audience with practical knowledge, empowering them to effectively integrate CDC data into their Data Vault implementations.

In Data Vault Friday

Extending Satellites in Data Vault

Watch the Video

In our continuous Data Vault Friday series, our skilled trainer, Marc Finger, delves into a question posed by an audience member.

“Changes in the source system (new column/s): New row in an existing Satellite or new Satellite?”

Marc provides valuable insights into handling changes in the source system, specifically when encountering the addition of new columns. The question revolves around whether it’s more appropriate to introduce a new row in an existing Satellite or create an entirely new Satellite to accommodate these changes.

Through a clear and concise discussion, Marc elucidates the considerations and factors that influence the decision-making process. He explores the implications of both options, emphasizing the importance of aligning the chosen approach with the specific requirements and goals of the Data Vault model.

The trainer guides the audience through the thought process involved in making this decision, providing practical tips and best practices. By the end of the episode, viewers gain a deeper understanding of how to navigate the challenges associated with changes in the source system within the context of Data Vault methodology.

In Data Warehouse, Expert

Mastering Metadata: Data Catalogs in Data Warehousing with Datahub

Mastering Metadata in Data Warehousing

In today’s data-driven world, it is essential to be able to manage and organize large amounts of data in an efficient way. Businesses across all industries are forced to contend with more data than ever before. The introduction and development of an enterprise data warehouse in a company naturally plays a central role, but does not solve a major challenge: How to effectively organize and manage the data, especially metadata, in an Enterprise Data Warehouse? This is where the concept of data catalogs comes into play and where tools like Datahub become essential. 

A data catalog serves as a comprehensive inventory of data assets in an organization, providing context, annotations, and metadata to facilitate the understanding and discovery of data. It’s like a map to your data, helping users navigate the complex data landscape to find the exact data they need.

A data catalog can help users to understand where to find specific data in the data warehouse that fits their needs and to investigate where it came from, as well as how it might be connected to other data. This can greatly simplify tasks like data analysis and reporting, making the data warehouse more accessible and usable for everyone in the organization.

Mastering Metadata: Data Catalogs in Data Warehousing with DataHub

Don’t miss our upcoming webinar about data catalogs! This session will explore in detail the critical role of data catalogs in data warehousing, with an exclusive focus on the powerful tool DataHub. You’ll gain practical insights on enhancing data discovery, metadata management, data lineage, and data governance. Sign up today and transform your data management strategies into a competitive advantage.

Watch webinar recording

In this article:

Understanding Data Catalogs

What is a Data Catalog?

In general, a data catalog is like a metadata inventory, which consists of organized and structured metadata regarding all data assets in an organization. It is a central place where all this metadata can be stored, combined, and categorized, which makes it a lot easier to discover and understand the corresponding data, for example in a data warehouse. A data catalog also has search functionalities to find specific data from the available indexed datasets. It serves like a single source of truth of your metadata, enabling users to trust the data they’re using for their analyses or business decisions.

Role of a Data Catalog in Data Warehousing

In the context of data warehousing, a data catalog brings a lot of benefits. It provides a way to explore and search all data stored in the data warehouse. Technical users, as well as Business Users, can discover relevant data, understand its context, and ensure it is up-to-date, reliable, and accurate. The following figure shows where data catalogs in data warehousing with Data Vault 2.0 take place. A Data Catalog should cover the entire Enterprise BI Solution. This also applies, for example, to a data lake, if available, and to the information delivery layer.

Data Vault 2.0 General Architecture

Now that we’ve understood what a data catalog is, let’s delve into how each component plays a part in a data catalog and explore how a tool like DataHub can assist organizations in these tasks.

Introduction to DataHub

In the world of data catalogs, DataHub stands out as an increasingly popular choice for many businesses. DataHub is a growing open-source software developed by LinkedIn to address its growing need for a more dynamic and scalable data management tool. It was created in part due to the fact that the existing tools were not sufficient with LinkedIns expanding needs.

As LinkedIn grew, so did its data volume, variety, and velocity. Recognizing the need for a more efficient way to manage its data, LinkedIn built and introduced DataHub in 2020. Open-sourcing DataHub allowed other organizations to benefit from this advanced tool, and it has since been adopted by many businesses looking for a modern, scalable data catalog solution.

DataHub supports both push-based and pull-based metadata ingestion, including a wide range of integrations for example Airflow, BigQuery, Databricks, dbt, Hive, Kafka, Looker, MSSQL, MongoDB, Oracle, S3, PowerBI, Snowflake, Spark, and much more. You can find a full list here. This gives datahub the ability to combine and show metadata of the same data from multiple sources, for example, a dbt model definition, and if the tests were running successfully, right next to the database schema and stats for all columns.

Datahub statistics

Key Features and Capabilities of DataHub

DataHub, as a metadata platform, goes beyond traditional data catalogs. DataHub offers all important features and capabilities:

1. Scalability: DataHub is designed to handle metadata from thousands of datasets, which makes it a great choice for large organizations.

2. Flexible and Extensible Data Model: The technical data model behind this tool is designed to be customizable and expandable to allow organizations to customize it to their specific business requirements

3. Powerful Search and Discovery: Leveraging Elasticsearch, DataHub offers robust search functionality that enables users to discover datasets quickly based on various attributes, such as the data’s origin, schema, and usage.

Datahub capabilities

4. Rich Metadata: Unlike traditional data catalogs, DataHub captures and presents a wide variety of metadata, including data lineage, operational metadata, and business metadata. This gives users a comprehensive understanding of their data.

5. Data Lineage and Relationships: DataHub automatically captures and visualizes data lineage, showing how data flows through various systems. It also displays relationships between datasets, allowing users to understand how different data assets interact with each other.

Lineage and Relationships in datahub

Conclusion

Using a data catalog comes with several benefits:

  1. Enhanced Data Discovery: With the search and categorization capabilities of a data catalog, users can quickly find the exact data they need without having to comb through large datasets.
  2. Improved Data Understanding: The metadata in a data catalog provides users with necessary context about the data, making it easier to interpret and use correctly.
  3. Better Compliance and Governance: A data catalog supports data governance initiatives by ensuring data is consistent, accurate, and compliant with relevant regulations.
  4. Increased Trust in Data: By providing transparency into data lineage, a data catalog helps build trust in the data by allowing users to see its history and verify its accuracy and reliability.
  5. Time and Resource Efficiency: By making it easier to locate and understand data, a data catalog can save the company resources, thus speeding up data-driven activities and reducing the burden on data management teams.

In conclusion, DataHub provides a flexible, feature-rich, and all-encompassing option for data catalogs in a data warehousing environment. By providing powerful features for data discovery, metadata management, data lineage, and data governance, it enables businesses to extract maximum value from their data.

If you’re interested in learning more about data catalogs, watch the recording here for free.

In Data Vault Friday

Metadata Translation in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke discusses a question from the audience.

“Our EDW should use English entity names for hubs, links, and satellites. However, our sources are in a variety of languages (English, and German mostly). Where is the best option to translate everything into English?”

Michael provides insightful guidance on tackling the challenge of maintaining consistency in entity names across a multilingual landscape. He explores different strategies for translating entity names, weighing the pros and cons of various approaches. Whether to perform the translation at the source level, during the ETL (Extract, Transform, Load) process, or within the EDW itself, Michael offers considerations to help make an informed decision based on the specific needs and characteristics of the project.

The CEO emphasizes the importance of aligning with business objectives and ensuring that the chosen translation strategy aligns with the overall goals of the data warehousing initiative. This episode provides valuable insights and best practices for handling multilingual challenges in Data Vault projects, contributing to the success of your data integration and management endeavors.

In Data Vault Friday

Hiding Dimension Members in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke addresses a query from the audience, exploring the dynamics of managing data visibility in the DIMENSION information mart.

“How can a record be hidden in the DIMENSION information mart if it is no longer in use? Our Data Warehouse (DWH) features a hierarchy of region, division, and zone, which may undergo splitting or merging multiple times. The challenge is that the deleted event is not signaled from the source side, and only a full refresh captures new hierarchy information. Users desire a consistently current status reflected in both FACT and DIM tables.

1. To handle this, the current relation can be flagged and counted. This approach involves managing the relationship with a counter, allowing for effective tracking and visibility.

2. Additionally, the last relation needs to remain visible in the FACT table, ensuring that historical relationships are retained for reference.”

In this engaging video, Michael elaborates on these strategies, providing insights into maintaining data integrity and visibility within complex hierarchies, while accommodating changes and updates efficiently.

In Data Vault Friday

Sampling (DB Subsetting) Production Data in Data Vault

Watch the Video

In our ongoing Data Vault Friday series, our CEO Michael Olschimke engages with a pertinent question from the audience, shedding light on best practices for structuring EDW environments.

“In one of the previous webinars (‘EDW Environments’), you mentioned about best practices for creating your EDW environments. Let’s consider a configuration where we have 4 environments, DEV + TST and PRE_PROD + PROD. Moreover, assume that the PROD environment is very heavy in the meaning of data volumes and we simply cannot handle such amounts of data on PRE PROD and TST (data on TST env. will be anonymized). Do you have any advice on how to create lightweight environments from PROD?”

In this insightful video, Michael delves into the complexities of managing EDW environments with varying data volumes. He offers practical advice on creating lightweight versions of the production environment for development, testing, and pre-production stages. The discussion encompasses strategies for data anonymization on the testing environment and optimizing resources to ensure efficiency across different stages of the EDW lifecycle.

In Data Vault Friday

Reference Tables With Effectivity Satellites in Data Vault

Watch the Video

In our continuous exploration of Data Vault concepts in the Data Vault Friday series, our CEO Michael Olschimke delves into an intriguing question posed by the audience.

“Do you use Effectivity Satellites also for Reference Data in Reference Satellites?”

This concise yet crucial inquiry prompts Michael to unravel the considerations and best practices associated with leveraging Effectivity Satellites in the context of Reference Data within Reference Satellites.

In this insightful video, Michael shares his expertise, discussing the potential applications and benefits of employing Effectivity Satellites for managing reference data. He sheds light on how this approach can enhance the flexibility and temporal aspects of Reference Satellites, contributing to a more robust and adaptable Data Vault architecture.

In Intermediate, Salesforce

Stärken der Datenanalyse Innerhalb von Salesforce

Watch the Webinar

Stärken der Datenanalyse Innerhalb von Salesforce

In diesem Webinar erfahren Sie, wie Sie Salesforce-Reports erstellen und nutzen können, um aussagekräftige Einblicke in Ihre KPI zu gewinnen.

Wir zeigen Ihnen bewährte Best Practices für die Erstellung von benutzerdefinierten Berichten und Dashboards sowie praktische Tipps für die effektive Nutzung von Salesforce-Reporting-Tools.

Salesforce unterscheidet sich unter allen anderen CRM-Systemen, da das Reporting auf allen Daten innerhalb des Systems möglich ist. Sie lernen, wie Sie Reports filtern, gruppieren und Diagramme erstellen können.

Dieses Webinar richtet sich sowohl an Anfänger als auch an fortgeschrittene Salesforce-Nutzer, die ihre Reporting-Fähigkeiten verbessern möchten.

Watch Webinar Recording

Webinar Agenda

1. Reports In Salesforce
2. Vorteile
3. Warum Nutzen
4. Demos
5. Fazit

In Data Tools, Intermediate

Data Vault 2.0 with Hadoop and Hive/Spark

Hadoop and Hive/Spark in Data Vault 2.0

In this article, you’ll receive an overview of what Hadoop and Hive is and why they can be used as an alternative to traditional databases.

Data Vault 2.0 with Hadoop and Hive/Spark

This webinar delves into the ins and outs of Hadoop and Hive, including what they are and how they communicate. The second part of the presentation focuses on a Data Vault 2.0 example architecture using batch loading, providing participants with insights into how a sample can look like to provide value in real-world scenarios. Whether you are a seasoned data professional or just starting out, this webinar is an invaluable resource for anyone seeking to learn more about Hadoop. So if you are looking to expand your knowledge of these technologies and explore their potential in the world of data analytics, this webinar is not to be missed.

Watch webinar recording

In this article:

Hadoop

Hadoop is used to process and analyze large volumes of data efficiently by distributing the workload across a cluster of commodity hardware, enabling parallel processing and providing fault tolerance through its distributed file system and resource management framework.

HDFS – Hadoop Distributed File System

HDFS is a distributed file system that provides reliable and scalable storage for big data. It breaks large files into blocks and distributes them across a cluster of commodity hardware. HDFS ensures data reliability and availability through data replication.

Yet Another Resource Negotiator – YARN

YARN provides a flexible, scalable resource management framework for Hadoop, enabling a variety of applications and workloads to coexist and efficiently utilize the cluster’s resources. It abstracts the underlying infrastructure and allows for the dynamic allocation of resources based on application requirements.

MapReduce – MR

MapReduce is a programming model and processing framework for distributed data processing in Hadoop. It allows for parallel processing of large datasets by dividing the workload into maps, reducing tasks. Map tasks process data in parallel and the output is combined and reduced to produce the final result.

Hadoop Common

Hadoop Common provides libraries, utilities, and infrastructure support for the other components of Hadoop. It includes common utilities, authentication mechanisms, and interfaces that are used by various Hadoop modules.

What is the benefit?

Scalability
Hadoop enables the storage and processing of massive amounts of data by scaling horizontally across a cluster of commodity hardware. It can handle petabytes of data without sacrificing performance.

Distributed Computing
Hadoop distributes data and processing tasks across multiple nodes in a cluster, allowing for parallel processing and faster data analysis. This distributed computing model enables efficient utilization of resources and enables high-performance data processing.

Fault Tolerance
Hadoop provides fault tolerance by replicating data across multiple nodes in the cluster. If a node fails, data can still be accessed from other replicas, ensuring data reliability and availability.

Cost-Effectiveness
Hadoop is designed to run on inexpensive commodity hardware, making it a cost-effective solution for storing and processing large volumes of data. It eliminates the need for expensive specialized hardware.

Flexibility and Extensibility
Hadoop’s modular architecture allows for integration with various tools and frameworks within the Hadoop ecosystem, providing flexibility and extensibility. It supports a wide range of data processing tasks, including batch processing, real-time processing, machine learning, and more.

Data Locality
Hadoop’s distributed file system, HDFS, aims to bring the computation closer to the data. By processing data where it is stored, Hadoop minimizes data movement across the network, reducing latency and improving overall performance.

Ecosystem and Community
Hadoop has a rich ecosystem with a wide range of tools, libraries, and frameworks that extend its functionality for different use cases. It also has a large and active community of users, developers, and contributors, providing support, resources, and continuous improvement.

These benefits make Hadoop a powerful, popular solution for handling big data, enabling organizations to efficiently store, process, and gain insights from vast amounts of structured and unstructured data. The whole ecosystem can also run on-premise, which can make it a good alternative if ‘cloud’ is not an option.

HIVE

Hive is a data warehouse infrastructure built on top of Hadoop that provides a high-level SQL-like query language called HiveQL for querying and analyzing large datasets.

What are the components?

Data Storage
Hive leverages Hadoop Distributed File System (HDFS) as its underlying storage system. It stores data in HDFS in a distributed and fault-tolerant manner, allowing for scalable, reliable data storage.

Schema Definition
Hive allows users to define a schema for their data using a language called Hive Data Definition Language, like DDL. This allows users to define tables, partitions, columns, data types, and other metadata associated with the data.

Query Optimization
Hive optimizes queries by performing query planning and optimization techniques. It aims to generate efficient query execution plans to minimize data movement, optimize resource utilization, and improve query performance.

Hive Metastore
Hive maintains a metadata repository called the Hive Metastore. It stores information about the tables, partitions, schemas, and other metadata associated with the data stored in HDFS. The metastore allows for efficient metadata management and retrieval during query processing.

Extensibility
Hive offers extensibility through User-Defined Functions (UDFs), User-Defined Aggregations (UDAs), and User-Defined Table Functions (UDTFs). These allow users to define custom logic and operations in programming languages like Java, Python, or other supported languages.

Integration with other tools
Hive integrates with various other tools and frameworks in the Hadoop ecosystem. For example, it can work alongside Apache Spark, Apache Pig, Apache HBase, and other components to provide a complete data processing and analytics solution.

Partitioning and Bucketing
Hive supports data partitioning and bucketing, allowing users to organize and store data in a structured manner. Partitioning involves dividing data into logical partitions based on specific criteria, while bucketing involves dividing data into equally sized buckets based on hash values.

SerDe
Hive uses a serialization/deserialization framework called SerDe (Serializer/Deserializer) to read and write data in different formats, such as CSV, JSON, Avro, and more. Users can specify the appropriate SerDe for their data format to ensure proper data processing.

Overall, Hive simplifies data querying and analysis on Hadoop by providing a familiar SQL-like interface. It abstracts the complexity of writing low-level MapReduce or Tez jobs and provides a declarative and user-friendly approach to interact with large-scale data stored in Hadoop.

Conclusion

Hadoop is a robust and feature-rich environment that can be challenging to manage. However, its numerous advantages make it a compelling choice, depending on the user’s needs and the available in-house expertise. If you’re interested in learning more about it, watch the following recording.

In Intermediate

Utilizing Potentials of Data Vault 2.0 – Overcoming Bad Practices – Part 2

Watch the Webinar

Utilizing Potentials of Data Vault 2.0 – Overcoming Bad Practices – Part 2

What are common mistakes when applying Data Vault 2.0 in enterprise data warehouse projects? Do you have questions regarding modeling in Data Vault and the realization of GDPR causes you great difficulties or is your project stuck because you are delivering no business value?

This webinar describes common Anti-patterns of Data Vault, their consequences, and the solution to eliminate them from your current or in your future projects.

Tune in and learn more to avoid bad practices and apply simple solutions.

Watch Webinar Recording

Webinar Agenda

1. How to use Data Vault for modeling business information
2. How to avoid the pitfalls of being unable to deliver business value
3. How to mask Business Keys from Hubs for privacy

In Data Tools, Intermediate

Get Started with Real-Time Processing in Data Vault 2.0 on Microsoft Azure

Data Vault 2.0 on Microsoft Azure

In this newsletter, you’re going to get an overview of what real-time processing is and what possibilities it can provide your Data Vault 2.0 implementation.

Real-Time Processing with Data Vault 2.0 on Azure

In this webinar, we’ll discuss the new data warehouse requirements for data and explore
Real-Time processing. We’ll cover various Real-Time processing architectures for an initial overview. The second part focuses on Real-Time data architecture with Data Vault 2.0 and includes a brief overview of Microsoft Azure. You’ll also see a Real-Time processing implementation of Data Vault 2.0 in Azure. This webinar is for anyone new to Real-Time Data with Data Vault 2.0 and interested in an overview and implementation in Azure.

Watch Webinar Part 1Watch Webinar Part 2

In this article:

What to expect

You will learn that real-time processing gives you the ability to create value out of data quicker, have the most up-to-date data in your reporting tools and allow more accurate decisions regarding data.
With that, your company will be able to adapt to changes in the market quicker by seeing developments right away with the most recent data.

Additionally, you can save costs by moving away from batch loading because the peak of computing power normally required for that gets reduced and is more evenly distributed throughout the day. That is especially the case when using cloud environments, because then it’s possible to replace promised environments and contribute the needed computing power perfectly.

The traditional way – batch-loading

Batch loading is a traditional method used to load data into a data warehouse system in large batches, mostly overnight. The data from data sources is delivered up to a certain time in the night to be transformed and loaded into the core data warehouse layer.

This method leads to a peak of data processing overnight, and organizations have to adjust their infrastructure needs to be able to deal with the expected maximum peak of required computing power.

The new way – real-time data

Real-time data is processed and made available immediately as it is generated, instead of being loaded in batches overnight. When using real-time approaches, the loading window is extended to 24 hours. So the overnight peak and its disadvantages are gone.
When using real-time data, it’s always modeled as a non-historized link or as a satellite.

Possible use cases for real-time data are vital monitoring in the healthcare industry, inventory tracking, user behavior on social media or production line monitoring.

Different types of real-time data

There are different types of real-time data based on how frequently the data is loaded and the degree of urgency or immediacy of the data.

Near real-time data refers to data that is loaded in mini-batches at least every fifteen minutes, with the data stored in a cache until it is loaded into the data analytics platform.
Actual real-time data, also called message streaming, involves loading every single message directly into the data analytics platform without any cache.
This type of real-time data is useful when it is important to have data available as soon as it is generated for dashboards or further analytics.

The acceptable processing delay for real-time data is typically defined by the consequences of missing a deadline. Additionally, there are three types of real-time systems: hard real-time, soft real-time, and firm real-time.

Real-time processing types

Implementing real-time processing

So, how do you implement real-time data processing into your data warehouse solution? There are many architectures for that, but we will focus on the Lambda and Data Vault 2.0 architecture.

Generic real-time processing architecture

The lambda architecture separates data processing into a speed layer and a batch layer. The speed layer processes real-time messages with a focus on speed and throughput, while the batch layer provides accuracy and completeness by processing high volumes of data in regular batches. The serving layer integrates data from both layers for presentation purposes.

At first, the Data Vault 2.0 architecture seems to be similar to the lambda architecture, but it treats some aspects differently. The lambda architecture has issues from a Data Vault 2.0 perspective, such as implementing a single layer in each data flow and lacking a defined layer for capturing raw, unmodified data for auditing purposes.

The Data Vault 2.0 architecture adds a real-time part called “message streaming” to the existing batch-driven architecture, with multiple layers implemented for capturing and processing real-time data, integrating it with the batch-driven flow at multiple points. Messages are pushed downstream from the publisher to the subscriber, loaded into the Raw Data Vault and forked off into the data lake. But the main process is the push inside the message streaming area. The architecture is able to integrate data from batch feeds or to stream the real-time data directly into the dashboard.

Using Microsoft Azure for real-time processing

Microsoft Azure is a cloud computing platform and set of services offered by Microsoft. It provides a variety of services, including virtual machines, databases, analytics, storage, and networking. These services can be used to create web and mobile applications, run large-scale data processing tasks, store and manage data, host websites and much more.

Microsoft Azure for real-time processing

The illustration describes a typical real-time architecture used by Scalefree consultants, which follows the conceptual Data Vault 2.0 architecture.

Data sources deliver data either in batches or real-time. This is loaded into the Azure Data Lake or accepted by the Event Hub beforehand. The Raw Data Vault Loader separates business keys, relationships and descriptive data using Stream Analytics and forwards the message to the Business Vault processor. The Business Vault processor applies transformation and other business rules to produce the target message structure for consumption by the (dashboarding) application. The results can be loaded into physical tables in the Business Vault on Synapse or be delivered in real-time without further materialization in the database. The target message is generated and sent to the real-time information mart layer implemented by a streaming dataset, which is consumed by PowerBI. The cache of the dashboard service will expire quickly, but the Synapse database has all data available for other uses, including strategic, long-term reporting.

Conclusion

In conclusion, real-time data processing offers numerous benefits over traditional batch loading methods, including the ability to create value out of data quicker, have the most up-to-date information in reporting tools, and make more accurate decisions. By adapting to changes in the market quicker, companies can stay ahead of the competition. Moving away from batch loading can also save costs by reducing the peak of computing power required.

As mentioned before, the last illustration shows an architecture that the Scalefree Consultants implemented to make use of real-time data.

Read more on our recently released Microsoft Blog Article.

How is your current experience with real-time data processing?
Are you thinking about kick-starting your Data Vault by also using real-time data?
Or are you already using it and thinking about improving it further?

Let us know your thoughts in the comment section!

Close Menu