Skip to main content
search
0

Improving Salesforce Data Quality: Practical Solutions for Business Users

Fix Your Salesforce Data

Improving Salesforce Data Quality

Data is at the heart of every modern business. Organizations invest heavily in CRM platforms like Salesforce to manage customer information, support decision-making, and automate key processes. But even the most powerful CRM is only as good as the data it holds. Poor data quality leads to errors, delays, missed opportunities, and ultimately, lost revenue.

In this article, we explore the most common Salesforce data quality challenges, why they matter, and how business users—not just technical teams—can play a key role in keeping data accurate, consistent, and reliable. We’ll also share a step-by-step approach using Salesforce reports and dashboards to empower business teams in their daily operations.



Why Salesforce Data Quality Matters

Salesforce enables organizations to capture, store, and analyze customer information at scale. However, when data is incomplete, duplicated, or inconsistent, the value of Salesforce declines dramatically. Poor data quality often results in:

  • Incomplete reporting: Missing data fields prevent business teams from generating accurate reports and dashboards. This makes data-driven decision-making difficult or impossible.
  • Process errors: Incorrect values or misused fields can trigger workflow failures or lead to flawed outputs, causing business disruptions.
  • Delays in operations: Missing information, such as a shipping address, can halt critical business processes and create costly delays.
  • Automation failures: Flows, triggers, and integrations depend on complete and validated data. Poor-quality data leads to automation breakdowns and system errors.

The bottom line: without quality data, Salesforce cannot deliver on its promise of smarter sales, marketing, and customer service.

Typical Salesforce Data Quality Challenges

Across organizations, several recurring issues appear when it comes to Salesforce data quality:

  • Duplicated records: Multiple entries for the same account or contact create confusion, reporting inconsistencies, and wasted effort.
  • Missing key fields: Fields like industry, VAT number, or shipping address may be left blank, leading to gaps in reporting or process blockages.
  • Misused fields: Fields designed for one purpose may be repurposed by different teams, resulting in inconsistent data and unreliable reports.
  • Outdated information: Customer details can change frequently. Without regular updates, Salesforce quickly fills with stale data.

These issues are not unique to your company. They affect organizations of all sizes and industries. The key is to recognize that data quality is a continuous responsibility—not a one-time cleanup exercise.

Why Business Users Should Be Involved

Traditionally, data quality has been seen as an IT or admin responsibility. But in practice, many issues arise in day-to-day operations where business users interact with Salesforce directly. For example:

  • A sales rep forgets to mark an account as active.
  • A customer service agent skips entering a shipping address.
  • A marketing user enters inconsistent industry categories.

These small mistakes compound over time. By empowering business users to identify and correct data quality problems early, organizations can dramatically reduce long-term issues and keep processes running smoothly. The secret is to provide them with the right tools—without overwhelming them with technical details.

Using Salesforce Reports to Identify Data Gaps

Salesforce reports are one of the most effective tools for supporting business users in maintaining data quality. Reports can highlight records that fail to meet business requirements, enabling users to quickly spot and correct issues. Let’s walk through two practical examples.

Example 1: Accounts Missing the “Active” Field

Imagine that your business requires all accounts to have the “Active” field correctly set. However, during migrations or bulk uploads, many accounts are left blank. This creates reporting gaps when sales managers try to analyze active accounts.

By creating a simple report filtered to show accounts where “Active” is not set, you can generate a list of problem records. A designated business user can then review this report, update the missing values, and ensure reporting accuracy going forward.

Example 2: Missing Shipping Addresses on Closed-Won Opportunities

Another critical scenario involves shipping addresses. Suppose you have accounts with closed-won opportunities but no shipping address. This creates immediate risks for order fulfillment.

By building a report with a cross-filter (accounts with won opportunities AND missing shipping address), you can provide a focused list of problematic records. Assign this report to the operations or logistics team, and they can update shipping addresses before orders are delayed.

Creating Dashboards for Ongoing Monitoring

Reports are useful, but dashboards make monitoring even easier. You can combine multiple data quality reports into a single dashboard, categorized by department or data type. Examples include:

  • Sales Data Health: Accounts missing “Active” status, opportunities missing key fields.
  • Marketing Data Health: Leads missing industry or source information.
  • Service Data Health: Cases missing priority or escalation status.

Dashboards provide a real-time overview of data quality, helping managers track progress and ensuring accountability. Each team can take ownership of their specific data health metrics.

Best Practices for Business-Led Data Quality Management

To make this approach effective, keep the following best practices in mind:

  • Keep it simple: Reports and dashboards should be easy to read. Focus on the most critical data quality issues.
  • Assign responsibility: Make sure each report has an owner who is accountable for keeping it clear of records.
  • Explain the “why”: Always include descriptions that explain why a field matters. Business users are more likely to correct data when they understand its impact.
  • Automate where possible: Use validation rules, required fields, or automation to prevent errors before they enter the system.
  • Review regularly: Schedule regular reviews of dashboards to ensure data quality remains a priority.

Conclusion

Salesforce is a powerful platform, but it relies on accurate and complete data to function effectively. Data quality challenges—whether missing fields, duplicates, or outdated information—can significantly hinder decision-making and operational efficiency. The good news is that these challenges are solvable.

By empowering business users with simple reports and dashboards, you can shift data quality management from a reactive IT task to a proactive, business-led practice. This not only improves Salesforce performance but also fosters a culture of accountability across your organization.

Start small: identify a handful of critical fields, build focused reports, and create a simple dashboard. Over time, you’ll see measurable improvements in data health, process reliability, and business outcomes.

Remember: data quality is not a one-time project. It’s an ongoing effort—and when business users are equipped to take ownership, everyone benefits.

Watch the Video

Multi-Tenant Data Vault

Multi-Tenant Environment

Designing and maintaining a Data Vault in a multi-tenant environment presents unique challenges. When a data warehouse must handle not just internal data, but also data from dozens of external clients with slightly different processes and systems, the complexity increases dramatically.

A recent question we received highlighted this exact situation:

“I’m struggling with link management and the evolution process in a multi-tenant warehouse, especially putting all data together in the Information Mart. Our Data Warehouse contains internal data as well as shared data from our clients, for which we perform job requisition processes using their internal systems. We plan to onboard 50–60 clients in the next 2–3 years. Right now, we’re still in the MVP phase, supporting just a few clients. How should I manage links with so many different systems, such a large number of source tables, and processes that are similar but not identical? The goal is to have one common Information Mart design for all clients to enable standardized reporting.”

This is a classic question in modern data architecture. Let’s explore how to approach Raw Vault, Business Vault, and Information Mart design in a multi-tenant context.



Multi-Tenancy in the Raw Data Vault

A cornerstone principle in Data Vault modeling is that each Satellite is sourced from a single source system. However, in a multi-tenant setup, this guideline needs some adaptation. Many tenants use the same source systems (e.g., Salesforce, SAP) with similar core structures. In such cases, you can load multiple tenants into the same Satellite as long as you introduce a Tenant ID as part of the key.

Why Add a Tenant ID?

  • Ensures uniqueness of business keys across tenants (e.g., Customer 42 in Tenant A ≠ Customer 42 in Tenant B).
  • Partitions data naturally, so Satellites contain subsets per tenant without overwriting each other’s records.
  • Provides a straightforward way to filter or secure records by tenant.

By combining the local business key with the Tenant ID, you create a unique enterprise-wide business key. This guarantees data integrity while simplifying downstream querying and reporting.

Where to Add the Tenant ID

In multi-tenant designs, the Tenant ID should ideally appear:

  • Hubs: As part of the business key or alternate key, ensuring uniqueness across tenants.
  • Links: As part of the Hub references, ensuring uniqueness in combined relationships.
  • Satellites: As a payload field for convenience, even if the hash key already includes the Tenant ID.

With this approach, every record in the Raw Vault can always be traced back to a specific tenant, which simplifies not only modeling but also governance and security.

Defining the Tenant ID

A natural question arises: what exactly is a “tenant”? The answer depends on your business context:

  • It could be a client organization you serve.
  • It could be a business unit, country, or factory in large enterprises.
  • It might also be defined by data ownership—who is responsible for the dataset.

In some cases, you may also need a reserved Tenant ID for global or shared data that is not owned by any specific tenant. This ensures consistency and supports role-based access control.

Staging and Tenant Assignment

The Tenant ID is typically introduced already in the staging layer. How it’s assigned depends on the source system:

  • Tenant-dedicated systems: Assign a constant Tenant ID for all data from that system.
  • Multi-tenant systems (e.g., SAP, Salesforce): Extract and map the Tenant ID from existing fields (e.g., business unit, org ID).
  • Global systems: Use a reserved Tenant ID (e.g., “GLOBAL”) when ownership is shared or unclear.

This is a hard rule (constant assignment), not a conditional transformation, which ensures repeatability and traceability.

Business Vault in Multi-Tenant Contexts

Once Tenant IDs are embedded in the Raw Vault, the Business Vault becomes much easier to design. Business rules can be applied consistently across tenants, while preserving tenant-specific contexts.

  • Same-as Links: Crucial for resolving duplicate entities across tenants (e.g., the same customer appears in different client systems).
  • Custom Satellites: Standardize where possible, but add additional Satellites for tenant-specific customizations.
  • Wide PIT Tables: Be prepared for them—multiple tenants and diverse source systems naturally lead to broader structures.

At this stage, the goal is harmonization without oversimplification. A balance must be struck between common modeling and tenant-specific flexibility.

Designing the Information Mart

The Information Mart is where tenants—or the enterprise as a whole—derive insights. The challenge is to provide both:

  • Enterprise-wide views: Merging data from all tenants for global reporting.
  • Tenant-specific views: Allowing clients or business units to see only their data.

Common Mart Design

A single common dimensional model for all tenants reduces development overhead and supports standardized reporting. By including the Tenant ID in dimensions and facts, you can apply row-level security to restrict access per tenant.

When Separate Marts Are Needed

In some cases, specific tenants may require custom Information Marts. This is typically justified when:

  • Unique KPIs or processes cannot be expressed in the common model.
  • Legal or contractual reasons require strict separation of data.

However, these should remain exceptions. A well-designed common mart, filtered by Tenant ID, is usually sufficient for most tenants.

Role of Same-as Links in Reporting

To unify data across systems and tenants, Same-as Links are critical. These resolve entity duplicates across different tenants and systems (e.g., a product appearing under different codes in SAP and Salesforce).

Same-as Links can be sourced from:

  • Raw data: Mapping tables provided by business or source systems.
  • Calculated logic: Fuzzy matching, soundex, or other deduplication algorithms.

This harmonization enables the creation of enterprise-wide dimensions that span multiple tenants.

Security and Governance in Multi-Tenant Data Vaults

By embedding Tenant IDs throughout the model, row-level security becomes straightforward. Each record can be tied to a tenant, and access can be granted or denied accordingly. This simplifies compliance with data privacy regulations and contractual obligations.

Governance practices should also establish clear rules for:

  • Defining and maintaining Tenant IDs.
  • Managing ownership of global vs. tenant-specific data.
  • Regular audits of access controls and Same-as Links.

Best Practices for Multi-Tenant Data Vaults

  1. Add Tenant IDs early: Introduce them in staging to ensure consistency across the pipeline.
  2. Unify where possible: Standardize Satellites for common structures, customize only when necessary.
  3. Reserve global IDs: Create special identifiers for shared or unclear ownership data.
  4. Secure with Tenant IDs: Use row-level security tied directly to the Tenant ID field.
  5. Leverage Same-as Links: Resolve duplicates to support enterprise-wide reporting.
  6. Design one common mart: Rely on row-level filtering instead of duplicating models per tenant.
  7. Scale incrementally: Start with MVP, refine the model as you onboard new tenants.

Conclusion

Multi-tenant Data Vault design requires careful thought about uniqueness, ownership, and harmonization. By embedding Tenant IDs consistently across Hubs, Links, and Satellites, you not only preserve data integrity but also simplify governance and security. The Business Vault and Information Mart can then be designed to support both tenant-specific and enterprise-wide perspectives.

As organizations grow and onboard more clients or business units, this approach ensures scalability without overwhelming complexity. With clear governance, Same-as Links, and standardized mart designs, you can build a robust multi-tenant data warehouse that serves diverse needs while staying maintainable and secure.

Watch the Video

How to Get Your Data Platform Ready for Agentic AI

AI Agent Anatomy

Not long ago, simple large language models were the pinnacle of AI. Today, they can feel almost rudimentary, as the domain of artificial intelligence is rapidly evolves. Lately, we are seeing a push trying to move beyond one-off prompts and towards AI agents. 

It only makes sense that businesses are eager to incorporate AI agents into their workflows, and one domain particularly primed for such transformation is the data team. AI agents can automate repetitive tasks, streamline operations, and enhance data analysis and allow data professionals to focus more on the business side.

Future-Proofing your Data Platform and Unlocking its value as an AI Asset

Many companies investing in enterprise AI find success is limited by the quality of their data platforms. A key issue is “architectural debt,” which hinders the performance and scalability of AI initiatives. This session will provide guidance on how to identify and address these architectural challenges, helping organizations transform their data platforms into reliable assets that support AI agent workflows. Register for our free webinar, October 21st, 2025!

Watch Webinar Recording

AI Agents: A Brief Introduction

AI agents are autonomous software systems that perceive their environment, reason over data, and take actions to achieve specified goals. They leverage large language models, tool‑use frameworks, and API integrations to connect with external services from CRM platforms and cloud storage to data platforms and real‑time event streams. Unlike static models, agents can maintain memory across sessions, chain multiple model calls, and adapt their workflows based on real‑time feedback from connected systems.

The Anatomy of AI Agents

AI Agent Anatomy

Figure 1: A conversation agent built in low-code automation tool n8n.

An AI agent is typically centered around a large language model that serves as its core reasoning engine, interpreting user inputs, generating plans, and orchestrating decision-making through chain-of-thought or self-prompting techniques​. Surrounding this core is a memory structure that can span across immediate working memory, episodic logs, and semantic knowledge stores that persistently captures and condenses interaction histories​. To provide durable, structured storage and enable symbolic multi-hop reasoning, agents integrate databases (e.g., SQL, graph, or vector stores) as their internal memory substrate, issuing queries to organize, link, and evolve knowledge beyond the context window of the LLM​. Finally, AI agents orchestrate a suite of external tools ranging from RESTful APIs and code execution environments to web scrapers and domain-specific plugins to act upon the world, extend their cognitive reach, and execute actions in both digital and physical domains​.

A key limitation of relying on custom APIs as connectors in an AI agent framework is scalability: as you add more agents, tools, and integrations, maintaining a separate API connection for every tool and action soon becomes unmanageable. That’s where MCPs come in.

Model Context Protocols (MCP)

Figure 2: A diagram showcasing Model Context Protocols (MCP)

Developed by Anthropic and open-sourced in November 2024, the Model Context Protocol (MCP) functions as a standardized integration layer that enables the reasoning engine to interface with external resources​. It accomplishes this by defining a uniform client–server protocol whereby MCP clients (the AI agents) discover available services via a registry, authenticate, and invoke capabilities such as database queries, function calls, or file retrieval through RESTful endpoints​. By decoupling the LLM from tool-specific protocols, MCP fosters a modular ecosystem in which new services can be plugged in dynamically, making AI agent development much more scalable.

Build a Solid Data Foundation for Agentic AI

Enterprises that aim to integrate AI agents into their data workloads must first build a solid data foundation. According to a cybersecurity report, 72% of professionals state that IT and security data are siloed within their organizations, creating corporate misalignment and increased security risks. Likewise, an industry study found that in three out of four companies, data silos hinder internal collaboration, and more than 40% report a growing number of such silos. 

When data remains in isolated, non-integrated environments, AI agents cannot establish a holistic overview of the data landscape of an enterprise, hence severely limiting its abilities in making meaningful impact.

AI Agent Enterprise Data Platform

Figure 3: An Enterprise Data Platform diagram, with an EDW

To overcome this, it is best to unify data sources into an enterprise data warehouse (EDW). The EDW must provide both current and historical data in a single data platform. By functioning as a true EDW, the data platform provides a single source of facts for all agents and analytics engines. This means that the AI agents across the enterprise are empowered to create what is needed with the increased availability of data. At Scalefree, we believe that a robust and well-designed data model is foundational to building a scalable and resilient EDW, that supports both operational efficiency and long-term analytical agility.

Ensure Data Quality and Metadata Management

Data quality is already a key issue in data warehousing. Poor data quality can lead to inaccurate insights, flawed decision-making, and ultimately compromise business success. The effectiveness of AI agents is also directly influenced by the quality of the data they consume. Issues such as duplicate records, missing values, and inconsistent schemas can result in erroneous behavior or reduced performance. These issues can be addressed through systematic data cleaning processes and the implementation of data quality tests across ingestion and transformation pipelines. Ongoing monitoring should be in place to detect anomalies and trigger remediation actions where necessary. 

Metadata management also plays a role in agent effectiveness. Shared taxonomies and ontologies provide agents with a consistent framework for understanding data definitions across domains. Without standardized metadata, agents may cause errors in reasoning or communication due to misinterpreted values. Establishing a well-maintained data catalog and promoting organization-wide metadata standards supports both data discoverability and semantic consistency, which are essential in multi-agent environments.

Prepare for Real-Time Processing and Efficient Retrieval

AI agents do not strictly require real-time data, but having access to it can significantly enhance their performance and decision-making capabilities. Real-time data allows AI agents to be informed in quickly changing conditions and provide more accurate and relevant responses. To support this, data platforms can be set up to process streaming data or near-real-time updates.

Additionally, indexing strategies must accommodate both structured and unstructured data. When needed, structured data can continue to rely on traditional indexing methods such as inverted indexes. For unstructured content, embedding-based vector search provides agents with the means to identify semantically similar data points.

Large data objects should also be broken into manageable segments through chunking. This practice enables agents to retrieve and reason over smaller, contextually meaningful portions of data, which improves both performance and interpretability. Determining appropriate chunk sizes may require tuning to balance context with precision.

Implement Orchestration and Observability for AI Workflows

The introduction of AI agents into business processes necessitates a layer of orchestration that governs how agents collaborate, pass information, and handle dependencies. A multi-agent orchestration system should trigger the right agents for a given task, coordinate their outputs, and manage error handling or fallback logic. Orchestrators also need to support asynchronous communication where agents operate independently but contribute to a shared goal.

Monitoring and testing these workflows is essential. Agents can fail, drift from intended behavior, or interact in unintended ways. Logging, alerting, and automated feedback loops can be integrated into orchestration frameworks to surface and correct such deviations. Performance metrics such as response time, accuracy, and success rates should be tracked to ensure continued alignment with business objectives.

AI Agents as Identity-Bearing Entities

AI agents should be treated as identity-bearing entities within the enterprise architecture. This means granting them access only to the data and systems necessary for their assigned roles. To that end, just as any other employee, AI agents should abide by the principle of least privilege. Role-Based Access Control ensures that each agent’s data permissions are explicitly defined and enforceable. For example, an AI agent responsible for financial forecasting should not have access to sensitive HR data.

Integrating AI agents into existing identity and access management (IAM) systems can help enforce compliance and support auditability. Just as human users have roles and access policies, agents should be provisioned, monitored, and offboarded in a controlled and traceable manner.

Embrace a Data Mesh for Scalable Multi-Agent Workflows

Organizations expecting to deploy multiple AI agents concurrently should consider transitioning from a centralized end-to-end model to a data mesh. A data mesh distributes data ownership across domain teams and treats data as a product, aligning well with the modular nature of AI agents. This architecture allows agents to scale horizontally across business functions while maintaining domain-specific ownership of data pipelines and logic. Each agent can operate on a defined domain without depending on a centralized data engineering team, reducing bottlenecks and increasing agility. In environments with high agent interaction, domain-driven decentralization ensures that systems remain responsive and maintainable as usage grows.

Design for Modularity and Scalability

To scale the use of AI agents across business processes, data pipelines should be decomposed into independently deployable and maintainable components. This approach allows new agents or features to be added without having to duplicate or fork existing systems. Event-driven architectures, in which agents react to messages or state changes, support this level of decoupling and flexibility.

Agent-to-agent communication should be standardized using standardized protocols and contracts to allow agent-to-agent interaction predictably. By designing systems with modular interfaces and reusable components, AI agent ecosystems can grow in an agile, iterative fashion.

How to Remove Duplicate Records in Salesforce with Standard Tools

No More Duplicates

Deduplication with Salesforce Standard Tools

Duplicate data is one of the most common and damaging problems in any CRM system. Whether it’s from manual entry, marketing campaigns, or automated integrations, duplicates create chaos across sales, marketing, and reporting. The good news is that Salesforce provides powerful standard tools to identify and prevent duplicates without needing third-party applications.

In this article, we’ll explore why duplicate data is such a problem, the consequences it has on your business, and how you can use Matching Rules and Duplicate Rules in Salesforce to take control of your data quality.



Why Duplicate Data Happens in Salesforce

CRM systems are only as good as the data inside them. Unfortunately, data can enter Salesforce through many channels, making duplicates almost inevitable if you don’t have safeguards in place.

  • Manual input by sales or marketing team members
  • Web forms capturing leads from campaigns
  • API integrations with other systems
  • Automations such as Flows or imports

When these channels are not synchronized or when human error occurs, duplicate records slip into the system. Once they’re in, they can have ripple effects across every part of your organization.

The Consequences of Duplicate Data

The saying “garbage in, garbage out” applies directly to CRM systems. If your Salesforce environment is filled with duplicate data, the results can be disastrous.

  • Wasted Marketing Spend: Sending the same campaign multiple times to the same contact drives up costs and reduces ROI.
  • Lost Sales Opportunities: Sales reps waste time figuring out which record is the “real” one, slowing down the pipeline.
  • Poor Customer Experience: Customers receive duplicate or confusing communications, lowering trust and satisfaction.
  • Untrustworthy Reports: Business leaders make decisions based on flawed dashboards and KPIs, leading to bad strategy.

Put simply, duplicate data undermines every aspect of CRM performance. But with Salesforce’s standard tools, you can fix it.

Salesforce’s Standard Deduplication Tools

Salesforce provides two native features that help with deduplication:

  1. Matching Rules: Define the criteria that determines when two records should be considered the same.
  2. Duplicate Rules: Decide what happens when a match is found — block the action, allow with a warning, or report it.

Let’s go step by step through how these work in practice.

Step 1: Understanding Matching Rules

A Matching Rule is the logic that Salesforce uses to evaluate whether two records are duplicates. For example, Salesforce provides a standard Lead Matching Rule that checks for:

  • Exact matches on email address
  • Similar matches on first and last names

In many cases, the standard rules are enough. However, you can create custom matching rules to account for your organization’s unique data entry patterns. For example, you may want to consider phone numbers, company names, or other fields when evaluating duplicates.

Step 2: Creating Duplicate Rules

Once you’ve defined how Salesforce recognizes duplicates, you need to decide what to do about them. That’s where Duplicate Rules come in.

When setting up a Duplicate Rule, you’ll need to decide:

  • Which object the rule applies to (e.g., Leads, Contacts, Accounts).
  • What happens when a duplicate is detected:
    • Block: Prevents the duplicate record from being saved.
    • Allow but Alert: Lets the record be saved but notifies the user that a duplicate exists.
  • The alert message that users will see when duplicates are found.
  • The matching rule to use (e.g., Standard Lead Matching Rule).

For example, if you create a Duplicate Rule for the Lead object, you can block users from creating a new Lead when the email address already exists in Salesforce. This ensures you never have two records for the same prospect.

Step 3: Activating and Testing

After creating a Duplicate Rule, don’t forget to activate it. Once it’s active, Salesforce will enforce it every time someone tries to create or update a record.

A quick test is to try creating a record that you know already exists. Salesforce should either block the action or display your custom alert, depending on your configuration.

Practical Example

Let’s say you already have a Lead record for John Miller at GlobalTech with the email [email protected]. A sales rep accidentally tries to create a new record for Jon Miller (without the “h”) at the same company, using the same email address. Without rules, Salesforce would allow both records, creating confusion and duplicate communications.

But with Matching and Duplicate Rules in place, Salesforce will flag the record as a duplicate and prevent it from being saved. The sales rep sees an alert message explaining why, and the system stays clean.

Best Practices for Salesforce Deduplication

  • Start simple: Use Salesforce’s standard rules before creating complex custom ones.
  • Block when possible: Preventing duplicates at the source is more effective than cleaning them later.
  • Alert strategically: In some cases, like large imports, allowing but warning might be more practical.
  • Review periodically: Duplicate patterns can change as your business evolves. Review and adjust rules every few months.
  • Combine with data cleanup: If your system already has duplicates, consider a one-time cleanup before enforcing rules.

Beyond Standard Tools

While Salesforce’s standard tools cover most use cases, large enterprises or organizations with very complex data structures may benefit from advanced deduplication solutions, such as third-party apps. These tools offer fuzzy matching, cross-object detection, and automated merging capabilities. However, starting with Salesforce’s built-in features is the most cost-effective and straightforward way to protect your CRM data quality.

Conclusion

Duplicate data can cripple the effectiveness of your Salesforce CRM by wasting resources, confusing teams, and eroding customer trust. Thankfully, Salesforce provides out-of-the-box Matching Rules and Duplicate Rules to help you detect, prevent, and manage duplicates effectively.

By setting up these rules, you can ensure your CRM stays clean, your reports stay accurate, and your teams can focus on what matters most — engaging customers and closing deals.

Watch the Video

Installing and Managing Packages in Coalesce.io

Coalesce.io Package Management

In this article, we will guide you through managing packages within your Coalesce.io environment. We’ll cover everything from what packages are and why they are essential to the step-by-step process of installing, upgrading, and uninstalling them. By the end, you’ll have a clear understanding of how to leverage Coalesce’s marketplace to expand the capabilities of your data platform and streamline your development workflow.

Your data platform is a powerful tool, and while it comes with a robust set of built-in features, its true power lies in its expandability. This is where the Coalesce.io marketplace comes into play, offering a vast array of packages that can introduce new features and functionalities to your environment. Think of it as a toolkit that you can customize and grow to meet your specific needs, whether you’re implementing a Data Vault, integrating testing frameworks, or leveraging specific Snowflake functions.



Exploring the Coalesce.io Marketplace

Before we jump into the installation process, let’s take a quick look at the marketplace itself. When you open the marketplace, you’ll find different categories of packages designed to serve various purposes. These include:

  • Feature Packages: These can add new functionalities, such as leveraging Snowflake’s dynamic tables or integrating powerful tests for data quality.
  • Base Node Types: These packages introduce new node types that can be used to build your data warehouse, such as the Data Vault for Coalesce.io package, which provides specific nodes for hub, link, and satellite entities.
  • Advanced Deploy Packages: These help in managing and deploying your data pipelines more efficiently.

Each package listing provides key information, including its latest version, supported platforms (e.g., Snowflake, Databricks), release date, and a unique package ID. This ID is crucial for the installation process, as it tells Coalesce.io exactly which package you want to install. The description also offers valuable insights into the package’s features and how to use it, along with links to more detailed resources.

Step-by-Step Guide to Installing a New Package

The process of installing a new package is straightforward and can be done directly from your Coalesce.io environment settings. Here’s how you do it:

  1. Copy the Package ID: First, head to the marketplace, find the package you want to install, and copy its unique package ID. This is your key to the installation.
  2. Navigate to Settings: In your Coalesce.io environment, go to your project settings, and then to ‘packages’. You’ll see an overview of all the packages currently installed in your environment.
  3. Browse and Install: Click on the ‘Browse’ button. Here, you can paste the package ID you copied earlier. Coalesce.io will then fetch all available versions of that package.
  4. Select Version and Alias: Choose the version you want to install. It’s highly recommended to give your new package an alias. An alias is a custom name that helps you easily identify the package, especially if you have multiple versions or a large number of packages installed. For example, naming it Data Vault for Coalesce.io - v2.01 provides a clear distinction from an older version.
  5. Complete Installation: Click ‘Install’. The process might take a few moments. Once complete, Coalesce.io will confirm that the package is installed and provide links to view its new macros and node types.

The use of aliases is a best practice that helps you maintain a clear overview of which package and which version you are using, preventing confusion as your project grows.

Upgrading and Managing Package Versions

Upgrading a package is just as simple as installing a new one. The process is particularly important when a package you are already using receives an update with new features or bug fixes. Here’s the recommended best practice for a smooth upgrade:

  1. Install the New Version: Follow the installation steps outlined above to install the latest version of the package.
  2. Transfer Existing Entities: Go through your existing Coalesce.io entities (nodes) that are using the old package. You will see a clear indication of which package and version is being used. Switch the node type to the new, updated version. This process ensures that your existing workflows benefit from the new features and stability of the latest release.
  3. Review and Deactivate Old Node Types: In the package settings, you can also manage the visibility of node types. If you want to prevent accidentally using an older version, you can simply turn off the node types from the old package. This cleans up your workspace and ensures you are always building with the latest tools.
  4. Uninstall the Old Package: Once all of your entities have been successfully migrated to the new version, you can safely uninstall the old package. Coalesce.io will alert you if any nodes are still using the old version, preventing you from accidentally breaking your project. This is a critical step to keep your environment clean and efficient.

This systematic approach ensures a seamless transition and keeps your project on the cutting edge of Coalesce’s capabilities without any disruption.

Discovering New Macros and Capabilities

Beyond new node types, packages often come with a set of powerful macros. These are reusable snippets of code that can significantly speed up your development process. In your Coalesce.io settings, you can navigate to the ‘macros’ section to see all available macros, including those from your installed packages. This allows you to explore what the package can do under the hood and even integrate some of its functionalities directly into your own custom nodes.

For example, if a package includes macros for data quality checks, you can use these in your own custom SQL queries to ensure data integrity at various stages of your pipeline. This level of extensibility is what makes Coalesce.io such a versatile platform for modern data engineering.

Final Thoughts on Coalesce.io Package Management

In this article, we’ve walked through the entire lifecycle of a package in Coalesce.io. We’ve shown you how to navigate the marketplace, install a new package, and follow a best-practice process for upgrading your project. We also touched upon the importance of managing node types and exploring the powerful macros that come with packages.

The ability to extend and customize your data platform is a key advantage of Coalesce.io. By actively managing your packages, you can ensure that your environment is always up-to-date, efficient, and equipped with the tools you need to tackle any data challenge. Remember, a well-managed environment is the foundation for a successful and scalable data platform.

Watch the Video

Cost Factors in Implementing and Maintaining Data Vault 2.0

Cost Factors in Data Vault 2.0

Implementing a modern data platform is never a one-size-fits-all endeavor. Every company has unique requirements, legacy systems, and business needs. When it comes to Data Vault 2.0 (or more precisely, Data Vault 2.1), understanding the main cost factors early on can help organizations budget realistically and avoid painful surprises later. In this article, we will explore the typical phases of a Data Vault project, break down the major cost drivers, and share best practices for cost optimization and governance.



How a Data Vault 2.1 Project Looks Like

While no two projects are exactly alike, a Data Vault journey often follows a recognizable structure:

  • Training & Onboarding: Equip your team with the right skills through workshops and tool hands-on sessions.
  • Requirements Analysis: Define the first use case and design an architecture that matches requirements.
  • Architecture & Setup: Prepare the platform, establish automation, and agree on standards and conventions.
  • First Tracer Bullet Sprint: Deliver an end-to-end flow for one use case, ensuring the first business value is realized.
  • Next Sprints & Cost Optimization: Add data sources incrementally, monitor resource usage, and optimize for efficiency.

The key difference compared to traditional data warehouse projects? Instead of building layer by layer and waiting months for business value, Data Vault emphasizes sprints with early, visible results. This agile approach not only accelerates delivery but also makes cost management more transparent.

The Major Cost Factors

What drives costs in a Data Vault implementation? Broadly, there are three categories:

1. People

The largest expense in most data projects is people. Costs include developers, data modelers, business analysts, and ongoing maintainers. Skilled professionals are needed not only for implementation but also for optimization and support. Investing in training early can reduce errors and long-term inefficiencies, making this a cost that pays back quickly.

2. Architecture

Whether you deploy on-premises or in the cloud, the technical backbone of your Data Vault incurs costs. Expect expenses for:

  • Compute: Running queries, data transformations, and analytical workloads.
  • Storage: Staging areas, raw vault, business vault, and marts require structured storage planning.
  • ETL / ELT: Orchestration pipelines and integration layers that keep the system running smoothly.

3. Tooling

Tools for automation, governance, and project management also add to the bill. However, Data Vault’s standards lend themselves well to automation, reducing manual effort and long-term costs. Tools like dbt Core or Coalesce provide strong value, often at lower costs compared to legacy ETL suites.

Cost Optimization Strategies

Once the platform is running, cost optimization should not be an afterthought. Instead, it should be a guiding principle from the very beginning.

Define Responsibilities

Every instance, warehouse, or resource that incurs costs needs a clear owner. Without ownership, cloud resources often remain active long past their usefulness, silently increasing bills.

Set End Dates

Many dashboards and data pipelines are built for temporary projects. Without end dates, they keep consuming compute and storage. Assign a sunset date for every resource and re-evaluate its necessity over time.

Use Tags for Transparency

Cloud platforms allow tagging by project, department, or cost center. This makes it easier to allocate expenses and understand who is using what. Clear tagging also improves accountability and enables granular reporting.

Define Purpose

Every instance, pipeline, or report should have a clear business purpose. If you cannot state who benefits from it and why, it is a strong candidate for decommissioning.

9 Best Practices for Cost Monitoring

Effective cost management requires discipline. These nine practices provide a structured approach:

  1. Involve Stakeholders: Ensure business and technical stakeholders understand cost implications.
  2. Set Up Budget Alerts: Get notified when costs exceed defined thresholds.
  3. Use Tags for Resources: Track usage by cost center, project, or department.
  4. Create Cost Dashboards: Tools like Snowsight provide real-time insights.
  5. Enable Usage Tracking: Know who uses which resources, and why.
  6. Review Allocations: Regularly audit and rebalance resource usage.
  7. Monitor Queries: Optimize inefficient SQL to cut unnecessary costs.
  8. Optimize Warehouses: Use auto-suspend/resume and right-size compute.
  9. Optimize Storage: Leverage zero copy cloning and transient tables to save space.

The Pareto Principle in Cost Saving

Not all cost optimizations are equal. According to the 80/20 rule, 20% of resources often account for 80% of costs. Identifying and addressing these high-impact areas—such as a handful of long-running queries—can unlock significant savings with minimal effort.

How Data Vault 2.0 Helps Reduce Costs

Beyond traditional cost-cutting measures, Data Vault 2.0 itself provides structural advantages that reduce expenses:

  • Automation: Standardized entities make it possible to automate much of the raw vault, lowering developer workload.
  • Agile Development: The tracer bullet approach allows incremental delivery of business value, avoiding expensive rework.
  • Auditing & Compliance: Built-in historization and auditability support GDPR compliance, preventing costly legal issues.

Conclusion

Estimating the exact cost of a Data Vault 2.0 implementation is impossible—each project has unique factors. However, by recognizing the primary cost drivers (people, architecture, tooling), adopting disciplined cost management practices, and leveraging the automation and agility inherent in Data Vault 2.0, organizations can keep their projects efficient and cost-effective.

Cost optimization is not a one-time activity. It’s an ongoing process of review, accountability, and continuous improvement. With the right governance and monitoring in place, Data Vault 2.0 is not only a robust data architecture—it’s a cost-conscious one too.

Watch the Video

Integration from WooCommerce to Salesforce (with AWS)

Salesforce WooCommerce Implementation

WooCommerce: A Robust E-commerce Solution

WooCommerce, a robust e-commerce solution, is built upon the open-source WordPress platform. This versatile platform empowers businesses to not only sell products but also offer training sessions and organize events seamlessly. Data generated from these transactions is stored in a structured database. While some user-generated information is accessible through the platform’s backend, a comprehensive view often requires extensive manual searching.



The Advantages of Salesforce Integration

The integration of Salesforce into this equation introduces a plethora of advantages. Salesforce is renowned for its capabilities in managing customer relationships, tracking buyer information over time, facilitating the distribution of marketing materials, and streamlining the invoicing process. By consolidating these functions within a unified system, businesses can significantly enhance their operational efficiency.

Our Integration Journey

Faced with the challenge of seamlessly integrating WooCommerce and Salesforce, our dedicated team embarked on a meticulous journey. This endeavor commenced with an in-depth assessment of the business requirements and a thorough evaluation of available plugins. Subsequently, our decision was to develop a custom integration solution, tailored to our specific needs.

Leveraging WooCommerce Webhooks

The core of our integration strategy hinges upon WooCommerce’s invaluable feature known as Webhooks. In this context, Webhooks enable the automatic triggering of events, such as the creation of an order, which subsequently initiates a POST request to a designated service. This service, in our case, takes the form of an AWS Lambda function script. Through this implementation, we achieved the ability to transmit order data seamlessly from WooCommerce to Salesforce.

Salesforce WooCommerce Implementation

Business Benefits of the Integration

This integration offers a myriad of advantages to our business users. Salesforce data becomes accessible within their purview, enabling them to monitor, update, and transform information into vital components such as contacts, accounts, and opportunities. Furthermore, this integration seamlessly connects to an automated billing system, enhancing the financial operations of the business.

Get in Touch With Us

If you seek further insights into the intricacies of our WooCommerce to Salesforce integration or wish to explore the possibilities of implementing a similar service for your business, we welcome you to initiate a conversation with us. Our team is readily available to address your inquiries and discuss how this integration can be tailored to meet your unique needs. Please do not hesitate to contact us through the provided channels below, and we will be delighted to assist you in optimizing your business processes.

Complex Computed Satellites in Data Vault

Complex Computed Satellites

When people first learn about computed satellites in Data Vault, they often encounter very simple examples: concatenating first and last names into a full name, or applying a basic calculation within a satellite. While these examples are valid, they don’t capture the full breadth of what computed satellites can do. In reality, computed satellites are a powerful mechanism for integrating, transforming, and enriching data across your vault — enabling business-driven insights while maintaining the Data Vault principles of auditability and traceability.

This article will walk through the broader concept of computed satellites, discuss how they are designed, and provide practical implementation patterns for handling more complex use cases.



What is a Computed Satellite?

At its core, a satellite in Data Vault is a structure that describes a business object (a hub or link) by holding descriptive attributes over time. A computed satellite differs from a raw satellite because its data does not come directly from the source system but is derived through business logic.

Examples include:

  • Concatenating FirstName and LastName into FullName.
  • Deriving an age from a birthdate.
  • Producing calculated scores, risk categories, or classifications.
  • Integrating attributes from multiple satellites across different hubs via links.
  • Creating artificial relationships, such as product recommendations based on purchase history.

Importantly, a computed satellite isn’t just about the calculation itself — it’s about what the result describes and where it logically belongs in your model.

Step 1: Defining the Parent Entity

Before you build a computed satellite, you must answer a critical question: What does the result describe?

Every satellite attaches to either a hub (a business key) or a link (a relationship between keys). If your calculation produces attributes describing a customer, then the computed satellite belongs on the Customer Hub. If it describes a relationship between customers and products, it belongs on the respective link.

For example:

  • A Full Name attribute describes a Customer Hub.
  • A product recommendation score describes a Customer–Product Link.
  • A risk category for an account describes an Account Hub.

This step ensures that your computed satellite stays aligned with the business meaning of your Data Vault model.

Step 2: Designing the Structure

Once you know the parent, the next step is to decide the structure of your results. Computed satellites can contain:

  • Simple attributes (e.g., strings, numbers, dates).
  • Multiple descriptive fields derived from logic.
  • Semi-structured data, such as JSON or XML.

For example, you might calculate a JSON object capturing a customer’s segmentation profile, or an XML document describing a product configuration.

The important point: the satellite reflects the structure of your results, not the mechanics of how you implemented them.

Step 3: Implementing the Business Logic

After modeling comes implementation. Computed satellites can be populated in several ways:

SQL Views

The most common approach is to implement a computed satellite as a SQL view. Here, the SQL query both expresses the logic (e.g., joins, transformations, calculations) and defines the result structure. If SQL is sufficient for your business rules, this is often the simplest and most maintainable approach.

External Scripts (Python, R, etc.)

For more advanced transformations, machine learning, or statistical processing, you may use external code. A Python script, for example, could pick up data from raw satellites, apply complex algorithms, and write results back into a computed satellite.

The golden rule: the implementation must remain under your control. Even if a data scientist creates an initial model using tools like Azure ML or RapidMiner, once it becomes part of your Business Vault, the deployment and maintenance are governed centrally. This ensures auditability and consistency.

Materialized Tables

Sometimes, business logic requires intermediate storage. In this case, you may materialize computed satellites as physical tables populated via INSERT statements or stored procedures. This is useful for performance optimization or managing dependency chains in cascading business rules.

Complex Use Cases for Computed Satellites

1. Filtering or Subsetting Business Keys

Imagine a Partner Hub with a single satellite. Business users may want to see only clients, employees, or vendors. Computed satellites can create filtered subsets that bring the model closer to business expectations. While not always the cleanest design, this is a practical option in some industries, such as insurance.

2. Artificial Links

A link doesn’t always need to come directly from a source system. You can create artificial links based on computed relationships. For example, by analyzing purchase history, you might generate product recommendations — effectively creating a Customer–Product Recommendation Link.

3. Cascading Business Rules

A powerful pattern is to break complex logic into smaller, reusable steps:

  1. Create a simple computed satellite that performs data cleansing or a basic calculation.
  2. Use that result in a second computed satellite to apply additional rules.
  3. Join results with other business vault entities to build richer attributes.

This cascading approach makes rules easier to maintain, document, and reuse — and avoids giant, unmanageable SQL queries filled with dozens of CTEs.

Best Practices

  • Start with the business meaning: Always clarify what the result describes before modeling.
  • Keep business logic in the Business Vault, not in downstream marts.
  • Favor cascading rules over monolithic transformations — it improves maintainability and reusability.
  • Control the code: All scripts, views, and procedures must be owned by the data warehouse team, not end-users.
  • Support multiple technologies: SQL for straightforward logic, external scripts for advanced logic, and materialized tables where necessary.

Dependencies and Execution

When you cascade rules or materialize results, you introduce dependencies. One entity must load before another. To manage this, many teams implement dependency tables that track loading order. This enables recursive or automated job scheduling, ensuring consistency across the Business Vault.

Virtualized approaches (SQL views) are often easier, since query optimizers can resolve dependencies dynamically. Materialized approaches, however, provide better performance and control at scale.

Why Computed Satellites Matter

Computed satellites are more than “extra calculated fields.” They enable organizations to:

  • Bridge the gap between raw data and business expectations.
  • Implement business rules in a controlled, auditable environment.
  • Support advanced analytics and machine learning workflows inside the Data Vault framework.
  • Enable modular, reusable logic that scales across domains and use cases.

By treating computed satellites as first-class citizens in your Business Vault, you ensure that business logic is not scattered in marts, reports, or ad hoc scripts — but is instead centralized, governed, and reusable.

Conclusion

Computed satellites in Data Vault can be as simple as a concatenated name, or as complex as multi-step cascading business rules that derive artificial relationships. The key is to start by identifying what your result describes, attach the satellite to the correct parent, design the structure of your attributes, and then implement the logic in a controlled, maintainable way.

Whether implemented via SQL, Python scripts, or materialized processes, computed satellites should remain under the stewardship of your data warehouse team. By following best practices, you’ll unlock the full potential of the Business Vault — keeping it business-aligned, auditable, and ready for advanced analytics.

Watch the Video

The Power of Data Contracts: From Data Chaos to Cohesion

The Power of Data Contracts

Have you ever had that feeling, the one where you wake up on a Monday morning and a familiar sense of dread washes over you? You get to your desk and hope against hope that no data pipeline has failed overnight, no dashboard has broken, and no server has crashed. For anyone working with data, this scenario is all too common. The modern data landscape is a sprawling, interconnected web where a small change in one area can trigger a cascade of failures downstream. A simple column rename, a change in data type, or an unexpected null value can bring a whole system to a grinding halt.

You spend your morning firefighting—analyzing the issue, pinpointing the source of the error, and scrambling to get everything back online. By the time you look at the clock, it’s lunchtime, and you’ve spent your entire morning just fixing a bug.

This chaos is exactly what a data contract is designed to solve. It’s a way to bring order to the madness, to create a foundation of trust and reliability. A data contract not only speeds up the bug-fixing process but also makes development and changes much easier, fostering a sense of accountability within your data teams.



What Exactly is a Data Contract?

Think of a data contract as a formal, machine-readable agreement between data producers and data consumers. It’s a pact that defines the expectations and promises between different teams in your organization. Imagine a sales dashboard team (the consumer) relying on data generated by the data engineering team (the producer). The data contract defines exactly what the data engineering team will deliver, creating a clear and reliable relationship.

Data Contract flow

While a data contract can be as detailed as needed, there are three core elements that should always be included.

1. Schema

The schema is the blueprint of your data. It defines exactly what your data will look like. This includes column names, data types, and the structure of the data. A data contract should define this schema and any potential schema changes, no matter how small. A minor change, like renaming a column, can easily break a downstream pipeline if it’s not communicated and managed properly. The schema element of the contract ensures that everyone is on the same page about the data’s structure.

2. Data Quality

Data quality is a crucial, yet often underestimated, aspect of data management. Your data contract should define data quality expectations that both producers and consumers can agree on. For example, a data warehouse team might require that a customer_id column in a source system table never be empty or null. A reporting team, on the other hand, might require that the quantity of an order never be zero. These are simple examples, but defining these expectations upfront prevents many common data problems.

3. Service Level Agreement (SLA)

An SLA is a promise that one party makes to another. In the context of a data contract, it can cover a variety of things. How quickly should a problem be fixed? How fresh does the data need to be (daily, weekly, real-time)? You can also use SLAs to manage changes. For instance, an SLA could stipulate that if the engineering team wants to rename a column, they must notify consumers one week in advance. This gives the dashboarding team time to implement the change in their reports before the new version goes live, ensuring a smooth transition without breaking anything.

Implementing Data Contracts in Practice

A data contract shouldn’t be a static PDF document that nobody uses. For it to be truly effective, it must be machine-readable and integrated into your daily workflow. Here’s how you can make that happen:

Automation is Key

Your data contract should be tested automatically against your data to ensure it’s being followed. You should also have automation in place for managing changes. For example, if a data producer updates the contract with a schema change, an automated process could send a notification to the data consumers. This automation makes people accountable for their data products. It ensures that any changes, even if they have a valid reason, are communicated clearly and don’t cause unexpected issues.

CI/CD Pipelines

You can integrate data contract checks into your Continuous Integration and Continuous Delivery (CI/CD) pipelines. Before a new deployment goes live, the pipeline can check if the changes adhere to the data contract. If they don’t, the deployment can be blocked. This prevents contract-breaking changes from ever reaching production.

Fostering Communication

While automation handles much of the communication, the ultimate goal is to foster a culture of collaboration. A data contract shouldn’t be a tool for finger-pointing (“They made the problem!”). Instead, it should be a framework that encourages teamwork, where everyone is working together to build reliable, trusted data products.

The Benefits of Data Contracts

Implementing data contracts might sound like a lot of work, especially the automation part, but the benefits are substantial:

  • Increased Developer Time: Automated testing and CI/CD pipelines significantly reduce the time spent on bug-fixing and troubleshooting. Your teams can focus on development and innovation instead of firefighting.
  • Data Reliability: With clear definitions and automated checks, your data becomes much more reliable. People can trust the data they are using, and they can easily check the contract to understand its quality and refresh schedule.
  • Autonomy: Data contracts enable autonomy. Teams can make changes and improvements without fear of breaking something downstream. They know that if a change is needed, the automated process will notify the right people, and everything can be managed safely and securely.

This newfound autonomy allows for a more dynamic and responsive data ecosystem. Teams are no longer afraid to innovate because they have a clear, safe process for doing so.

Getting Started with Data Contracts

If you’re ready to start, don’t try to tackle everything at once. Begin with a single use case—a small, easy-to-manage dataset. The goal is to test the process, not to solve every problem overnight.

  1. Start with Collaboration: Explain the benefits to your teams and get them working together. Don’t frame data contracts as a top-down mandate. Instead, show them how this will make their lives easier and their work more effective.
  2. Automate Everything: This is a critical step. Bring in DevOps expertise to help you build out automated testing and CI/CD pipelines. Look at the testing you already have in place and see how you can build on it.
  3. Remember the Culture and the Tech: Data contracts are both a cultural shift and a technical one. A PDF document alone won’t solve your problems. You need the technical implementation—the automation, the testing—to make the cultural shift truly stick.

Data contracts are a powerful tool for transforming your data landscape from a state of chaos to one of cohesion and trust. They empower your teams, increase data reliability, and free up valuable time for innovation.

Watch the Video

Data Vault Hashing on Databricks with XXHASH64

Hashing on Databricks

Hashing is a core element of Data Vault modeling. Hash keys are used to uniquely identify Hubs, Links, and Satellites, and they need to be consistent, reproducible, and efficient. A common debate is whether to use widely adopted hashing algorithms like MD5 or SHA-1, or to opt for faster and smaller hash functions such as xxhash64, which Databricks natively supports.

The question is simple: What if we stored hash keys as 64-bit integers (int64) using xxhash64 instead of 128-bit MD5 values? On the surface, this looks attractive — faster generation, better join performance, and reduced storage. But as we’ll explore in this article, the trade-offs around collisions, scalability, and platform independence make this a risky choice in Data Vault 2.0 architectures.



Why Smaller Hash Keys Look Tempting

There are clear benefits to using smaller hash values such as int64:

  • Speed of generation: Algorithms like xxhash64 or CRC-64 are significantly faster than MD5 or SHA-1. OpenSSL and Linux benchmarks consistently show xxhash64 outperforming older cryptographic functions.
  • Join performance: Joining 64-bit integers is naturally faster than joining 128-bit binary or 32-character string columns.
  • Storage efficiency: An int64 hash key requires only 8 bytes, compared to 16 bytes for binary MD5 or 32 bytes for an MD5 stored as a string.

From a performance perspective, the appeal is undeniable. But in Data Vault, performance is not the only concern. The fundamental question is: How safe are these smaller hashes when used as surrogate keys at scale?

The Risk of Collisions

A hash collision occurs when two different inputs produce the same hash value. In Data Vault, this means two different business keys could be treated as the same Hub, Link, or Satellite record — corrupting your data integrity.

With 64-bit hashes, the number of possible unique values is 2^64. While that sounds huge, probability tells a different story when you start loading millions or billions of rows. The so-called birthday paradox makes collisions far more likely than intuition suggests.

For example:

  • With just 10,000 records, a 32-bit hash already has a 1 in 100 chance of collision.
  • A 64-bit hash greatly reduces the risk, but at large scales (hundreds of millions of rows), the probability becomes uncomfortably high.
  • A 128-bit hash (MD5) pushes collision risk into the realm of trillions of rows before it becomes statistically relevant.

That’s why MD5 and SHA-1 — despite being slower — are still standard in Data Vault: they provide mathematically safe keyspace sizes for large enterprise datasets.

Performance vs. Integrity

The argument for int64 hashing often emphasizes query performance:

  • Joins on numeric columns are faster than joins on strings or binary.
  • xxhash64 is faster to compute than MD5.

These points are true. But in practice, the cost of a collision far outweighs the performance gains. A single collision can undermine the entire lineage of your data warehouse. Once data integrity is compromised, every downstream analytic and report is suspect.

As a result, most Data Vault practitioners will gladly accept the slightly higher CPU cost of MD5 or SHA-1 in exchange for peace of mind.

Platform Portability Matters

Another issue with xxhash64 is platform availability. While Databricks supports it, you also need to consider:

  • Can you compute xxhash64 in Snowflake, BigQuery, SQL Server, or Oracle?
  • Can you reproduce xxhash64 consistently in Python, Java, or ETL tools?
  • Will the algorithm be supported 5–10 years from now?

One of the strengths of MD5 and SHA-1 is their ubiquity. They are implemented in nearly every database, programming language, and ETL platform. This cross-platform reproducibility is essential when building a Data Vault that may span multiple systems. By contrast, xxhash64 locks you into Databricks (or requires custom implementations elsewhere).

Why 128-Bit is the Safe Minimum

Let’s compare hash sizes:

  • CRC-32: Very fast, but collisions appear after ~10,000 records. Unusable for Data Vault.
  • xxhash64 / CRC-64: Better, but collisions become likely as datasets grow into the hundreds of millions.
  • MD5 (128-bit): Standard choice. Safe up to trillions of rows.
  • SHA-1 (160-bit): Provides even more headroom for extremely large datasets.

For most enterprises, MD5 hits the sweet spot: fast enough, widely supported, and statistically collision-free at realistic data volumes.

Alternative Approaches

If performance is a serious concern, consider these approaches instead of shrinking hash size:

  • Binary storage: Store MD5 as a 16-byte binary instead of a 32-character string. This cuts storage in half and improves join performance.
  • Partitioning strategies: Optimize joins by partitioning your Data Vault tables, reducing the need for full-table joins.
  • Hardware acceleration: Modern CPUs have optimized instructions for MD5 and SHA-1, making them faster than you might expect.
  • Consider SHA-256 only if required: While SHA-2 offers stronger guarantees, it’s rarely necessary in Data Vault and adds performance overhead.

Summary: Should You Use xxhash64 in Data Vault?

While xxhash64 looks attractive in Databricks because of its speed and smaller footprint, it’s not a good fit for Data Vault 2.0:

  • Collision risk is too high for large-scale data warehouses.
  • Portability is limited — you risk vendor lock-in.
  • Long-term maintainability suffers if your algorithm isn’t standardized across platforms.

In Data Vault, data integrity always comes first. That’s why MD5 (128-bit) or SHA-1 (160-bit) remain the recommended standards for hash keys. They provide the balance of performance, portability, and collision safety needed for enterprise-scale solutions.

Conclusion

If you’re working in Databricks, it might be tempting to adopt xxhash64 for hash keys. But resist that temptation. The risks of collisions and platform lock-in far outweigh the benefits. Stick with MD5 or SHA-1 for your Data Vault hash keys, store them efficiently as binary values, and optimize performance through storage and join strategies.

Remember: a faster broken key is still a broken key. In Data Vault, correctness and consistency are always the highest priority.

Watch the Video

Joining SCD2 Tables Using Data Vault

Joining SCD2 Tables

When working with Data Vault 2.0, one of the most common challenges is how to handle Slowly Changing Dimension type 2 (SCD2) tables when loading Link Satellites. Imagine a scenario where you need to join three SCD2 tables, each with valid_from and valid_to dates, and bring them together into a single Satellite hanging from a Link. At first, this might sound straightforward, but the details matter — and depending on your approach, you could make your warehouse harder to maintain, less performant, or less flexible.

In this article, we’ll walk through the best practices for handling this situation, based on Data Vault principles. We’ll discuss why you shouldn’t rush into joining multiple SCD2 tables into one Satellite, how to handle business timelines, the role of PIT and T-PIT tables, and strategies for keeping your design scalable and future-proof.



Understanding SCD2 in the Context of Data Vault

Slowly Changing Dimensions type 2 are a way to track historical changes in dimensional data. Each record typically has valid_from and valid_to dates that describe its period of effect. In a Data Vault model, however, the focus isn’t on interpreting those business dates upfront. Instead, the Raw Data Vault stage is all about capturing what the source gives you, as-is.

That means when we first bring data into the Raw Vault, the valid_from and valid_to fields should be stored as descriptive attributes — not as part of the Satellite’s primary key. Trying to interpret and align them too early will only create unnecessary complexity.

Why Not Load One Satellite From Three SCD2 Tables?

At first glance, it may seem attractive to combine all three SCD2 tables directly into one Satellite. But this goes against Data Vault best practices:

  • Each Satellite should usually source from a single table or source system.
  • Denormalizing multiple sources into one Satellite complicates your Raw Vault and makes it harder to maintain.
  • You risk having to undo the denormalization later when new requirements come in.

The recommended approach is to create at least three Satellites — one per SCD2 table. You may even need more Satellites if you have to split them based on privacy, rate of change, or security rules. This simplifies your Raw Vault and sets you up for flexibility later.

Handling Validity Dates in Satellites

Once your Satellites are created, the question is how to handle valid_from and valid_to. Here are two key approaches:

1. Multi-Active Satellites

If your SCD2 records represent multiple simultaneously valid states (for example, different price lists where both current and future prices are valid), you can use a multi-active Satellite. In this design:

  • The primary key is composed of hash_key + load_date + subsequence.
  • valid_from and valid_to are stored as descriptive attributes, not as key parts.
  • A staging-generated subsequence ensures uniqueness within a load.

2. Standard (CDC) Satellites with Subsequence

If only one record is active at a time (the classic SCD2 case), then you don’t need a multi-active design. Instead:

  • Stick with the standard Satellite primary key: hash_key + load_date.
  • Handle multiple intraday changes by adding micro- or nanosecond subsequences to load_date.
  • This ensures only one active row per parent key at any given time.

Choosing between multi-active and CDC-style Satellites depends entirely on your data. Do you need multiple simultaneously valid records? Or does one replace the other in sequence? Your answer determines the right design.

Aligning Data with PIT Tables

Once the three Satellites are loaded, you’ll need to bring them together for reporting. This is where PIT (Point-in-Time) tables come in. PIT tables align deltas across Satellites to a common snapshot date, making it possible to present a unified view of related changes.

There are two main flavors:

  • Standard PIT – aligns data based on load_date (technical historization).
  • Temporal PIT (T-PIT) – aligns data based on business timelines like valid_from and valid_to.

A T-PIT lets you activate rows based on both the technical and business timelines. However, it comes with maintenance challenges: whenever valid_from or valid_to dates change, you may need to rebuild PIT partitions — which can be expensive if large date ranges are altered.

Where Should You Apply Business Timelines?

There are three strategies for applying business timelines like valid_from and valid_to:

  1. Downstream in reporting or dashboards – simplest to maintain, and business users can define how timelines should be applied.
  2. In dimension views – apply filters and conditions directly in the SQL layer that feeds reports.
  3. Upstream in PIT/T-PIT – most performant but requires heavier maintenance whenever business timelines change.

A practical approach is to start downstream and only move timeline application upstream if performance issues demand it.

Building a Business Satellite

Once your PIT table aligns the Satellites, you may still need a Business Satellite (or Computed Satellite). This Satellite:

  • Uses the PIT table as its foundation.
  • Combines attributes from the three original Satellites using COALESCE or other business rules.
  • Optionally applies T-PIT logic if strict business timeline alignment is required.

By separating Raw Satellites (which store raw source data) from Business Satellites (which apply interpretation and business logic), you preserve Data Vault’s flexibility while still meeting analytical requirements.

Best Practices Recap

  • Create one Satellite per SCD2 source table — don’t denormalize too early.
  • Treat valid_from and valid_to as descriptive attributes in the Raw Vault.
  • Use multi-active Satellites only when multiple records are simultaneously valid; otherwise, stick with standard CDC Satellites.
  • Align Satellites with PIT tables; consider T-PITs only if business timelines must be applied upstream.
  • Whenever possible, push business timeline interpretation downstream to dimensions or reports for easier maintenance.
  • Use Business Satellites when you need to merge multiple sources into a single unified view.

Conclusion

Joining multiple SCD2 tables into a single Satellite is rarely the right first step in a Data Vault 2.0 implementation. Instead, build your Raw Vault Satellites closely aligned with their sources, use PIT tables to align changes, and apply business timelines carefully — starting as far downstream as possible. This approach keeps your architecture maintainable, flexible, and scalable while still supporting complex historical analysis.

By following these principles, you’ll not only simplify your data model but also give your business users the power to interpret validity ranges in ways that make sense for them — without locking your warehouse into rigid rules that are hard to maintain.

Watch the Video

Monitoring of a Snowflake Powered EDW

Snowflake Built-In Tools

Introduction to Snowflake

Snowflake established itself as one of the most widely used cloud data platforms, providing a scalable and flexible architecture for building, maintaining, and operating Enterprise Data Warehouses. While it aims to eliminate data silos and simplify the data structure, with billions of queries a day (overall), its elastic computing handles the workload for good performance and a satisfied customer.

However, as organizations rely increasingly on Snowflake, monitoring their solution might be crucial for performance optimization, error and root cause analysis, and cost control. Although Snowflake handles these tasks very well on its own, experience shows some need for human interference when it comes to more complex queries, to increase performance, lowering the runtime without upscaling or scaling out.

This blog article aims to provide general information about and technical insights into Snowflake. Focusing on techniques as well as built-in tools to analyze bottlenecks, we aim to potentially increase performance and therefore decrease unnecessary costs.



Snowflake Built-In Tools

Query History

Starting with Snowflake’s built-in tools, we will dive into its roots and combinations of several metrics.

Monitoring query performance includes several depths of insight, from an overview of the general query history, down to the performance of each individual step within a query.

Snowflake Built-In Tools

Let’s assume the top level as an overview of our queries. Including metrics such as:

  • Start Time,
  • Total Time,
  • Status and
  • Query Type.

For the first level, we can gain insights when the queries start, how much time they were consuming, whether they succeeded or failed, or even are still running. Lastly, we can see the type of query, such as ‘CREATE TABLE’, ‘INSERT’, or ‘SELECT’. At this level, a brief overview could be created about the traffic within the Warehouse.

When hearing about this, you may be familiar with a built-in tool called ‘Query History’, which tracks this kind of information automatically without the need for user interference. It is easily available within the sections ‘Monitoring’ and ‘Query History’.

Query Profile

Further, it is easily possible to extract more information about the executed queries. Every single query within the ‘Query History’ has a unique identifier and attached metrics. Within the ‘Query History’ section, when choosing a query by simply clicking on it, we gain insights into the single query. These include two main sections.

‘Query Details’ delivers information about the query in general. Including the above-mentioned ‘Start Time’, ‘End Time’, ‘Warehouse Size’, and ‘Duration’, which is divided into ‘Compilation’, ‘Execution’, and further time-consuming steps.

‘Query Profile’ describes the execution plan of the query as a graphical representation. Each step within the query, such as ‘SELECT’, ‘JOIN’, and ‘WHERE CONDITION’ are represented as nodes within the graph. These nodes are operations, triggered by the query. Each fulfilling a task like filtering data, selecting data rows, but also scanning results, and reading data from cache. Snowflake interprets the written SQL and creates a performant execution plan by itself.

Despite that, each node consumes more or less resources, depending on the SQL code itself, the underlying dataset, and the Warehouse configurations. Using the ‘Query Profile’, we can identify the most time-consuming steps in our query, such as false ‘JOINS’, and work around those nodes to identify more performant solutions. Therefore, the ‘Query Profile’ provides detailed information about each node, like processing time, cache scanning, and partition pruning. For further information, check out the Snowflake Documentation.

Custom Monitoring Solution

But when Snowflake already provides a solution, why do we need separate monitoring?

Although already delivered, your requirements may vary from the existing solution. With the built-in tools mentioned above, the possibility to check on each query is given. Additionally, Snowflake may not be the only tool used for data processing. ELT tools like dbt or Coalesce help data engineers improve processes. To include these further metrics from outside Snowflake, a custom monitoring solution, based on Snowflake’s delivered metrics, is needed.

With this in mind, the following section focuses on our own scalable monitoring solution.

Account Usage and Information Schema

With the idea that the data of the graphical output exists as a table or view, containing all data necessary, it turns out that Snowflake provides us the schemas ‘ACCOUNT_USAGE’ and ‘INFORMATION_SCHEMA’, fulfilling this purpose. They provide similar opportunities for this challenge, although there are some differences.

‘ACCOUNT_USAGE’ provides insights into Snowflake metadata of several objects. This includes ‘QUERY_HISTORY’, which contains the information described in the sections above. The query history is available for all queries within the last 365 days. This enables the loading of metrics entities in the Data Warehouse, making the information readily accessible and persistent for daily, weekly, or monthly processes. The downside is that access to the ‘ACCOUNT_USAGE’ schema is often limited.

The ‘INFORMATION_SCHEMA’ contains a set of system-defined views and tables, providing metadata and information about created objects. Using predefined functions on the ‘INFORMATION_SCHEMA’, metadata can be retrieved, which, on the other hand, is limited to some factors. As this option is more accessible for developers, the focus remains on this option.

Account Usage and Information Schema

Extract Metrics Data

At the beginning, it is crucial to point out the limitations and possibilities of this approach. For the scope of this blog, two functions on the ‘INFORMATION_SCHEMA’ are needed.

‘QUERY_HISTORY_BY_WAREHOUSE()’ is equal to ‘QUERY_HISTORY’.

‘GET_QUERY_OPERATOR_STATS()’ is the equivalent of ‘QUERY_PROFILE’.

‘QUERY_HISTORY_BY_WAREHOUSE()’ takes four different arguments.

  • ‘WAREHOUSE_NAME’ – The name of the Warehouse executing the queries
  • ‘END_TIME_RANGE_START’ – Time range within the last 7 days, in which the query started running
  • ‘END_TIME_RANGE_END’ – Time range within the last 7 days, in which the query completed running
  • ‘RESULT_LIMIT’ – The number of the maximum returned rows. The default lies by ‘100’, the range by ‘1’ to ‘10.000’

The output is the ‘QUERY_HISTORY’ as a structured table with some extra information. Each query is displayed in one row. Therefore, the metrics are aggregated into one specific query.

The most critical limitation seen here is the ‘RESULT_LIMIT’ and ‘END_TIME_RANGE_*’, as it forces us to retrieve the data before the 7-day retention ends and before 10.000 other queries have been executed. Depending on the size of the Warehouse and the scope of the monitoring range, the extract and load process must be customized.

‘GET_QUERY_OPERATOR_STATS()’ takes one argument.

  • ‘QUERY_ID’ – The unique ID of the executed query.

The output is a structured table of the ‘QUERY_PROFILE’ of one specific query. Each step in the execution plan is displayed as one row, so the output size depends on the complexity of the query itself. Each node is detailed with available metadata, breaking down information from ‘GET_QUERY_OPERATOR_STATS()’ into individual steps. This allows for a deeper analysis of performance metrics, helping to identify any bottlenecks.

As the function is on query-level, it is not done by joining the table outputs of those two functions together to get a satisfactory result. Further steps are needed.

Load Metrics Data

Load Metrics Data

Before filtering, transforming, or joining the data, it is suggested to load the data as it comes into persistent tables.

Snowflake Query History

The first step should be to create the table when calling the function, already combining these two components to avoid missing capture data, columns, or false data types. To keep continuous loading, use your preferred ELT tool to append the process at the end, or set up a scheduled Snowflake Task, considering the limitations mentioned above.

Snowflake Query Profile

This process is more complex. To load the data, it is not sufficient to simply call the function once for the query history. To ensure data integrity and a clear 1:n solution (1 row in the query history, n>0 rows in the query profile), it is needed to iterate through the list of all query IDs inside the query history and run the function for each one of them. Each of the resulting datasets needs to be inserted into the corresponding table.

Information Delivery

Now, as the data is extracted and loaded, the last remaining steps are transforming the data and information delivery. As this strongly depends on the use case, we focus on linking the two metrics tables and creating a standard dashboard inside Snowflake to show the results.

Our two tables ‘snowflake_query_history’ and ‘snowflake_query_profile’ are in a soft relation, where the “QUERY_ID” serves as a unique identifier to link each query from the query history to its query profile. Therefore, joining both tables on the “QUERY_ID” is indispensable. The first decision to be made is whether to keep one row for each query and aggregate the object constructs or to split it up into its individual steps, like within the query profile. For simplicity, we focus on the second option to avoid complex SQL statements for the dashboard. As the SQL statement only includes the ‘SELECT’ and ‘JOIN’ operators, you may choose a view as materialization, instead of a table.

With the possibility of creating dashboards out of Snowflake worksheets, it is relatively easy to integrate simple SQL statements into the dashboard, represented as charts. For some examples, we will take a look at the resulting data.

Query Performance

This example shows the top 20 queries by ‘TOTAL_ELAPSED_TIME’, the overall time the query needs to compile, run, waiting time, and other technical steps. As noticed by the chosen x-axis label, Snowflake’s metrics can be combined using other tools, such as dbt. It is set to compare this time against the produced rows to gain some insights into whether the queries can handle much data or not. As can be seen, the query consuming the most time, and therefore credits, does not produce the most rows. At this point, we may be interested in gaining some insights into the query itself.

Query Profile

Showing each Operator with its corresponding relative runtime, only a handful exceed 5% of the total execution time. This may indicate that within the SQL statement, only a few steps need to be optimized to create an overall more performant query.

Query Dashboard

With this in mind, the single query can also be analyzed in the dashboard. For example, you may take into account input rows and output rows, bytes spilled, partitions scanned, and partitions total, or the join condition. Furthermore, you may not only gain insights into your query performance, but also into your table management, such as clustering and partitioning. Additionally, you are also capable of integrating more metrics coming from external sources and linking them directly to Snowflake, making it a powerful monitoring solution.

Conclusion

Snowflake offers a powerful and highly adaptable cloud data platform that meets the demands of modern enterprise data warehousing. However, as data volume and complexity grow, so does the need for proactive monitoring and optimization to ensure continued performance and cost efficiency. By leveraging Snowflake’s built-in tools and implementing strategic performance-enhancing techniques, organizations can address bottlenecks effectively and optimize query performance.

In this article, we addressed the opportunities given through Snowflake’s built-in tools and how to effectively use them. We learned that although these tools are great, to gain a fast and high-level overview, as well as detailed insights at the same time, it is relatively easy to create your own dashboard with the loaded data. Therefore, you are able to analyze and evaluate it, as well as make optimum use of resources to enable high-performance operation without increasing cost by scaling up or scaling out.

Close Menu