A leading kitchenware manufacturer and retailer wants to set up a scalable data platform for its analysts to be able to analyze their product and sales performance across different channels.
About the Client
Problem Statement
Before actively starting to build a new data platform, a strategy and a “how-to-start” plan usually contains uncertainties. Decisions like:
- Which database technology fits best?
- How to automate as much as possible and what tools can do this?
- How to be GDPR compliant?
- What orchestration tool to use?
- What data modeling technique to select?
- What kind of skills do we need?
- How much of the budget we are able to spend should be done upfront?
The Challenge
The client needed to be able to create a complete data platform from the ground by considering some challenges regarding:
- scalability (reaction on changes in the operating world),
- standardized processes,
- complex data structures,
- deletion/anonymization of personal identifiable information,
- how to meet auditability requirements
That’s why they were looking for a consulting partner with deep knowledge of the provided challenges to help them to define a strategy and overall methodology in those areas.
The Solution
The client selected Scalefree to perform a three day workshop to go through the challenges described above. As a methodology, Data Vault 2.0 was already considered, which was discussed in more detail in the workshop itself.
Also, the complex data structures were discussed based on best practices related to where and how to resolve the structures. Additionally, a solution was found for the GDPR challenge as well by introducing a process to deal with it while considering performance and auditability.
Also, the complex data structures were discussed based on best practices related to where and how to resolve the structures. Additionally, a solution was found for the GDPR challenge as well by introducing a process to deal with it while considering performance and auditability.
Tangible Results for the Client
During the workshop, the technical architecture, the Data Vault 2.0 methodology, processes, and best practices were discussed and a plan was created. This saved a lot of time for additional Proof of Concepts. The elaborated plan during the workshop was also supported by Scalefree consulting to answer additional questions.
Technologies involved
- dbt
- Snowflake
- Microsoft Azure
