The client is a subsidiary company specializing in information technology and telecommunications within the transportation sector while providing support for its parent company and various business units.
About the Client
Problem Statement
The client is looking to improve their data integration and data product delivery while better adhering to regulatory requirements. However, their existing data warehouse severely hampers their ability to reach that goal:
- High maintenance costs due to an outdated technology stack
- Costly, time-intensive integration of their complex source systems
- Current solution was not well equipped to deal with GDPR compliance
The Challenge
While technically proficient, the client’s data team is restricted by their unfamiliarity in designing and implementing a Data Vault 2.0-style architecture and a somewhat slow decision making process. In addition to the stated problems, the team encountered other challenges:
- Time constraints with an already decided end-of-life legacy warehouse
- Resource issues as team members were committed to keeping production running
The Solution
As part of our solution, we reviewed the client’s current data warehouse and conducted a workshop about our findings with them. We focused our efforts mainly on the organization and modeling, offered recommendations on how to implement GDPR compliance, and leveraging automation tools like Scalefree’s Turbovault4dbt and dbt with datavault4dbt, as well as how to swiftly integrate their source systems into the data platform while keeping costs down.
Tangible Results for the Client
We focused on transferring our knowledge, empowering the data team, and equipping them with everything to start implementation of a data warehouse according to the Data Vault 2.0 standards, speeding up their way towards faster data integration and a more efficient, cost-saving data product delivery.
Technologies involved
- Meltano
- Data Lake on AWS S3
- EDW on Redshift
- dbt Core
- Dashboarding with Tableau
- Visual modeling with draw.io
- Gitlab
