A leading energy provider wants to make rational decisions by becoming more data-driven. To support this strategy, they want to build a data-driven enterprise platform for data analytics and self-service Business Intelligence.
About the Client
Problem Statement
The organization’s need and use of BI solutions for a sustainable data warehouse became inevitable. They needed a bigger number of reports in different business areas such as discovering customer insights, invoice tracking and systems performances. Hence the demand of more teams and end-users.
However, the current solution and infrastructure was not suitable for either goals nor scalable due to multiple reasons:
- Scalability issues hindered the expansion of BI solutions
- Maintenance overhead of multiple teams and their divergence
- Manual processes in testing and deployments delayed business deliveries
- EDW was neither fully historised nor auditable therefore it couldn’t be utilised as a long-term solution.
The Challenge
The provider’s first challenge was to look for a different data warehouse approach and methodology that can handle such scalability and then the appropriate infrastructure and technology for enhancing the performance of development cycle management would be sought. To overcome these challenges, the provider needed the following:
- Choosing a data warehouse model to achieve scalability and long term stability
- Strict development standards (performance, conventions, code reviews etc.) to reduce maintenance costs
- Automation of development and testing to accelerate releases of BI reports required by business-users
- Provisioning of infrastructure with less risk and greater speed and reduced costs
The Solution
Together with the client, the following solutions took place:
- A scalable state-of-the-art DWH (DV 2.0, Azure, Snowflake)
- Documentation of models, developments and deployment process to ease onboardings and teams scalability
- Using Automation tool (dbt) for faster developments
- CI/CD (Azure DevOps) to expedite deployment processes
- Strict QA-pipelines teams deployments must go through to ensure quality and reduce re-engineering to minimum.
- Infrastructure as Code (IaC) that led to facilitating financial savings
Tangible Results for the Client
- Major cost savings were achieved through:
- Minimising waste of valuable resources with automation and easy integration.
- Reducing re-engineering to minimum
- BI-Reports required by Business-users were delivered with agility and punctuality.
The project has essentially gone so well, the provider was able to consolidate its DWH and infrastructure as a Data Platform Product that could be used by sub-organizations that use identical systems.
Technologies used
- Snowflake DB
- Azure DevOps, Pipelines, ADF
- Dbt
- Airflow
- Terraform
