VGW: Data migration & modelling

The Client

VGW is an innovative and highly profitable game design workshop. They are the pioneers of Social Sweepstake Gaming with over a million players and multiple product lines including Online Social Casinos and Poker platforms.

VGW had made the decision to use Snowflake as a key data warehouse tool across their product suite, endeavouring to accelerate its productionisation and usage. They have recently established a fit-for-purpose governance framework, the Snowflake Data Framework (SDF), with robust role based access controls and repeatable patterns to accelerate the migration of their data into Snowflake.

The Challenge

Adoption of the new framework had been strong across the business, however VGW’s core product, Chumba Casino, had yet to make the transition.

In their journey of ensuring trustable, timely and accurate data across their enterprise, VGW partnered with Mechanical Rock to provide their data engineering expertise to support Chumba Casino’s move away from their legacy data warehousing solution into the new Snowflake Data Platform.

The Solution

Mechanical Rock provided technical leadership and data engineering capability, along with guidance on data architecture, to support Chumba’s migration to SDF.

After reviewing the current state of the legacy data warehouse, identifying what does and does not need to be migrated, Mechanical developed a migration strategy plan for the Chumba gaming platform.

To action the plan the team used an agile/scrum methodology to establish the initial backlog of work, delivering functionality in increments and at a regular bi-weekly cadence, in conjunction with reviews by the VGW Product Owner.

Working together with VGW, Mechanical Rock delivered:

• The implementation of a new ingestion mechanism to transfer mutable data into Snowflake through Kafka.
• Modelled data into star schema design to answer business questions for 14 different business domains.
• Automated integration tests to check the transformation business logic for all tables in the curated layer.
• Comprehensive data quality checks for all tables in cleansed and curated layers.
• A separate data quality test to report on source data issues and send them to Alation.
• Automatic alerting and notification integration with Slack to report on any task failures as well as data quality test results.
• The integration of DQ test results into Alation to report on the success and failures of checks.
  
  

Snowflake data migration architecture diagram

Achievements

At the end of the engagement to migrate Chumba Casino to the new SDF platform, Mechanical Rock and VGW were able to achieve the following results.

• Migrated more than 40 tables to Snowflake raw and cleansed layers.
• Curated 50+ data sets into fact and dimension tables.
• Conducted comprehensive analysis and consumer interviews to understand their requirements before designing the curated data models.
• The new data quality tests integration with Alation has allowed the engineering team to identify and fix those issues at the source database, improving time to restore metrics.
• A new observability dashboard to show information about task failures, DQ test failures, credit usage and suspended tasks due to concurrent failures.
• Provisioning of the data catalogue documentation including sample queries for all tables in the curated layer, has provided a head start to all consumers of the new models.
• Uplifted the capabilities within VGW by documenting articles on how to map business questions to specific tables in the curated layer.
• The new data models provided faster and more trusted access to key data reducing the need for exception processing.
• Comprehensive report on recommendations outside of the scope of work to continue VGW’s journey in providing a safe, responsible and positive user experience.