Bamboo: A simple & scalable data warehouse and pipeline

The Client

Bamboo is a Perth based micro-investment company.

Their flagship mobile app allows users to make incremental investments into cryptocurrencies and precious metals every time they spend money.

With recent capital investments, Bamboo is gearing up for a global expansion, with the US in its sights, and a goal of reaching 100,000 users.

The Challenge

In order to meet reporting and analytical needs, Bamboo opted to use the industry leading Snowflake cloud data warehouse.

But a data warehouse is only as good as the data that stocks it, and Bamboo required a robust and reliable data platform that would meet their needs with minimal overhead, and scalable automation.

They turned to Mechanical Rock to design, configure and build the data platform based on the modern Extract-Load-Transform (ELT) pattern. :

• Secure authentication and authorisation
• Automate ingestion pipelines
• Transformation, data cleansing & privacy masking
• Scalable PAYG cloud data warehousing
• Basic data presentation and integration

DataOps with dbt

By applying the principles of DevOps to data pipelines, we can deliver solutions that are:

• Automated repeatable, flexible and agile
• Reliable and scalable
• Low maintenance and low overhead

DataOps utilises tools like dbt to produce repeatable automated patterns with minimal overhead.

This allows analytics teams to be more self sufficient and autonomous, developing analytical solutions that are repeatable and deliver value faster.

dbt is a transformation workflow that lets teams quickly and collaboratively deploy analytics code using software engineering best practices like modularity, portability and CI/CD. Using dbt, anyone who knows SQL can build production-grade data pipelines.

The Solution

Mechanical Rock integrated authentication and authorisation with the company’s Google GSuite providing a secure and seamless Single Sign On (SSO) experience.

A simple pattern for ELT was established, using the following industry-standard patterns:

• A simple, low maintenance mechanism for streaming live DynamoDB data to S3 using DynamoDB Streams and Kinesis Firehose.
• Ingestion from S3 to Snowflake with built-in ‘Snowpipes’
• Transformation in place using Data Build Tool (dbt) - introducing CICD for data transformation.

All of which was delivered using Infrastructure-as-Code and continuous delivery pipelines to deliver a repeatable DataOps experience for Bamboo.

The Benefits

The Mechanical Rock DataOps solution for Bamboo delivers:

• A simple but scalable automated data platform that is secure and requires minimal overhead to manage and run.
• A cloud native solution that requires no servers to manage and features and extremely cost effective PAYG cost model.
• A simple way to secure, govern and manage data on the platform
• An extensible architecture that will allow Bamboo to easily adopt more advanced Business Intelligence (BI) and Machine Learning (ML) tooling in the future.