RWWA: Managed orchestration solution

The Challenge

The Data Platform team had implemented a proof of concept (POC) for Apache Airflow on Amazon Elastic Kubernetes Service (EKS), managed by the Platform Delivery team. However, both teams lacked the expertise to effectively run open-source Airflow on Kubernetes, resulting in an inability to automate data workflows or run DBT (Data Build Tool) in a scalable environment. This hindered the organisation’s ability to efficiently manage and analyse data, leading to delays and increased operational risk.

Mechanical Rock was brought in to assess the situation, evaluate managed services for running data workflows, and provide a solution that would not only meet RWWA's operational needs but also ensure security and cost-effectiveness.

The Solution

Mechanical Rock recommended the implementation of AWS Managed Workflows for Apache Airflow (MWAA) with Fargate workers as the optimal solution for RWWA. This recommendation was based on a thorough analysis of various managed services, considering factors such as feature set, risk, maturity, costs, and procurement ease.

Implementation of MWAA

The team delivered an MWAA implementation that was:

• Easily Deployable and Secure: The infrastructure was designed for security and repeatability, using Infrastructure-as-Code (IaC) expressed in CloudFormation and deployment scripts executed via AWS Lambda.
• Operationally Efficient: The solution allowed Airflow to automatically run workflows and DBT jobs on schedule, enhancing availability and resilience across multiple Availability Zones. Operational monitoring was integrated through CloudWatch dashboards to track key metrics like worker queue and CPU utilisation.

Performance Tuning

To ensure that MWAA performed optimally, Mechanical Rock tailored the system to fit RWWA's specific access patterns. This involved configuring environment sizes, scaling parameters, and optimising Directed Acyclic Graphs (DAGs) for improved performance. Additional performance improvements were achieved through the subdivision of monolithic DAGs and fine-tuning DBT configurations.

Security Enhancements

Mechanical Rock also conducted a comprehensive security review, focusing on Snowflake and recommending improvements in software quality, role-based access control (RBAC), and data masking. Key recommendations included:

• Implementing CI/CD practices for Snowflake deployments.
• Redesigning RBAC to align with the principle of least privilege.
• Transitioning to a source-controlled environment to enhance security and auditability.

Handover and Continued Support

To facilitate a smooth transition to the Data Platform team, Mechanical Rock provided hands-on training sessions, templates, and examples of Airflow best practices tailored to RWWA's use cases. This empowered the team to manage MWAA independently without relying on external support.

The Benefits

The implementation of MWAA and the accompanying improvements brought several significant benefits to RWWA:

1. Increased Automation: The ability to automate data workflows and DBT processes drastically reduced manual intervention, minimising human error and increasing efficiency.
2. Enhanced Scalability: With MWAA running on AWS, RWWA can easily scale their workflows to accommodate varying workloads without compromising performance.
3. Improved Security Posture: The comprehensive security review and subsequent recommendations fortified RWWA's data governance practices, ensuring compliance with evolving regulatory requirements and reducing vulnerability to breaches.
4. Operational Independence: The Data Platform team gained the knowledge and tools necessary to manage MWAA independently, reducing reliance on external teams and enabling faster response times to operational needs.
5. Cost Efficiency: By leveraging managed services like MWAA, RWWA benefitted from predictable pricing models and reduced operational costs associated with self-managed infrastructures.
6. Performance Optimization: Tailored configurations and performance tuning led to better resource utilisation, ensuring that RWWA can process data more quickly and efficiently.
7. Continuous Improvement: The adoption of CI/CD practices and Infrastructure-as-Code for Snowflake deployments enhances the organisation’s ability to iterate on their data processes, leading to ongoing improvements in data quality and delivery.

Mechanical Rock's engagement with RWWA led to the successful implementation of a robust, secure, and highly available data workflows system. The MWAA solution not only met the immediate operational needs but also positioned RWWA for future scalability and security compliance. Through strategic recommendations and targeted training, the Data Platform team is now equipped to manage their workflows organically, leading to improved efficiency and reduced operational risks.