Thank you to AWS and the AWS All Builders Welcome Grant for making this trip possible!
2 years out of uni and only 6 months into my tech career, I recently had the opportunity to travel to Las Vegas and attend re:Invent 2024. From a whirlwind week of talks, workshops, and drinking networking, here are my 3 highlights from the event.
The AWS All Builders Welcome Grant: The Future is Diverse
This trip was made possible by the AWS All Builders Welcome (ABW) Grant - a scholarship for those the first 5 years of their career and belonging to an underrepresented group within the tech landscape. The ABW Grant provided:
- A ticket to re:Invent,
- Return flights and 5 nights accommodation in Las Vegas,
- Welcome and closing events for the 400+ grant recipients,
- Mentoring sessions and fireside chats with AWS Leadership,
- Reserved seating for keynotes, and
- Of course, exclusive swag.
The 2024 ABW cohort included people from over 60 different countries and of all gender identities. There were graduates fresh from university, mothers looking to get back into the workforce, and experienced professionals undertaking a career change. It was exciting and energising to meet so many passionate and intelligent people. From people working at one of the world’s biggest banks in London, to people in Kenya driving internet accessibility in their country, every conversation came with a different, invigorating perspective.
The ABW Grant is a clear demonstration that not only is there diversity in tech, but also the importance of explicitly providing opportunities to those who may not otherwise have access. Between the re:Invent ticket, mentoring sessions with AWS leaders, and connections with such a wide range of people, the ABW Grant has provided me a leg up in my career and some fantastic opportunities.
The Power of AWS
It would be remiss of me to not mention some of the incredibly cool tools and use cases of AWS that were on display.
Scaling and evolving media storage at Netflix with Amazon S3
To start off the week, Netflix gave an overview of how they manage petabytes of multimedia assets using their custom metadata abstraction backed by S3. Their solution, called content drive, enables the globally distributed studio to ingest massive amounts of data, apply lifecycle policies, and provide access control both at scale and with significant savings.
Content drive is optimised using distributed caching, follower reads, and default ID sorting due to time-based UUIDs for low-latency database reads, parallel batch writes to allow high throughput, and CTE and unique filters to simplify queries.
Additionally, by leveraging user access patterns and understanding business rules, Netflix were able to dramatically reduce storage costs by writing a hybrid multi-tier smart storage solution. By hosting their own storage algorithm on Amazon’s multi-tier storage infrastructure, Netflix are able to have fine-grain control over allocating storage based on user needs and provide easy retrieval, whilst leveraging the low cost of S3 Glacier storage. This talk was a demonstration of the sheer scale of AWS, and an interesting combination of custom tooling and AWS infrastructure.
Scaling technology for millions of cricket fans on AWS
As a cricket tragic, I planned an entire afternoon around attending this one session. Dream11 is the world’s largest fantasy sports platform - with a peak user concurrency of 15 million users making up to 376 million requests per minute. Such a platform comes with some unique considerations and challenges:
- Impulse traffic - Requests to the platform spike by 70 million half an hour before the game begins - when the team line ups are announced. Such a surge won’t be adequately covered by auto-scaling, but is instead solved by using a custom solution called Scaler. Using years of historical data, Scaler uses a concurrency prediction model to continuously predict the next 15-30 minutes of traffic, which is then used to scale each of the 200+ microservices.
- Real-time leaderboard - A requirement of the platform is to rank teams within each of the 1 million+ contests every single delivery - as quick as every 5 seconds. Additionally, these contests range up to 20 million teams, equating to approximately 80GB of data for a single contest. Ranking the leaderboard faces memory issues, scaling costs, and bottlenecks when parallelising. The solution for this turns out to be relatively simple: salting. The contest is broken down into smaller groups, these groups are then ranked, and then the top spots are compared to make the overall leaderboard.
- Resilient Architecture - Minimising the impact of outages is always a key requirement for any platform. Dream11 uses an active-passive database setup that has been performance tested to handle up to 250,000 writes per second. Data is written and synced to both the active and passive databases. If the active database goes down, the passive database becomes active and the user experiences negligible disruptions.
Dream11 has cleverly leveraged the size, reliability, and compute of AWS to power the largest fantasy sports app.
AWS Women of the Cloud
The AWS Women of the Cloud events were a personal highlight. These included two panel discussions with talented software engineers and high level executives, a networking session, and a empowering walk across the expo floor as a group.
Outside of these events there were a whole host of people also passionate about strengthening gender diversity - mentors at mentoring sessions, presenters at fireside chats, and in conversation with other attendees. Across these discussions and events there were 3 key themes:
- Lack of talent in the pipeline - Data from the ABS shows that in Australia, the proportion of girls enrolled in engineering, IT, mathematics, and physics in year 12 has barely shifted since 2013. In IT specifically, girls made up only 26% of enrolments in 2022. This further drops off in university, with women making up only 22% of enrolments in IT in the same year.
- Women dropping out of tech careers - It is one hurdle to encourage women to pursue tech careers, and an entirely different challenge to keep them in the industry. Challenges that women face once they enter a tech career include culture that excludes them, being overlooked for promotions, difficulties reentering the industry following children, and lack of support for childcare responsibilities.
- Importance of allyship - There is something simple that everyone can do to help counteract these issues: be an ally to the women around you. Be supportive, welcoming, and aware.
re:Invent 2024 was an incredible experience, and I would strongly recommend anyone eligible to apply for the All Builders Welcome Grant. Attending such a large scale conference so early in my career has provided learnings and opportunities not just for technical growth, but also for personal growth.