Get ready for a data revolution! Pinterest, the digital pinboard giant, is rewriting the rules of big data processing with its new platform, Moka. But here's where it gets controversial... they're moving away from the traditional Hadoop infrastructure and embracing the power of Kubernetes.
In a thought-provoking two-part blog series, Pinterest's Big Data Platform team, led by Soam Acharya, Rainie Li, William Tom, and Ang Zhang, shares their journey. They reveal how they considered various options for their next-generation data processing platform, ultimately settling on Moka as their cloud-native solution.
Part one focused on the high-level design and application layer, while part two delves into the infrastructure aspects and future directions. The team emphasizes how they've successfully operated Spark at an unprecedented scale on Kubernetes. They've implemented logging, metrics, and job history services, providing engineers with the tools to debug and optimize jobs without diving into cluster complexities.
Pinterest's commitment to infrastructure-as-code is evident as they utilize Terraform and Helm to create reproducible EKS clusters and deploy supporting components. The engineers also tackle the challenge of different hardware architectures, ensuring their data workloads perform optimally on both Intel and ARM-based instances.
The broader industry conversation adds an interesting twist. While Spark remains Pinterest's primary engine, the success of Moka has sparked a shift towards other processing frameworks like Flink Batch and Apache Ray. This highlights the flexibility of Moka as a base platform, accommodating diverse engine needs.
External observers view Moka as a reference architecture for cloud-native data systems. However, the Pinterest team emphasizes that their migration journey is far from over. They've shared valuable insights on the challenges and learnings, emphasizing the importance of observability, automation, and multi-engine support for organizations undergoing similar transformations.
So, is Pinterest's Moka the future of big data processing? What do you think? Join the discussion and share your thoughts in the comments!
