First Step to Hybrid Cloud Computation: Elastic YARN and Kubernetes

Zhankun Tang, Sunil Govind

2019-09-10 14:30 #bigdata

‘In this presentation, we will introduce our work: Elastic YARN, which is an elastic, cloud-native solution and it allows co-location YARN and Kubernetes for both on-premises and on-cloud. Audiences can learn how to build a hybrid, low-cost and easy-to-use computation platform while ensuring the SLA by leveraging our work. Abstract Enterprise continues to evolve towards data-driven, more types of workloads will come onboard, the challenges of managing mixed workloads emerge. It’s essential to think about what is the most flexible architecture to manage batch jobs and long-running services both on-premises and on-cloud. Apache Hadoop YARN and Kubernetes are the de-facto enterprise level resource management system for big data workloads and long-running services. More specifically, big data applications like Spark/Hive-on-Tez/Map-Reduce are all prod-ready on YARN; on the other hand, Kubernetes is good to deploy services. Hadoop is running everywhere inside on-premises clusters, and Kubernetes is running everywhere inside clusters on the cloud. If a user wants to lift-and-shift workload from on-premises to on-cloud, it is still an extremely challenging task. In this presentation, we will introduce our work: Elastic YARN, which is an elastic, cloud-native solution and it allows co-location YARN and Kubernetes for both on-premises and on-cloud. Audiences can learn how to build a hybrid, low-cost and easy-to-use computation platform while ensuring the SLA by leveraging our work.n’