Hadoop Storage in the Cloud Native Era

Márton Elek

2019-09-10 18:00 #bigdata

Hadoop was born much earlier than the Cloud Native era. But the question is still the same: what can it offer in the time of Kubernetes, containerization and hybrid clouds? Apache Hadoop Ozone is a new subproject of Hadoop. It is based a generic low-level binary layer, the Hadoop Distributed Data Storage (HDDS) and a S3 compatible Object Store implementation on top of it. The separation of the data storage layer has multiple benefits: it can help to solve the “small files problem” and increase the scalability of the Hadoop storage clusters but it also helps us to provide access to the same data via multiple interfaces. The same data can be accessed via a Hadoop compatible file system API, with an S3 compatible file system API or as a raw, mountable block device. Storage is still a hot topic with Kubernetes and in Cloud Native environments. Container Storage Interface specification is a vendor neutral standard to provide storage plugin for multiple container orchestration system. With implementing the vendor-independent Container Storage Interface, Ozone can provide block level access on top of the Hadoop Distributed Data Storage layer and it can be first class citizen of the containerized word. In this talk we will demonstrate how the Apache Hadoop Ozone storage could be used from containers, how can the same storage space can be used as Object Store, Hadoop compatible file system or native block device. We will explain the basic storage type of Kubernetes clusters and show how Apache Hadoop Ozone can help to solve the storage problem in an industry standard way.