Managing Hundreds of Petabytes of data in the Cloud

Lohit VijayaRenu

2019-09-10 12:15 #bigdata

Twitter’s on premise HDFS clusters are scaled out to handle hundreds of petabyte data serving thousands of developers every day. Managing such large scale data introduces several challenges. Organization, discovery, data life cycle, replication, schema management and many more questions needs to be answered at scale. Our data infrastructure is built on different open source and in house projects working together to solve this problem. While we extend our data to cloud, we had to solve similar challenges on cloud environment at Twitter scale. In this talk we lay out our framework, talk about challenges we faced on cloud and solutions we are working towards to support thousands of users dealing with hundreds of petabytes of data on cloud.