Building an open source analytics distribution on Kubernetes that meets the needs of a modern data scientist.

Jay Vyas

2019-09-12 17:30 #machinelearning

Modern machine learning libraries such as Jupyter, Kubeflow, PyTorch on technologies that are managed outside of the linux foundation, often forcing users to pay for cloud or hosted offerings to manage integration and tool dependencys. This challenges the ability of the ASF to continue blazing a trail of open innovation in the data science and analytics frontiers which are transforming so many businesses today. New projects in the ASF: such as Airflow, PredictionIO, SystemML, and NiFi - combined with technologies such as Spark and HDFS can be used to build an end-to-end analytics solution that is flexible and modular enough to meet the needs of modern data scientists, at pedabyte scale. In this talk, we’ll introduce a prototypical ‘next generation’ prototype which we propose as the future direction of ApacheBigtop, which runs entirely on Kubernetes, to minimize cost and maximize the portability of your modern hybrid analytics architecture.