Back To Schedule
Friday, November 13 • 2:30pm - 3:00pm
Cloud-Native Apache Spark: why and how to migrate your Spark pipelines to Kubernetes

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
Apache Spark can run on top of Kubernetes (as opposed to Hadoop YARN or Standalone mode) since Spark versions 2.3 (2018). In the past two years, the support for running Spark on Kubernetes has grown a lot, and a lot of companies have adopted it -- in fact, Spark-on-Kubernetes will be officially considered "production ready" with the upcoming release of Spark 3.1. In this talk, we will go over the main reasons why many companies decide to adopt Spark-on-Kubernetes, and our best practices for making Spark on Kubernetes reliable and performant at scale. No prior knowledge of Spark or Kubernetes is required, but you should expect a technical session heavy with code-examples and real-life tips to help you productionize Spark on Kubernetes.

avatar for Jean-Ives Stephan

Jean-Ives Stephan

Co-Founder & CEO, Data Mechanics
JY is the co-founder of Data Mechanics, a cloud-native Spark platform making Spark easy-to-use and cost-effective for data engineers.Their platform is deployed on a Kubernetes cluster inside their customers cloud account (AWS, GCP, and Azure are supported).Prior to Data Mechanics... Read More →

Friday November 13, 2020 2:30pm - 3:00pm PST