Scale By the Bay 2020

Apache Spark can run on top of Kubernetes (as opposed to Hadoop YARN or Standalone mode) since Spark versions 2.3 (2018). In the past two years, the support for running Spark on Kubernetes has grown a lot, and a lot of companies have adopted it -- in fact, Spark-on-Kubernetes will be officially considered "production ready" with the upcoming release of Spark 3.1. In this talk, we will go over the main reasons why many companies decide to adopt Spark-on-Kubernetes, and our best practices for making Spark on Kubernetes reliable and performant at scale. No prior knowledge of Spark or Kubernetes is required, but you should expect a technical session heavy with code-examples and real-life tips to help you productionize Spark on Kubernetes.