Spark Streaming on K8s

31/2-029 (CERN)



Show room on map
    • 15:00 15:20
      Spark Streaming on K8s 20m

      Spark Streaming on K8s

      Speakers: Piotr Mrowczynski (CERN), Prasanth Kothuri (CERN)

      The presentation and the demo of the spark streaming examples on K8s was appreciated.

      Spark Streaming on K8s is something IT MON wanted to explore as possible evolution for the current Mesos/Marathon workflow, so this fits well.

      Given our (IT-DB_SAS) expertise,they have requested dedicated meeting where we show the main functionalities, in particular the ones which are more relevant to IT MON workflow. Such as:

      • Web UI capability (start/stop jobs, access logs, spark UI)
      • Job scheduling ( long-lived, cron-like, health check capabilities, deploy modes)
      • Job configuration (via web UI, CI options, docker image, etc.)
      • Security options / Secrets sharing
      • YARN/Hadoop/HDFS configuration, when needed

      Dates for Kubernetes workshop for IT MON to be decided