Conveners
Complementary Technology Solutions: Session 1
- Sharon Broude Geva (University of Michigan)
Complementary Technology Solutions: Session 2
- Sharon Broude Geva (University of Michigan)
Description
User experiences, challenges and requests
HPC traditionally handles data at rest. The acquisition of streaming data presents a different set of challenges that, at scale, can be difficult to tackle. The approach to building data ingestion infrastructure at ARC-TS involves treating every service as a swappable building block. With this pluggable design using Docker containers you are free to choose which component is best. We will use...
Apache Spark, a popular open source big data tool form the Hadoop ecosystem is seeing rapid adoption across industry and academia, yet it is still generally not well known. For this talk we will demonstrate some large scale samples of how easy it is to benefit form spark SQL and Data Frames for Python and R programmers.
Continually increasing computational resources and improved efficiency of parallelized software for data generation and manipulation in the field of scientific computation have led to the requirement of more systematic approaches for data management. We present a data management framework designed to work on both desktop computers and in high-performance computing environments with special...
