The International Conference for High Performance Computing, Networking, Storage and Analysis
Storage Support for Data-Intensive Applications on Extreme-Scale HPC Systems.
Student: Dongfang Zhao (Illinois Institute of Technology)
Advisor: Ioan Raicu (Illinois Institute of Technology)
Abstract: Many believe that current HPC design would not meet the I/O requirement of the emerging exascale computing due to the segregation of compute and storage resources. Indeed, our simulation predicts, quantitatively, that system availability would go towards zero at exascale. This work proposes a storage architecture with node-local disks for HPC systems. Although co-locating compute and storage is not a new idea, it has not been widely adopted in HPC systems. We build a node-local filesystem, FusionFS, with two major principles: maximal metadata concurrency and optimal file write, both of which are crucial to HPC applications. We also discuss FusionFS’ integral features such as hybrid and cooperative caching, efficient accesses to compressed files, space-efficient data redundancy, distributed provenance tracking, and integration with data management systems. We have evaluated FusionFS on petascale supercomputers with 64K-cores and, compared its performance with major storage systems such as GPFS, PVFS, HDFS, and S3.