SC14 New Orleans, LA

The International Conference for High Performance Computing, Networking, Storage and Analysis

Fast Storage for File System Metadata.

Student: Kai Ren (Carnegie Mellon University)
Advisor: Garth Gibson (Carnegie Mellon University)
Abstract: Conventional distributed have used centralized single-node metadata services. However, the single-node metadata server design inherently limits the scalability of the file system in terms of the number of stored objects and concurrent accesses to the file system expected by massive parallel applications. Inefficient on-disk metadata representation also limits the metadata performance. To tackle these challenges, I implemented a middleware called IndexFS that can be layered on top of existing file systems and improve their metadata performance. IndexFS uses a tabular-based architecture that incrementally partitions the namespace on a per- directory basis, preserving server and disk locality for small directories. An optimized log-structured layout is used to store metadata and small files efficiently. Several caching techniques are also used to mitigate hot spots. By combining these techniques, IndexFS can improve the metadata performance of existing distributed file systems by as much as an order of magnitude for various metadata workloads.

Poster: pdf
Summary: pdf

Doctoral Showcase Index