SC14 New Orleans, LA

The International Conference for High Performance Computing, Networking, Storage and Analysis

Efficient Data Compression by Efficient Use of HDF5 Format.


Authors: Katsumi Hagita (National Defense Academy), Manabu Omiya (Hokkaido University), Takashi Honda (ZEON Corporation), Masao Ogino (Nagoya University)

Abstract: Recently, in the area of supercomputing, data transfer problem becomes significant. We proposed efficient data compression method without changing common API like HDF5. JHPCN-DF (Jointed Hierarchical Precision Compression Number - Data Format) is a compression method using Bit Segmentation and Huffman coding. For visualizations and analyses, lower bit of IEEE 754 format is used to be not necessary. The required bits can be determined by direct check to numerical precisions corresponding to visualizations and analyses. We evaluated data size of HDF5 files coordinated by the JPHCN-DF framework in various simulations such as plasma electromagnetic particle-in-cell simulations, finite element method, and phase separated polymer materials. We confirmed visualization software work well with HDF5 files coordinated by the JPHCN-DF framework. For visualization, data sizes of HDF5 file seem to be reduced to 1/5 of original HDF5 file. It is found that JHPCN-DF is effective for wide area of simulations and data sciences.

Poster: pdf
Two-page extended abstract: pdf


Poster Index