Results 1 -
9 of
9
Fast and efficient compression of floating-point data
- IEEE Transactions on Visualization and Computer Graphics
, 2006
"... Abstract—Large scale scientific simulation codes typically run on a cluster of CPUs that write/read time steps to/from a single file system. As data sets are constantly growing in size, this increasingly leads to I/O bottlenecks. When the rate at which data is produced exceeds the available I/O band ..."
Abstract
-
Cited by 39 (5 self)
- Add to MetaCart
(Show Context)
Abstract—Large scale scientific simulation codes typically run on a cluster of CPUs that write/read time steps to/from a single file system. As data sets are constantly growing in size, this increasingly leads to I/O bottlenecks. When the rate at which data is produced exceeds the available I/O bandwidth, the simulation stalls and the CPUs are idle. Data compression can alleviate this problem by using some CPU cycles to reduce the amount of data needed to be transfered. Most compression schemes, however, are designed to operate offline and seek to maximize compression, not throughput. Furthermore, they often require quantizing floating-point values onto a uniform integer grid, which disqualifies their use in applications where exact values must be retained. We propose a simple scheme for lossless, online compression of floating-point data that transparently integrates into the I/O of many applications. A plug-in scheme for data-dependent prediction makes our scheme applicable to a wide variety of data used in visualization, such as unstructured meshes, point sets, images, and voxel grids. We achieve state-of-the-art compression rates and speeds, the latter in part due to an improved entropy coder. We demonstrate that this significantly accelerates I/O throughput in real simulation runs. Unlike previous schemes, our method also adapts well to variable-precision floating-point and integer data. Index Terms—High throughput, lossless compression, file compaction for I/O efficiency, fast entropy coding, range coder, predictive coding, large scale simulation and visualization. 1
Lossless Geometry Compression for Steady-State and
- In Proc. Eurographics/IEEE Symposium on Visualization (EuroVis ’06
, 2005
"... In this paper we investigate the problem of lossless geometry compression of irregular-grid volume data represented as a tetrahedral mesh. We propose a novel lossless compression technique that effectively predicts, models, and encodes geometry data for both steady-state (i.e., with only a single ..."
Abstract
-
Cited by 3 (2 self)
- Add to MetaCart
In this paper we investigate the problem of lossless geometry compression of irregular-grid volume data represented as a tetrahedral mesh. We propose a novel lossless compression technique that effectively predicts, models, and encodes geometry data for both steady-state (i.e., with only a single time step) and time-varying datasets.
Lossless Compression of Point-Based 3D Models
"... this paper, we present a single-resolution technique for lossless compression of point-based 3D models. An initial quantization step is not needed and we can achieve a truly lossless compression. The scheme can compress geometry information as well as attributes associated with the points. We employ ..."
Abstract
- Add to MetaCart
this paper, we present a single-resolution technique for lossless compression of point-based 3D models. An initial quantization step is not needed and we can achieve a truly lossless compression. The scheme can compress geometry information as well as attributes associated with the points. We employ a three-stage pipeline that uses several ideas including k-d-tree-like partitioning, minimum-spanning-tree modeling, and a two-layer modified Huffman coding technique based on an optimal alphabet partitioning approach using a greedy heuristic. We show that the proposed technique achieves excellent lossless compression results
Abstract Fast and Efficient Compression of Floating-Point Data
"... Large scale scientific simulation codes typically run on a cluster of CPUs that write/read time steps to/from a single file system. As data sets are constantly growing in size, this increasingly leads to I/O bottlenecks. When the rate at which data is produced exceeds the available I/O bandwidth, th ..."
Abstract
- Add to MetaCart
(Show Context)
Large scale scientific simulation codes typically run on a cluster of CPUs that write/read time steps to/from a single file system. As data sets are constantly growing in size, this increasingly leads to I/O bottlenecks. When the rate at which data is produced exceeds the available I/O bandwidth, the simulation stalls and the CPUs are idle. Data compression can alleviate this problem by using some CPU cycles to reduce the amount of data that needs to be transfered. Most compression schemes, however, are designed to operate offline and try to maximize compression, not online throughput. Furthermore, they often require quantizing floatingpoint values onto a uniform integer grid, which disqualifies their use in applications where exact values must be retained. We propose a simple and robust scheme for lossless, online compression of floating-point data that transparently integrates into the I/O of a large scale simulation cluster. A plug-in scheme for data-dependent prediction makes our scheme applicable to a wide variety of data sets used in visualization, such as unstructured meshes, point sets, images, and voxel grids. We achieve state-of-the-art compression rates and compression speeds, the latter in part due to an improved entropy coder. We demonstrate that this significantly accelerates I/O throughput in real simulation runs. Unlike previous schemes, our method also adapts well to variable-precision floating-point and integer data.
Information Processing Letters 97 (2006) 133--137
- Information Processing Letters
"... this paper we measure fidelity using relative entropy because, in the asymmetric communication example above, the relative entropy is roughly how many more bits we expect the client to send with E-mail address: travis@cs.toronto.edu (T. Gagie) ..."
Abstract
- Add to MetaCart
this paper we measure fidelity using relative entropy because, in the asymmetric communication example above, the relative entropy is roughly how many more bits we expect the client to send with E-mail address: travis@cs.toronto.edu (T. Gagie)
Lossless Geometry Compression for Floating-Point Data in Steady-State and Time-Varying Fields over Irregular Grids
"... Abstract — In this paper we investigate the problem of lossless geometry compression of irregular-grid volume data represented as a tetrahedral mesh. We propose a novel lossless compression technique that effectively predicts, models, and encodes geometry data for both steady-state (i.e., with only ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract — In this paper we investigate the problem of lossless geometry compression of irregular-grid volume data represented as a tetrahedral mesh. We propose a novel lossless compression technique that effectively predicts, models, and encodes geometry data for both steady-state (i.e., with only a single time step) and time-varying datasets. Our geometry coder applies to floatingpoint data without requiring an initial quantization step and is truly lossless. However, the technique works equally well even if quantization is performed. Moreover, it does not need any connectivity information, and can be easily integrated with a class of the best existing connectivity compression techniques for tetrahedral meshes with a small amount of overhead information. We present experimental results which show that our technique achieves superior compression ratios. Index Terms — graphics compression, lossless geometry compression, irregular grids, steady-state and time-varying fields. 1.
Lossless Geometry Compression for Steady-State and Time-Varying Irregular Grids †
"... In this paper we investigate the problem of lossless geometry compression of irregular-grid volume data represented as a tetrahedral mesh. We propose a novel lossless compression technique that effectively predicts, models, and encodes geometry data for both steady-state (i.e., with only a single ti ..."
Abstract
- Add to MetaCart
(Show Context)
In this paper we investigate the problem of lossless geometry compression of irregular-grid volume data represented as a tetrahedral mesh. We propose a novel lossless compression technique that effectively predicts, models, and encodes geometry data for both steady-state (i.e., with only a single time step) and time-varying datasets. Our geometry coder is truly lossless and also does not need any connectivity information. Moreover, it can be easily integrated with a class of the best existing connectivity compression techniques for tetrahedral meshes with a small amount of overhead information. We present experimental results which show that our technique achieves superior compression ratios, with reasonable encoding times and fast (linear) decoding times. 1.
On the Reduction of Entropy Coding Complexity via Symbol Grouping: I – Redundancy Analysis and Optimal Alphabet Partition Amir
, 2004
"... data compression, symbol grouping, dynamic programming, Monge matrices We analyze the technique for reducing the complexity of entropy coding that consists in the a priori grouping of the source alphabet symbols, and in the decomposition of the coding process in two stages: first coding the number o ..."
Abstract
- Add to MetaCart
(Show Context)
data compression, symbol grouping, dynamic programming, Monge matrices We analyze the technique for reducing the complexity of entropy coding that consists in the a priori grouping of the source alphabet symbols, and in the decomposition of the coding process in two stages: first coding the number of the symbol's group with a more complex method, followed by coding the symbol's rank inside its group using a less complex method, or simply using its binary representation. This technique proved to be quite effective, yielding great reductions in complexity with reasonably small losses in compression, even when the groups are designed with empiric methods. It is widely used in practice and it is an important part in standards like MPEG and JPEG. However, the theory to explain its effectiveness and optimization had not been sufficiently developed. In this work, we provide a theoretical analysis of the properties of these methods in general circumstances. Next, we study the problem of finding optimal source alphabet partitions. We demonstrate a necessary optimality condition that eliminates most of the possible solutions, and guarantees that a more constrained version of the problem, which can be solved via dynamic programming, provides the optimal solutions. In addition, we show that the data used by the dynamic programming optimization has properties similar to the Monge matrices, allowing the use of much more efficient solution methods.
