Results 1 -
6 of
6
A Flexible Framework for Asynchronous In Situ and In Transit Analytics for Scientific Simulations
"... Abstract—High performance computing systems are today composed of tens of thousands of processors and deep memory hierarchies. The next generation of machines will further increase the unbalance between I/O capabilities and processing power. To reduce the pressure on I/Os, the in situ analytics para ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Abstract—High performance computing systems are today composed of tens of thousands of processors and deep memory hierarchies. The next generation of machines will further increase the unbalance between I/O capabilities and processing power. To reduce the pressure on I/Os, the in situ analytics paradigm proposes to process the data as closely as possible to where and when the data are produced. Processing can be embedded in the simulation code, executed asynchronously on helper cores on the same nodes, or performed in transit on staging nodes dedicated to analytics. Today, software environ-nements as well as usage scenarios still need to be investigated before in situ analytics become a standard practice. In this paper we introduce a framework for designing, deploying and executing in situ scenarios. Based on a com-ponent model, the scientist designs analytics workflows by first developing processing components that are next assembled in a dataflow graph through a Python script. At runtime the graph is instantiated according to the execution context, the framework taking care of deploying the application on the target architecture and coordinating the analytics workflows with the simulation execution. Component coordination, zero-copy intra-node communications or inter-nodes data transfers rely on per-node distributed daemons. We evaluate various scenarios performing in situ and in transit analytics on large molecular dynamics systems sim-ulated with Gromacs using up to 2048 cores. We show in particular that analytics processing can be performed on the fraction of resources the simulation does not use well, resulting in a limited impact on the simulation performance (less than 9%). Our more advanced scenario combines in situ and in transit processing to compute a molecular surface based on the Quicksurf algorithm.
An Approach to Lowering the In Situ Visualization Barrier
"... Coupling visualization and analysis software with simulation code is a resource-intensive task. As the usage of simulation-based science grows, we asked ourselves: what would it take to enable in situ visualization for every simulation in ex-istence? This paper presents an alternative view focusing ..."
Abstract
- Add to MetaCart
(Show Context)
Coupling visualization and analysis software with simulation code is a resource-intensive task. As the usage of simulation-based science grows, we asked ourselves: what would it take to enable in situ visualization for every simulation in ex-istence? This paper presents an alternative view focusing on the approachability of in situ visualization. Utilizing a number of techniques from the program analysis community and taking advantage of commonalities in scientific software, we find that we can vastly reduce the time investment re-quired to achieve visualization-enabled simulations.
In-Situ Feature Extraction of Large Scale Combustion Simulations Using Segmented Merge Trees
"... Abstract—The ever increasing amount of data generated by scientific simulations coupled with system I/O constraints are fu-eling a need for in-situ analysis techniques. Of particular interest are approaches that produce reduced data representations while maintaining the ability to redefine, extract, ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract—The ever increasing amount of data generated by scientific simulations coupled with system I/O constraints are fu-eling a need for in-situ analysis techniques. Of particular interest are approaches that produce reduced data representations while maintaining the ability to redefine, extract, and study features in a post-process to obtain scientific insights. This paper presents two variants of in-situ feature extraction techniques using segmented merge trees, which encode a wide range of threshold based features. The first approach is a fast, low communication cost technique that generates an exact solution but has limited scalability. The second is a scalable, local approximation that nevertheless is guaranteed to correctly extract all features up to a predefined size. We demonstrate both variants using some of the largest combustion simulations available on leadership class supercomputers. Our approach allows state-of-the-art, feature-based analysis to be performed in-situ at significantly higher frequency than currently possible and with negligible impact on the overal simulation runtime. Keywords—topological data analysis, feature extraction, in situ analysis, merge tree computation, segmented merge tree I.
Lessons Learned from Building In Situ Coupling Frameworks
"... Over the past few years, the increasing amounts of data produced by large-scale simulations have motivated a shift from traditional offline data analysis to in situ analysis and visualization. In situ processing began as the coupling of a parallel simulation with an analysis or visualization library ..."
Abstract
- Add to MetaCart
(Show Context)
Over the past few years, the increasing amounts of data produced by large-scale simulations have motivated a shift from traditional offline data analysis to in situ analysis and visualization. In situ processing began as the coupling of a parallel simulation with an analysis or visualization library, motivated primarily by avoiding the high cost of accessing storage. Going beyond this simple pairwise tight coupling, complex analysis workflows today are graphs with one or more data sources and several interconnected analysis com-ponents. In this paper, we review four tools that we have developed to address the challenges of coupling simulations with visualization packages or analysis workflows: Damaris, Decaf, FlowVR and Swift. This self-critical inquiry aims to shed light not only on their potential, but most importantly on the forthcoming software challenges that these and other in situ analysis and visualization frameworks will face in or-der to move toward exascale.
Computing
"... 4. Application Domains......................................................................4 4.1.1. Joint genetic and neuroimaging data analysis on Azure clouds 4 4.1.2. Structural protein analysis on Nimbus clouds 5 4.1.3. I/O intensive climate simulations for the Blue Waters post-Petascale machin ..."
Abstract
- Add to MetaCart
(Show Context)
4. Application Domains......................................................................4 4.1.1. Joint genetic and neuroimaging data analysis on Azure clouds 4 4.1.2. Structural protein analysis on Nimbus clouds 5 4.1.3. I/O intensive climate simulations for the Blue Waters post-Petascale machine 5 5. Software and Platforms................................................................... 5
Eurographics Symposium on Parallel Graphics and Visualization (2014), pp. 1–8
"... In situ visualization has become a popular method for avoiding the slowest component of many visualization pipelines: reading data from disk. Most previous in situ work has focused on achieving visualization scalability on par with simulation codes, or on the data movement concerns that become preva ..."
Abstract
- Add to MetaCart
In situ visualization has become a popular method for avoiding the slowest component of many visualization pipelines: reading data from disk. Most previous in situ work has focused on achieving visualization scalability on par with simulation codes, or on the data movement concerns that become prevalent at extreme scales. In this work, we consider in situ analysis with respect to ease of use and programmability. We describe an abstraction that opens up new applications for in situ visualization, and demonstrate that this abstraction and an expanded set of use cases can be realized without a performance cost. Categories and Subject Descriptors (according to ACM CCS): 1. Introduction and
