Results 1 - 10
of
58
ACDS: Adapting Computational Data Streams for High Performance
- IN PROCEEDINGS OF INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS
, 2000
"... Data-intensive, interactive applications are an important class of metacomputing (Grid) applications. They are characterized by large dataflows between data providers and consumers, like scientific simulations and remote visualization clients of simulation output. Such dataflows vary at runtime, ..."
Abstract
-
Cited by 61 (27 self)
- Add to MetaCart
Data-intensive, interactive applications are an important class of metacomputing (Grid) applications. They are characterized by large dataflows between data providers and consumers, like scientific simulations and remote visualization clients of simulation output. Such dataflows vary at runtime, due to changes in consumers' data needs, changes in the nature of the data being transmitted, or changes in the availability of computing resources used by flows. The topic
An Object-Based Infrastructure for Program Monitoring and Steering
- In Proceedings of the 2nd SIGMETRICS Symposium on Parallel and Distributed Tools (SPDT'98
, 1998
"... Program monitoring and steering systems can provide invaluable insight into the behavior of complex parallel and distributed applications. But the traditional event-streambased approach to program monitoring does not scale well with increasing complexity. This paper introduces the Mirror Object Mode ..."
Abstract
-
Cited by 52 (19 self)
- Add to MetaCart
(Show Context)
Program monitoring and steering systems can provide invaluable insight into the behavior of complex parallel and distributed applications. But the traditional event-streambased approach to program monitoring does not scale well with increasing complexity. This paper introduces the Mirror Object Model, a new approach for program monitoring and steering systems. This approach provides a higher-level object-based abstraction that links the producer and the consumer of data and provides a seamless model which integrates monitoring and steering computation. We also introduce the Mirror Object Steering System (MOSS), an implementation of the Mirror Object Model based on CORBAstyle objects. This paper demonstrates the advantages of MOSS over traditional event-stream-based monitoring systems in handling complex situations. Additionally, we show that the additional functionality of MOSS can be achieved without significant performance penalty. 1 Introduction As applications have grown more com...
dQUOB: Managing Large Data Flows Using Dynamic Embedded Queries
, 2000
"... The dQUOB system satisfies client need for specific information from high-volume data streams. The data streams we speak of are the flow of data existing during large-scale visualizations, video streaming to large numbers of distributed users, and high volume business transactions. We introduces the ..."
Abstract
-
Cited by 46 (10 self)
- Add to MetaCart
(Show Context)
The dQUOB system satisfies client need for specific information from high-volume data streams. The data streams we speak of are the flow of data existing during large-scale visualizations, video streaming to large numbers of distributed users, and high volume business transactions. We introduces the notion of conceptualizing a data stream as a set of relational database tables so that a scientist can request information with an SQL-like query. Transformation or computation that often needs to be performed on the data en-route can be conceptualized ascomputation performed on consecutive views of the data, with computation associated with each view. The dQUOB system moves the query code into the data stream as a quoblet; as compiled code. The relational database data model has the significant advantage of presenting opportunities for efficient reoptimizations of queries and sets of queries. Using examples from global atmospheric modeling, we illustrate the usefulness of the dQUOB system. We carry the examples through the experiments to establish the viability of the approach for high performance computing with a baseline benchmark. We define a cost-metric of end-to-end latency that can be used to determine realistic cases where optimization should be applied. Finally, we show that end-to-end latency can be controlled through a probability assigned to a query that a query will evaluate to true.
Fast Heterogeneous Binary Data Interchange
- IN PROCEEDINGS OF THE HETEROGENEOUS COMPUTING WORKSHOP (HCW2000
, 2000
"... As distributed applications have become more widely used, they often need to leverage the computing power of a heterogeneous network of computer architectures. Modern communications libraries provide mechanisms that hide at least some of the complexities of binary data interchange among heterogen ..."
Abstract
-
Cited by 28 (14 self)
- Add to MetaCart
As distributed applications have become more widely used, they often need to leverage the computing power of a heterogeneous network of computer architectures. Modern communications libraries provide mechanisms that hide at least some of the complexities of binary data interchange among heterogeneous machines. However, these mechanisms may be cumbersome, requiring that communicating applications agree a priori on precise message contents, or they may be inefficient, using both "up" and "down" translations for binary data. Finally, the semantics of many packages, particularly those which require applications to manually "pack" and "unpack" messages, result in multiple copies of message data, thereby reducing communication performance. This paper describes PBIO, a novel messaging middleware which offers applications significantly more flexibility in message exchange while providing an efficient implementation that offers high performance.
PreDatA- Preparatory Data Analytics on Peta-Scale Machines
"... Abstract—Peta-scale scientific applications running on High End Computing (HEC) platforms can generate large volumes of data. For high performance storage and in order to be useful to science end users, such data must be organized in its layout, indexed, sorted, and otherwise manipulated for subsequ ..."
Abstract
-
Cited by 25 (10 self)
- Add to MetaCart
(Show Context)
Abstract—Peta-scale scientific applications running on High End Computing (HEC) platforms can generate large volumes of data. For high performance storage and in order to be useful to science end users, such data must be organized in its layout, indexed, sorted, and otherwise manipulated for subsequent data presentation, visualization, and detailed analysis. In addition, scientists desire to gain insights into selected data characteristics ‘hidden ’ or ‘latent ’ in the massive datasets while data is being produced by simulations. PreDatA, short for Preparatory Data Analytics, is an approach for preparing and characterizing data while it is being produced by the large scale simulations running on peta-scale machines. By dedicating additional compute nodes on the peta-scale machine as staging nodes and staging simulation’s output data through these nodes, PreDatA can exploit their computational power to perform selected data manipulations with lower latency than attainable by first moving data into file systems and storage. Such in-transit manipulations are supported by the PreDatA middleware through RDMAbased data movement to reduce write latency, application-specific operations on streaming data that are able to discover latent data characteristics, and appropriate data reorganization and metadata annotation to speed up subsequent data access. As a result, PreDatA enhances the scalability and flexibility of current I/O stack on HEC platforms and is useful for data pre-processing, runtime data analysis and inspection, as well as for data exchange between concurrently running simulation models. Performance evaluations with several production peta-scale applications on Oak Ridge National Laboratory’s Leadership Computing Facility demonstrate the feasibility and advantages of the PreDatA approach. I.
On-line Automated Performance Diagnosis on Thousands of Processes
, 2006
"... Performance analysis tools are critical for the effective use of large parallel computing resources, but existing tools have failed to address three problems that limit their scalability: (1) management and processing of the volume of performance data generated when monitoring a large number of appl ..."
Abstract
-
Cited by 21 (5 self)
- Add to MetaCart
Performance analysis tools are critical for the effective use of large parallel computing resources, but existing tools have failed to address three problems that limit their scalability: (1) management and processing of the volume of performance data generated when monitoring a large number of application processes, (2) communication between a large number of tool components, and (3) presentation of performance data and analysis results for applications with a large number of processes. In this paper, we present a novel approach for finding performance problems in applications with a large number of processes that leverages our multicast and data aggregation infrastructure to address these three performance tool scalability barriers. First, we show how to design a scalable, distributed performance diagnosis facility. We demonstrate this design with an on-line,
A Flexible Architecture Integrating Monitoring and Analytics for Managing Large-Scale Data Centers
- In Proceedings of the 8 th ACM International Conference on Autonomic Computing
, 2011
"... To effectively manage large-scale data centers and utility clouds, operators must understand current system and application behaviors. This requires continuous, real-time monitoring along with on-line analysis of the data captured by the monitoring system, i.e., integrated monitoring and analytics – ..."
Abstract
-
Cited by 19 (4 self)
- Add to MetaCart
(Show Context)
To effectively manage large-scale data centers and utility clouds, operators must understand current system and application behaviors. This requires continuous, real-time monitoring along with on-line analysis of the data captured by the monitoring system, i.e., integrated monitoring and analytics – Monalytics [28]. A key challenge with such integration is to balance the costs incurred and associated delays, against the benefits attained from identifying and reacting to, in a timely fashion, undesirable or non-performing system states. This paper presents a novel, flexible architecture for Monalytics in which such trade-offs are easily made by dynamically constructing software overlays called Distributed Computation Graphs (DCGs) to implement desired analytics functions. The prototype of Monalytics implementing this flexible architecture is evaluated with motivating use cases in small scale data center experiments, and a series of analytical models is used to understand the above trade-offs at large scales. Results show that the approach provides the flexibility to meet the demands of autonomic management at large scale with considerably better performance/cost than traditional and brute force solutions.
Scalability analysis of spmd codes using expectations
- In ICS ’07: Proceedings of the 21st annual international conference on Supercomputing
, 2007
"... We present a new technique for identifying scalability bottle-necks in executions of single-program, multiple-data (SPMD) parallel programs, quantifying their impact on performance, and associating this information with the program source code. Our performance analysis strategy involves three steps. ..."
Abstract
-
Cited by 15 (8 self)
- Add to MetaCart
(Show Context)
We present a new technique for identifying scalability bottle-necks in executions of single-program, multiple-data (SPMD) parallel programs, quantifying their impact on performance, and associating this information with the program source code. Our performance analysis strategy involves three steps. First, we collect call path profiles for two or more executions on different numbers of processors. Second, we use our ex-pectations about how the performance of executions should differ, e.g., linear speedup for strong scaling or constant ex-ecution time for weak scaling, to automatically compute the scalability of costs incurred at each point in a program’s ex-ecution. Third, with the aid of an interactive browser, an application developer can explore a program’s performance in a top-down fashion, see the contexts in which poor scal-ing behavior arises, and understand exactly how much each scalability bottleneck dilates execution time. Our analysis technique is independent of the parallel programming model. We describe our experiences applying our technique to ana-lyze parallel programs written in Co-array Fortran and Uni-fied Parallel C, as well as message-passing programs based on MPI.
FINESSE: A Prototype Feedback-guided Performance Enhancement System
, 2000
"... FINESSE is a prototype environment designed to support rapid development of parallel programs for single-addressspace computers by both expert and non-expert programmers. The environment provides semi-automatic support for systematic, feedback-based reduction of the various classes of overhead assoc ..."
Abstract
-
Cited by 15 (6 self)
- Add to MetaCart
FINESSE is a prototype environment designed to support rapid development of parallel programs for single-addressspace computers by both expert and non-expert programmers. The environment provides semi-automatic support for systematic, feedback-based reduction of the various classes of overhead associated with parallel execution. The characterisation of parallel performance by overhead analysis is first reviewed, and then the functionality provided by FINESSE is described. The utility of this environment is demonstrated by using it to develop parallel implementations, for an SGI Origin 2000 platform, of Tred2, a wellknown benchmark for automatic parallelising compilers.
