DMCA
SciBORQ: Scientific data management with Bounds On Runtime and Quality (2011)
| Venue: | In Proc. of the Int’l Conf. on Innovative Data Systems Research (CIDR |
| Citations: | 21 - 3 self |
BibTeX
@INPROCEEDINGS{Sidirourgos11sciborq:scientific,
author = {Lefteris Sidirourgos and Martin Kersten and Peter Boncz},
title = {SciBORQ: Scientific data management with Bounds On Runtime and Quality},
booktitle = {In Proc. of the Int’l Conf. on Innovative Data Systems Research (CIDR},
year = {2011},
pages = {296--301}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Data warehouses underlying virtual observatories stress the capabilities of database management systems in many ways. They are filled, on a daily basis, with large amounts of factual information derived from intensive data scrubbing and computational feature extraction pipelines. The predominant data processing techniques focus on parallel loads and map-reduce feature extraction algorithms. Querying these huge databases require a sizable computing cluster, while ideally the initial investigation should run interactively, using as few resources as possible. In this paper, we explore a different route, one based on the observation that at any given time only a fraction of the data is of primary value for a specific task. This fraction becomes the focus of scientific reflection through an iterative process of ad-hoc query refinement. Steering through data to facilitate scientific discovery demands guarantees for the query execution time. In addition, strict bounds on errors are required to satisfy the demands of scientific use, such that query results can be used to test hypotheses reliably. We propose SciBORQ, a framework for scientific data exploration that gives precise control over runtime and quality of query answering. We present novel techniques to derive multiple interesting data samples, called impressions. An impression is selected such that the statistical error of a query answer remains low, while the result can be computed within strict time bounds. Impressions differ from previous sampling approaches in their bias towards the focal point of the scientific data exploration, their multi-layer design, and their adaptiveness to shifting query workloads. The ultimate goal is a complete system for scientific data exploration and discovery, capable of producing quality answers with strict error bounds in pre-defined time frames. 1.
Keyphrases
scientific data exploration virtual observatory parallel load predominant data ultimate goal query answer scientific discovery initial investigation focal point scientific reflection pre-defined time frame intensive data scrubbing query execution time statistical error different route specific task multi-layer design query answering query result strict time bound many way computational feature extraction pipeline data warehouse multiple interesting data sample query workload complete system factual information sizable computing cluster quality answer strict error bound daily basis precise control scientific use iterative process strict bound huge database database management system large amount map-reduce feature extraction algorithm primary value ad-hoc query refinement
