This paper presents a new persistent data management layer designed to simplify cluster-based Internet service construction. This self-managing layer, called a distributed data structure (DDS), presents a conventional single-site data structure interface to service authors, but partitions and replicates the data across a cluster. We have designed and implemented a distributed hash table DDS that has properties necessary for Internet services (incremental scaling of throughput and data capacity, fault tolerance and high availability, high concurrency, consistency, and durability). The hash table uses two-phase commits to present a coherent view of its data across all cluster nodes, allowing any node to service any task. We show that the distributed hash table simplies Internet service construction by decoupling service-specic logic from the complexities of persistent, consistent state management, and by allowing services to inherit the necessary service properties from the DDS rather ...

(SDDS). An LH*RS file is hash partitioned over the distributed RAM of a multicomputer, e.g., a network of PCs, and supports the unavailability of any of its k ≥ 1 server nodes. The value of k transparently grows with the file to offset the reliability decline. Only the number of the storage nodes potentially limits the file growth. The high-availability management uses a novel parity calculus that we have developed, based on the Reed-Salomon erasure correcting coding. The resulting parity storage overhead is about the minimal ever possible. The parity encoding and decoding are faster than for any other candidate coding we are aware of. We present our scheme and its performance analysis, including experiments with a prototype implementation on Wintel PCs. The capabilities of LH*RS offer new perspectives to data intensive applications, including the emerging ones of grids and of P2P computing.

Mirroring is a popular technique for enhancing file availability. We incorporate this technique into the LH* algorithms for scalable distributed linear hash files. Several schemes for mirroring LH* files are presented in this paper. The schemes increase the availability of LH* files in the presence of node failures. Every record remains accessible in the presence of a single node failure, and usually in the presence of multiple-node failures. The price is, as usual, twice as much storage for data, and an increase in the number of messages. The different schemes are characterized by different trade-offs, and they accommodate diverse application requirements. The additional messaging cost per insert is about the same for all the schemes, and is roughly only one message. The cost of a bucket recovery may in contrast vary greatly, from one message for one type of scheme, to a few for another, and many for yet another.

Telecom databases are databases used in the operation of the telecom network and as parts of applications in the telecom network. The first telecom databases were Service Control Points (SCP) in Intelligent Networks. These provided mostly number translations for various services, such as Freephone. Also databases that keep track of the mobile phones (Home Location Registers, HLR) for mobile telecommunications were early starters. SCPs and HLRs are now becoming the platforms for service execution of telecommunication services. Other telecom databases are used for management of the network, especially for real-time charging information. Many information servers, such as Web Servers, Cache Servers, Mail Servers, File Servers are also becoming part of the telecom databases. These servers have in common that they all have to answer massive amounts of rather simple queries, that they have to be very reliable, and that they have requirements on short response times. Some of them also need lar...

Modern applications increasingly require scalable, highly available and distributed storage systems. High-availability schemes typically deliver data despite up to n 1 simultaneous unavailabilities of the storage nodes (disks, processors with storage, or entire computers), where n is fixed. Such schemes are insufficient for scalable files, since the probability of more than n failures increases arbitrarily with file size. We propose a new schema termed LH*sa withstanding up to n simultaneous unavailabilities with n scaling with the file. We present LH*sa file manipulation and recovery algorithms. We discuss the access and storage performance, and variants tuning selected features. We show that LH*sa files may scale to any number of nodes, keeping the probability of data unavailability arbitrarily small. 1

The GRID initiative provides an infrastructure for distributed computations among widely distributed high-performance computers. This will allow for exchanging and processing very large amounts of data. The LOFAR project (www.nfra.nl/lofar) is an international initiative to build a versatile, geographically distributed, multi-point radio facility for astrophysics, space physics, atmospheric physics, and radio research, utilizing very high performance GRID computing. LOIS is a proposed Swedish outrigger to LOFAR providing a software radar. As the volume of processed data by LOFAR/LOIS is very large and dynamic there will be need for very high performing data management systems. For this a high-performance stream-oriented distributed data manager and query processor is being developed that allows very efficient execution of database queries to streamed data involving numerical and other data. Very high performance is attained by utilizing many object-relational main-memory database engines running on PCs and connected through the GRID. The project leverages upon a highperformance, extensible, and object-oriented database engine, the Amos II kernel, developed in the Uppsala Database Laboratory. A very high performing stream-oriented DBMS is being developed for representing and querying non-relational data representations extracted from the data flows used in space and environmental physics applications. Of particular interest is the development of new distributed data population and query processing techniques for this kind of applications and thereby utilizing distributed and scalable data structures for high-performance stream data processing. 1

Spatial data storage stresses the capability of conventional DBMSs. We present a scalable distributed data structure, hQT*, which offers support for efficient spatial point and range queries using order preserving hashing. It is designed to deal with skewed data and extends results obtained with scalable distributed hash files, LH*, and other hashing schemas. Performance analysis shows that an hQT* file is a viable schema for distributed data access, and in contrast to traditional quad-trees it avoids long traversals of hierarchical structures. Furthermore, the novel data structure is a complete design addressing both scalable data storage and local server storage management as well as management clients addressing. We investigate several different client updating schemes, enabling better access load distribution for many "slow" clients. Keywords Scalable Distributed Data Structure, Spatial Point Index, Ordered Files, Multicomputers 1 Introduction Research is increasingly focusing o...

We propose that the challenges in the design and implementation of an SDDS can be significantly eased by separating the design of the scalable high-availability part (HADRAM) from the design of the SDDS proper. We have designed and partially implemented a HADRAM system that allows measurement to prove the validity of the concept. All other highly available SDDS provide failure tolerance at the record level, whereas HADRAM provides it at the memory level. 1

. Scalable, distributed data structures can provide fast access to large volumes of data. In this paper, we present a simulation study in which the performance behaviour of one of these data structures, called LH*LH, is evaluated for a multicomputer architecture. Our experimental results demonstrate that the access-time to LH*LH can be very small and does not deteriorate for increasing data structure sizes. Furthermore, we also show that parallel access to the LH*LH data structure may speed up client applications. This speed-up is, however, shown to be constrained by a number of effects. 1 Introduction Modern database applications require fast access to large volumes of data. Sometimes the amount of data is so large than it cannot be efficiently stored or processed by a uniprocessor system. Therefore, a distributed data structure can be used that distributes the data over a number of processors within a parallel or distributed system. This is an attractive possibility because the achi...

LH* schema is among most studied Scalable Distributed Data Structures. LH* variants have been in particular developed for the high-availability files, capable of serving all the data despite unavailability of some storage sites. The scalable availability schemes, tolerating increasingly more failures when the file grows, are of particular importance. We present three high-availability LH* schemes using new concept of record grouping. We discuss the common building blocks and the specific features of each schema. We compare the design issues, properties and performance. 1