Distributed content-based publish-subscribe middleware provides the necessary decoupling, flexibility, expressiveness, and scalability required by modern distributed applications. Unfortunately, this kind of middleware usually does not provide reliability guarantees, as this problem has been thus far largely disregarded by the research community and solutions developed in other contexts are not immediately applicable.

The emergence of computationally-enabled sensors and the applications that use sensor data introduces the need for a software infrastructure designed specifically to enable the rapid development and deployment of applications that draw upon data from multiple, heterogeneous sensor networks. We present the Hourglass infrastructure, which addresses this need. Hourglass is an Internet-based infrastructure for connecting a wide range of sensors, services, and applications in a robust fashion. In Hourglass, a stream of data elements is routed to one or more applications. These data elements are generated from sensors inside of sensor networks whose internals can be entirely hidden from participants in the Hourglass system. The Hourglass infrastructure consists of an overlay network of well-connected dedicated machines that provides service registration, discovery, and routing of data streams from sensors to client applications. In addition, Hourglass supports a set of in-network services such as filtering, aggregation, compression, and buffering stream data between source and destination. Hourglass also allows third party services to be deployed and used in the network. In this paper, we present the Hourglass architecture and describe our test-bed and implementation. We demonstrate how our design maintains streaming data flows in the face of disconnection, allows discovery of and access to data from sensors, supports participants of widely varying capabilities (servers to PDAs), takes advantage of wellprovisioned, well-connected machines, and provides separate efficient communication paths for short-lived control messages and long-lived stream-oriented data. 1

Achieving availability and scalability while providing service guarantees such as in-order, gapless delivery is essential for deploying publish/subscribe messaging middleware in wide area networks. Scalability often requires a publish/subscribe system to propagate subscription information and perform content matching across the network. Existing subscription propagation algorithms do not support in-order, gapless delivery in a redundant overlay network. This paper

Abstract. Distributed event routing has emerged as a key technology for achieving scalable information dissemination. In particular it has been used as preferential communication backbone within publish/subscribe communication system. Its aim is to reduce the network and computational overhead per event diffusion to a set (possibly large) of interested recipients. This paper introduces an unifying framework, namely a publish/subscribe architecture, that points out the functional decomposition between event-based routing layer, the overlay infrastructure layer and the network protocols layer. Hence the paper, firstly, surveys current algorithms for event based routing and possible overlay infrastructures in wired and mobile systems and, secondly, it discusses how and when single solutions at each level can be combined in the publish/subscribe architecture. Finally the paper positions existing publish/subscribe systems within the proposed architecture. 1

We introduce SpiderCast, a distributed protocol for constructing scalable churn-resistant overlay topologies for supporting decentralized topic-based pub/sub communication. SpiderCast is designed to effectively tread the balance between average overlay degree and communication cost of event dissemination. It employs a novel coverage-optimizing heuristic in which the nodes utilize partial subscription views (provided by a decentralized membership service) to reduce the average node degree while guaranteeing (with high probability) that the events posted on each topic can be routed solely through the nodes interested in this topic (in other words, the overlay is topic-connected). SpiderCast is unique in maintaining an overlay topology that scales well with the average number of topics a node is subscribed to, assuming the subscriptions are correlated insofar as found in most typical workloads. Furthermore, the degree grows logarithmically in the total number of topics, and slowly decreases as the number of nodes increases. We show experimentally that, for many practical workloads, the SpiderCast overlays are both topic-connected and have a low per-topic diameter while requiring each node to maintain a low average number of connections. These properties are satisfied even in very large settings involving up to 10, 000 nodes, 1, 000 topics, and 70 subscriptions per-node, and under high churn rates. In addition, our results demonstrate that, in a large setting, the average node degree in SpiderCast is at least 45 % smaller than in other overlays typically used to support decentralized pub/sub communication (such as e.g., similarity-based, rings-based, and random overlays).

This paper presents congestion control mechanisms for reliable and scalable message-oriented middleware following the publish/subscribe communication model. We identify the key requirements of congestion control in this environment, how it differs from congestion control for the Internet, and propose a combination of two congestion control mechanisms, (1) driven by a publisher hosting broker (PDCC), (2) driven by a subscriber hosting broker (SDCC). SDCC decouples the notion of a receive window and a NACK window, and is used by subscriber hosting brokers in recovery mode. PDCC implements a scalable and low latency feedback loop between a publisher hosting broker and all subscriber hosting brokers, which is used to adjust the rate of publishing new messages, to allow brokers in recovery to eventually catch up, and other brokers to keep up. We present a detailed experimental evaluation of our implementation of these mechanisms in the Gryphon system by injecting network failures and link congestion.

Abstract not found

Abstract not found

We describe algorithms to scalably support durable subscriptions in a publish-subscribe system. Durable subscriptions are guaranteed exactly-once message delivery, despite periods of disconnection from the system. Our algorithms persistently log each message only once in the system, and can support administratively specified 'early-release' policies that reclaim persistent storage in the presence of misbehaving durable subscribers. To efficiently recover messages missed by a disconnected durable subscriber, without refiltering messages published while the subscriber was disconnected, we persistently log filtering information in a manner optimized for the read/write pattern of durable subscriptions. Consolidation of data-structures across all subscribers that are done with catching up (after a disconnection) , helps the system support a larger number of subscribers. We experimentally demonstrate the low-latency and scalability properties of our implementation, both in the presence and absence of failures.

Research in sensor networks, continuous queries (CQ), and other domains has been motivated by powerful applications that aim to aggregate, assimilate, and interact with scores of sensor networks in parallel. Numerous system ingredients are necessary to make these applications possible. Sensor network research is building some of these components from the bottom up, dealing with issues such as wireless connectivity and battery life. CQ, peer-to-peer (P2P), and other research areas are building top down, examining in-network services, naming, decentralized queries, and scale. While many research groups use the same types of applications to motivate their work, many of these applications cannot be built today because of missing bridge research. These challenges include: uniting vastly differing devices and services, managing intermittent connectivity, placing in-network services with QoS and other constraints, developing unified security models, and correlating between sensor networks. This paper distills these new problems and outlines one proposed system that explores solutions to these concerns.