Results 1 -
7 of
7
Distributed Event Aggregation for Content-based Publish/Subscribe Systems
- In DEBS
, 2014
"... Modern data-intensive applications handling massive event streams such as real-time traffic monitoring require support for both rich data filtering and aggregation. While the pub/sub communication paradigm provides an effective solution for the sought semantic di-versity of event filtering, the even ..."
Abstract
-
Cited by 2 (1 self)
- Add to MetaCart
(Show Context)
Modern data-intensive applications handling massive event streams such as real-time traffic monitoring require support for both rich data filtering and aggregation. While the pub/sub communication paradigm provides an effective solution for the sought semantic di-versity of event filtering, the event processing capabilities of exist-ing pub/sub systems are restricted to singular event matching with-out support for stream aggregation, which so far can be accommo-dated only at the subscriber edge brokers. In this paper, we propose the first systematic solution for sup-porting distributed aggregation over a range of time-based aggrega-tion window semantics in a content-based pub/sub system. In order to eschew the need to disseminate a large number of publications to subscribers, we strive to distribute the aggregation computation within the pub/sub overlay network. By enriching the pub/sub lan-guage with aggregation semantics, we allow pub/sub brokers to aggregate incoming publications and forward only results to the next broker downstream. We show that our baseline solutions, one which aggregates early (at the publisher edge) and another which aggregates late (at the subscriber edge), are not optimal strategies for minimizing bandwidth consumption. We then propose an adap-tive rate-based heuristic solution which determines which brokers should aggregate publications. Using real datasets extracted from our traffic monitoring use case, we show that this adaptive solution leads to improved performance compared to that of our baseline solutions.
PubliyPrime: Exploiting Overlay Neighborhoods to Defeat Byzantine Publish/Subscribe Brokers
"... Abstract. Publish/Subscribe (pub/sub) systems operating in hostile multi-domain environments face various types of internal and external threats. This paper focuses on internal threats manifested when pub/sub brokers misbehave by deviating from their prescribed protocol. Broker misbehavior may parti ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
Abstract. Publish/Subscribe (pub/sub) systems operating in hostile multi-domain environments face various types of internal and external threats. This paper focuses on internal threats manifested when pub/sub brokers misbehave by deviating from their prescribed protocol. Broker misbehavior may partially or fully compromise the pub/sub service integrity. In this paper, we exploit the notion of overlay neighborhoods and develop a solution to protect content-based pub/sub systems against misbehaving (a.k.a., byzantine) brokers. Our approach gives correct brokers the ability to oversee the actions of their neighbors, identify cases of deviation from the protocol and take steps to subvert the threat. As a result of this collective oversight, the system can preserve its service integrity despite presence of a configurable number of byzantine brokers. We have implemented our solution and report on our experimental evaluation results. 1
Towards byzantine fault tolerant publish/subscribe: A state machine approach
- in Proceedings of the 9th Workshop on Hot Topics in Dependable Systems
, 2013
"... More than a decade of research has gone into techniques aimed at tolerating arbitrary failures in client/server inter-action, using consensus based replication. These works made Byzantine fault tolerance possible [5], competitive [18], ro-bust [7], and feasible to apply [6]. In this paper we estab-l ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
More than a decade of research has gone into techniques aimed at tolerating arbitrary failures in client/server inter-action, using consensus based replication. These works made Byzantine fault tolerance possible [5], competitive [18], ro-bust [7], and feasible to apply [6]. In this paper we estab-lish a connection between the pub/sub interaction model and consensus based replication protocols, that makes the above results applicable to the design of large scale event-based middleware. We propose a Byzantine fault tolerant pub/sub system, on a tree-based overlay, tolerating a con-figurable number of failures in any part of the system, with minimal divergence from traditional pub/sub specifications and forwarding schemes.
Introducing Publiy: A Multi-Purpose Distributed Publish/Subscribe System ABSTRACT
"... ..."
(Show Context)
Grand Challenge: The BlueBay Soccer Monitoring Engine
"... This paper presents the design and implementation of a custombuilt event processing engine called BlueBay developed for live monitoring of soccer games. We experimentally evaluated our system using a real workload and report on its performance. Our results indicate that BlueBay achieves a throughput ..."
Abstract
- Add to MetaCart
(Show Context)
This paper presents the design and implementation of a custombuilt event processing engine called BlueBay developed for live monitoring of soccer games. We experimentally evaluated our system using a real workload and report on its performance. Our results indicate that BlueBay achieves a throughput of up to 790k events per second, therefore processing the game’s input sensor stream about 60 times faster than real-time. In addition to our custom implementation, we also investigated the applicability of offthe-shelf general-purpose event processing engines to address the soccer monitoring problem. This effort resulted in two additional and fully functional implementations based on Esper and Storm. 1.
StreamHub: A Massively Parallel Architecture for
"... By routing messages based on their content, publish/subscribe (pub/sub) systems remove the need to establish and main-tain fixed communication channels. Pub/sub is a natural candidate for designing large-scale systems, composed of ap-plications running in different domains and communicating via midd ..."
Abstract
- Add to MetaCart
(Show Context)
By routing messages based on their content, publish/subscribe (pub/sub) systems remove the need to establish and main-tain fixed communication channels. Pub/sub is a natural candidate for designing large-scale systems, composed of ap-plications running in different domains and communicating via middleware solutions deployed on a public cloud. Such pub/sub systems must provide high throughput, filtering thousands of publications per second matched against hun-dreds of thousands of registered subscriptions with low and predictable delays, and must scale horizontally and vertically. As large-scale application composition may require complex publications and subscriptions representations, pub/sub sys-tem designs should not rely on the specific characteristics of a particular filtering scheme for implementing scalability.
P2S: A Fault-Tolerant Publish/Subscribe Infrastructure
"... The popular publish/subscribe communication paradigm, for building large-scale distributed event notification systems, has attracted attention from both academia and industry due to its performance and scalability characteristics. While ordinary “web surfers ” typically are not aware of minor packet ..."
Abstract
- Add to MetaCart
(Show Context)
The popular publish/subscribe communication paradigm, for building large-scale distributed event notification systems, has attracted attention from both academia and industry due to its performance and scalability characteristics. While ordinary “web surfers ” typically are not aware of minor packet loss, industrial applications often have tight timing constraints and require rigorous fault tolerance. Some past research has addressed the need to tolerate node crashes and link failures, often relying on distributing the brokers on an overlay network. However, these solutions impose significant complexity both in terms of implementation and deployment. In this paper, we present a crash tolerant Paxos-based pub/sub (P2S) middleware. P2S contributes a practical solution by replicating the broker in a replicated architecture based on Goxos, a Paxosbased fault tolerance library. Goxos can switch between various
