## The Variable-Increment Counting Bloom Filter

### Cached

### Download Links

Citations: | 6 - 3 self |

### BibTeX

@MISC{Rottenstreich_thevariable-increment,

author = {Ori Rottenstreich and Isaac Keslassy},

title = {The Variable-Increment Counting Bloom Filter},

year = {}

}

### OpenURL

### Abstract

Abstract—Counting Bloom Filters (CBFs) are widely used in networking device algorithms. They implement fast set representations to support membership queries with limited error, and support element deletions unlike Bloom Filters. However, they also consume significant amounts of memory. In this paper we introduce a new general method based on variable increments to improve the efficiency of CBFs and their variants. Unlike CBFs, at each packet arrival, the hashed counters increase by a hashed variable increment instead of a unit increment. Then, to query a packet, the exact value of a counter is considered and not just its positiveness. We present two simple schemes based on this method. We demonstrate that this method can always achieve a lower false positive rate and a lower overflow probability bound than CBF in large systems. We also show how it can be easily implemented in hardware, with limited added complexity and memory overhead. We also explain how this method can extend many variants of CBF that have been published in the literature. Last, using simulations, we show how it can improve the false positive rate of CBFs by up to an order of magnitude given the same amount of memory.

### Citations

1475 | Space/time trade-offs in hash coding with allowable errors
- Bloom
- 1970
(Show Context)
Citation Context ...ng Bloom Filters (VI-CBFs). A Bloom Filter (BF) is a well-known simple data structure used to represent a set of elements S = {x1, . . . , xn} of n elements from a universe U using an array of m bits =-=[7]-=-. However, BF is not designed to support deletions of elements, which are often needed in networking device algorithms. As illustrated in Figure 1(a), BF uses k uniformlydistributed hash functions ove... |

691 | Summary cache: a scalable wide-area web cache sharing protocol
- Fan, Cao, et al.
- 2000
(Show Context)
Citation Context ...ket currently queued (set membership), how many of its packets are queued (counter representation), whether it is in a given state (state representation), or where to forward its packets (IP lookups) =-=[2]-=-–[6]. CBFs are often used because they can be easily implemented in hardware. In particular, element insertions, deletions and queries can be implemented in CBFs using a constant complexity that is es... |

353 | Network applications of bloom filters: A survey
- Broder, Mitzenmacher
- 2004
(Show Context)
Citation Context ...sider a CBF with m counters, n elements and k hash functions. Its false positive rate F P R(CBF ) is optimized when k = m n F P R(CBF ) = = ( ( 1 − 1 − ( 1 − 1 ) ) kn k m ( 1 − 1 ) ) ln(2)m k m ln(2) =-=[11]-=- and equals m→∞ −−−−→ ( 1 ) k . 2 (7) Incidentally, when k is not integer, it is possible to let each flow use either k1 = ⌊k⌋ hash functions with probability pk = ⌈k⌉ − k (i.e., as determined by an a... |

40 | Counter Braids: A novel counter architecture for per-flow measurement
- Lu, Montanari, et al.
- 2008
(Show Context)
Citation Context ...s, it is possible to represent the set of two states hashing into the same entry as the sum of their hashed values, instead of simply storing a DK (Don’t Know) value as currently done. Counter Braids =-=[5]-=-: This efficient counter based architecture incrementally compresses flow counters as it uses a hierarchy of counters braided via a random graph. It later uses an iterative reconstruction scheme for t... |

30 | Beyond bloom filters: from approximate membership checks to approximate state machines
- Bonomi, Mitzenmacher, et al.
(Show Context)
Citation Context ...ic Bh − Bloom scheme, we have that fD(j) = 1 for j ∈ [0, h] and fD(j) = 0 for j > 0. We now try to calculate fD(j) for our improved Bh − Bloom scheme. As explained earlier, fD(j) = 1 for j ∈ [0, h] = =-=[0, 3]-=-. For instance, to calculate fD(4), we consider the 44 = 256 possible combinations of four ordered elements of D. Among them there are, for each of the four (9) values of D, 34 combinations in which i... |

16 |
A complete annotated bibliography of work related to Sidon sequences
- O’Bryant
(Show Context)
Citation Context ...ce the Bh − Bloom scheme, a variable-increment CBF (Counting Bloom Filter) based on Bh sequences [9]. We start with the formal definition of Bh sequences. B2 sequences are also called Sidon sequences =-=[10]-=-. Definition 1 (Bh Sequence): Let (A, +) be an abelian (commutative) group. Let D = {v1, v2, ..., vℓ} ⊆ A be a sequence of elements of A. Then D is a Bh sequence over A iff all the sums vi1 +vi2 +· · ... |

9 |
IPv6 lookups using distributed and load balanced bloom filters for 100 Gbps core router line cards
- Song, Hao, et al.
- 2009
(Show Context)
Citation Context ...currently queued (set membership), how many of its packets are queued (counter representation), whether it is in a given state (state representation), or where to forward its packets (IP lookups) [2]–=-=[6]-=-. CBFs are often used because they can be easily implemented in hardware. In particular, element insertions, deletions and queries can be implemented in CBFs using a constant complexity that is essent... |

9 | Hash-based techniques for high-speed packet processing
- Kirsch, Mitzenmacher, et al.
- 2009
(Show Context)
Citation Context ...heme.) C. False Positive Rate We now provide the false positive rate of Bh −CBF . As shown in the literature, in real-world systems, practical hash functions usually work as if they were fully random =-=[14]-=-. We assume in our proofs that the hash functions map items to random numbers uniformly distributed over their given range, and that the inserted elements and the elements in the query are independent... |

5 |
Integer Hash Function. http://www.concentric.net/ Ttwang/tech/inthash.htm
- Wang
- 2007
(Show Context)
Citation Context ...h the model using real life trace and real hash functions We conduct experiments using real-life traces recorded on a single direction of an OC192 backbone link [13], using a 64-bit mix hash function =-=[14]-=- to implement the requested hash functions. The hash functions are calculated of the tuple (Source IP, Destination IP, Source Port, Destination Port, Protocol). Any two packets that differ in at least... |

4 | The Bloom paradox:When not to use a Bloom filter
- Rottenstreich, Keslassy
- 2012
(Show Context)
Citation Context ... losing some information, but being able to recover it with high probability upon deletion (i.e. decrement) of some of the states. • And many additional schemes like Counter Braids [4], Selective CBF =-=[9]-=- and Access-Efficient Balanced BF [10]. However, there are also a few variants to CBF that the generic variable-increment idea does not necessarily improve. For instance, [11] uses counters to estimat... |

3 |
Multilayer compressed counting Bloom filters
- Ficara, Giordano, et al.
- 2008
(Show Context)
Citation Context ...hod relies on a generic idea that can be implemented to improve or extend most variants of CBFs. For instance, it could be applied to: • • CBF [2]: As presented above. ML-HCBF (MultiLayer Hashed CBF) =-=[8]-=-: This algorithm uses a hierarchical compression of CBF filters. The same hierarchical compression idea could be combined with any CBF with variable increments and is expected to improve the scheme pe... |

3 |
Bh sequences, Analytic number theory
- Graham
- 1995
(Show Context)
Citation Context ...se positive rate by more than 40%. A. Bh Sequences II. THE Bh − Bloom SCHEME In this section we introduce the Bh − Bloom scheme, a variable-increment CBF (Counting Bloom Filter) based on Bh sequences =-=[9]-=-. We start with the formal definition of Bh sequences. B2 sequences are also called Sidon sequences [10]. Definition 1 (Bh Sequence): Let (A, +) be an abelian (commutative) group. Let D = {v1, v2, ...... |

3 |
CAIDA Anonymized 2008 Internet Trace equinix-chicago 2008-03-19
- Shannon, Aben, et al.
(Show Context)
Citation Context ...ee bits per counter in all the other levels. A. Trace-Driven Simulation VI. EXPERIMENTAL RESULTS We conduct experiments using real-life traces recorded on a single direction of an OC192 backbone link =-=[17]-=-, and rely on a 64-bit mix hash function [18] of the IP 5-tuple to implement the requested hash functions.Spectral Improved ML- MLCBF BF dlCBF ML-HCBF Bh-CBF Bh-CBF VI-HCBF, VI-CBF, VI-CBF, VI-HCBF, ... |

2 |
improved construction for counting Bloom filters
- “An
- 2006
(Show Context)
Citation Context ...each of its stages, it would provide more information to its reconstruction scheme, which might help ensure better guarantees on the termination of the decoding scheme. Fingerprint-based Schemes [3], =-=[4]-=-: Fingerprint-based schemes typically use d-left hashing to obtain balanced allocations of elements into buckets, and then use hashed fingerprints within each bucket to store information associated wi... |

2 |
The variable-increment counting Bloom filter,” Comnet
- Rottenstreich, Kanizo, et al.
- 2011
(Show Context)
Citation Context ...lse positive rates goes to 0. (iii) VI −CBF obtains a lower counter overflow probability bound than the classical bound for CBF from [1]. Proof Outline: (The full proof is longer, and can be found in =-=[15]-=-.) Given a CBF withm ≥ m0 = 10 counters of four bits and k hash functions, we build a VI −CBF with DL=4 = ⌋ counters of eight bits and the same number k of hash functions. This VI −CBF can fit within ... |

2 | Cache-, hash- and space-efficient Bloom filters
- Putze, Sanders, et al.
- 2007
(Show Context)
Citation Context ...h entry. Otherwise, it determines that x /∈ S. Memory Throughput — In order to increase the memory throughput, we can implement the two schemes using the ideas implemented in the Blocked Bloom Filter =-=[16]-=-. For each element, all hash functions are mapped into a single block in the memory, i.e. a single memory word. Although this technique suffers from a higher false positive rate, it is clearly energy-... |

1 |
Yossi Kanizo and Isaac Keslassy, “The variableincrement Counting Bloom Filter
- Rottenstreich
(Show Context)
Citation Context ...le increment, is incremented by the element v gi(x) ∈ D, thus the increment is TABLE II Bh VARIATIONS WITH THE CONSTRAINT vℓ ≤ 15 h, ℓ Bh Example Optimal k FPR (for 30 bits per element) h = 1, ℓ = 15 =-=[1,15]-=- 2 0.04630 h = 2, ℓ = 5 {1,2,4,8,13} 4 0.01963 h = 3, ℓ = 4 {1,2,5,14} 5 0.01521 h = 12, ℓ = 3 {1,2,15} 5 0.02902 at most vℓ. Since each of the counters has a fixed size we must bound this increment. ... |

1 |
Enhancing counting Bloom filters through Huffman-coded multilayer structures
- Ficara, Pietro, et al.
(Show Context)
Citation Context ...currently queued (set membership), how many of its packets are queued (counter representation), whether it is in a given state (state representation), or where to forward its packets (IP lookups) [1]–=-=[6]-=-. CBFs are often used because they can be easily implemented in hardware. In particular, element insertions, deletions and queries can be implemented in CBFs using a constant complexity that is essent... |

1 |
A variable length counting Bloom filter
- Li, Wang, et al.
- 2010
(Show Context)
Citation Context ...in in Section V how the same hierarchical compression idea could be combined with VI −CBF , and show in Section VI how this combination can achieve even better results. • VL-CBF (Variable Length CBF) =-=[8]-=-: This algorithm uses a variable-length coding, such as the Huffman coding, to represent counters with a variable number of bits. A similar coding can be used for efficient representation of counters ... |

1 | Access-efficient balanced Bloom filters
- Kanizo, Hay, et al.
- 2012
(Show Context)
Citation Context ...ble to recover it with high probability upon deletion (i.e. decrement) of some of the states. • And many additional schemes like Counter Braids [4], Selective CBF [9] and Access-Efficient Balanced BF =-=[10]-=-. However, there are also a few variants to CBF that the generic variable-increment idea does not necessarily improve. For instance, [11] uses counters to estimate item multiplicities and the suggeste... |