Low-Latency, High-Throughput Access to Static Global Resources within the Hadoop Framework

Low-Latency, High-Throughput Access to Static Global Resources within the Hadoop Framework (2009)

Cached

Download Links

by Jimmy Lin , Shravya Konda , Samantha Mahindrakar

Versions

Version 0

Version History

Metadata Version 0

Datum	Value	Source
TITLE	Low-Latency, High-Throughput Access to Static Global Resources within the Hadoop Framework	SVM HeaderParse 0.2
AUTHOR NAME	Jimmy Lin	SVM HeaderParse 0.2
AUTHOR AFFIL	1; The iSchool, College of Information Studies; 2; Department of Computer Science; University of Maryland	SVM HeaderParse 0.2
AUTHOR ADDR	College Park, Maryland 20742	SVM HeaderParse 0.2
AUTHOR NAME	Shravya Konda	SVM HeaderParse 0.2
AUTHOR AFFIL	1; The iSchool, College of Information Studies; 2; Department of Computer Science; University of Maryland	SVM HeaderParse 0.2
AUTHOR ADDR	College Park, Maryland 20742	SVM HeaderParse 0.2
AUTHOR NAME	Samantha Mahindrakar	SVM HeaderParse 0.2
AUTHOR AFFIL	1; The iSchool, College of Information Studies; 2; Department of Computer Science; University of Maryland	SVM HeaderParse 0.2
AUTHOR ADDR	College Park, Maryland 20742	SVM HeaderParse 0.2
ABSTRACT	Hadoop is an open source implementation of Google’s MapReduce programming model that has recently gained popularity as a practical approach to distributed information processing. This work explores the use of memcached, an open-source distributed in-memory object caching system, to provide low-latency, high-throughput access to static global resources in Hadoop. Such a capability is essential to a large class of MapReduce algorithms that require, for example, querying language model probabilities, accessing model parameters in iterative algorithms, or performing joins across relational datasets. Experimental results on a simple demonstration application illustrate that memcached provides a feasible general-purpose solution for rapidly accessing global key-value pairs from within Hadoop programs. Our proposed architecture exhibits the desirable scaling characteristic of linear increase in throughput with respect to cluster size. To our knowledge, this application of memcached in Hadoop is novel. Although considerable opportunities for increased performance remain, this work enables implementation of algorithms that do not have satisfactory solutions at scale today. 1	SVM HeaderParse 0.2
YEAR	2009	SVM HeaderParse 0.2
CITATIONS	29 found	ParsCit 1.0