Citation Context

...he D-dimensional attribute space. Since the computational costs for all attributes are unacceptably high in practice, we use a simple but effective semantic tool, i.e., Latent Semantic Indexing (LSI) =-=[12, 13]-=- to generate semantically correlated groups as shown in Section 3. The notion of semantic correlation has been used in many systems designs, optimizations and real-world applications. In what follows ...

Citation Context

...contents, while ignoring filescontext that is utilized by most users in organizing and searching their data [38]. Furthermore, typical techniques successful for the web search, such as HITS algorithm =-=[39]-=- and Googlessearch engine [40], leverage tagged and contextual links thatsdo not inherently, let alone explicitly, exist in large-scale filessystems.s6.2 Directory-based Subtree PartitioningsSubtree-p...

Citation Context

...antics, denoted as R-tree. On thesother hand, each Bloom filter embedded within an R-tree nodesfor point query is assigned 1024 bits with k = 7 hash functions tosfit memory constraints. We select MD5 =-=[34]-=- as the hash functionsfor its relatively fast implementation. The value of an attributesis hashed into 128 bits by calculating its MD5 signature, whichsis then divided into four 32-bit values. We set ...

Citation Context

...ponents respectively from user and system views with automatic configuration to match query patterns.s2.1 OverviewsA semantic R-tree as shown on the right of Figure 1 issevolved from classical R-tree =-=[26]-=- and consists of index unitss(i.e., non-leaf nodes) containing location and mapping information and storage units (i.e., leaf nodes) containing file metadata, both of which are hosted on a collection ...

Citation Context

...either anson-line multicast-based approach or an off-line pre-computation-based approach to locating the corresponding R-tree node.sSpecifically, for a point query, the home unit checks Bloom filters =-=[27]-=- stored locally in a way similar to the group-based hierarchical Bloom-filter array approach [28] and, for a complexsquery, the home unit checks the Minimum Bounding Rectangless(MBR) [26] to determine...

Citation Context

.... Similar workload scale-up approaches havesalso been used in other studies [28, 32].sWe compare SmartStore with two baseline systems. The firstsone is a popular database approach that uses a B+ tree =-=[33]-=- to index each metadata attribute, denoted as DBMS that here doessnot take into account database optimization. The second one issa simple, non-semantic R-tree-based database approach that organizes ea...

Citation Context

...he D-dimensional attribute space. Since the computational costs for all attributes are unacceptably high in practice, we use a simple but effective semantic tool, i.e., Latent Semantic Indexing (LSI) =-=[12, 13]-=- to generate semantically correlated groups as shown in Section 3. The notion of semantic correlation has been used in many systems designs, optimizations and real-world applications. In what follows ...

Citation Context

...r inner product.sDue to space limitation, this paper only gives basic introductionsto LSI and more details can be found in [12, 13].sWhile there are other tools available for grouping, such as Kmeans =-=[30]-=-, we choose LSI because of its high efficiency and easysimplementation. The K-means [30] algorithm exploits multi-dimensional attributes of n items to cluster them into K (K ≤ n)spartitions. While the...

Citation Context

...mputing [1]. As the storage capacity is approaching Exabytes and the number of files stored is reaching billions, directory-tree based metadata management widely deployed in conventional file systems =-=[2, 3]-=- can no longer meet the requirements of scalability and functionality. For the next-generation large-scale storage systems, new metadata organization schemes are desired to meet two critical goals: (1...

Citation Context

... the next-generation file systems, metadata accesses will very likely become a severe performance bottleneck as metadata-based transactions not only account for over 50% of all file system operations =-=[4, 5]-=- but also result in billions of pieces of metadata in directories. Given the sheer scale and complexity of the data and Permission to make digital or hard copies of all or part of this work for Permis...

Citation Context

...ta, which exploits higherdimensional static and dynamic attributes, or higher-dimensional localities than the simple temporal or spatial locality utilized in existing approaches. Semantic correlation =-=[10]-=- comes from the exploitation of highdimensional attributes of metadata. To put things in perspective, linear brute-force approach uses 0-dimensional correlation while spatial/temporal locality approac...

Citation Context

...nce.sDatabase research community has argued that existingsDBMS for general-purpose applications would not be a “onessize fit all” solution [44] and improvements may result from semantic-based designs =-=[45]-=-.s7. ConclusionsThe paper presents a new paradigm for organizing file metadata for next-generation file systems, called SmartStore, by exploiting file semantic information to provide efficient and sca...

Citation Context

...0 files that are closest to this description?". From a system’s point of view, SmartStore may help optimize storage system designs such as de-duplication, caching and prefetching. Data de-duplication =-=[22, 23]-=- aims to effectively and efficiently remove redundant data and compress data into a highly compact form for the purpose of data backup and archiving. One of the key problems is how to identify multipl...

Citation Context

...anges that have not been directly updated in the original replicas. This method eliminates many small, random and frequentsvisits to the index and has been widely used in most versioningsfile systems =-=[8, 9, 31]-=-.sIn order to maintain semantic correlation and locality, Smart-sStore creates versions for every group, represented as the firstlevel index unit that has been replicated to other index units. Atstime...

Citation Context

...cess locality and “hot spot” data, to enhance system performance.sDatabase research community has argued that existingsDBMS for general-purpose applications would not be a “onessize fit all” solution =-=[44]-=- and improvements may result from semantic-based designs [45].s7. ConclusionsThe paper presents a new paradigm for organizing file metadata for next-generation file systems, called SmartStore, by expl...

Citation Context

...ncy of content-based search heavily depends on files thatscontain explicitly understandable contents, while ignoring filescontext that is utilized by most users in organizing and searching their data =-=[38]-=-. Furthermore, typical techniques successful for the web search, such as HITS algorithm [39] and Googlessearch engine [40], leverage tagged and contextual links thatsdo not inherently, let alone expli...

Citation Context

...ile metadata prefetching performance. The probability of inter-file access is found to be up to 80% when considering four typical file system traces. Our preliminary results based on these and the HP =-=[17]-=-, MSN [18], and EECS [19] traces further show that exploiting semantic correlation of multi-dimensional attributes can help prune up to 99.9% search space [20]. Therefore, in this paper we proposed a ...

Citation Context

...of real traces where 45% requests from all 11,568,086 requests visit only 6.5% files from all 65,536 files that are sorted by file popularity. Measurement of large-scale network file system workloads =-=[15]-=- further verifies that fewer than 1% clients issue 50% file requests and over 60% re-open operations take place within one minute. Semantic correlation can be exploited to optimize system performance....

Citation Context

... the next-generation file systems, metadata accesses willsvery likely become a severe performance bottleneck as metadata-based transactions not only account for over 50% of all filessystem operations =-=[4, 5]-=- but also result in billions of pieces ofsmetadata in directories. Given the sheer scale and complexity ofsthe data and metadata in such systems, we must seriously ponder a few critical research probl...

Citation Context

...ta prefetching performance. The probability of inter-file access is found to be up to 80% when considering four typical file system traces. Our preliminary results based on these and the HP [17], MSN =-=[18]-=-, and EECS [19] traces further show that exploiting semantic correlation of multi-dimensional attributes can help prune up to 99.9% search space [20]. Therefore, in this paper we proposed a novel dece...

Citation Context

... billions of records. Some databasesvendors developed parallel databases to support large-scalesdata management, such as Oracle’s Real Application Cluster database [42] and IBM’s DB2 Parallel Edition =-=[43]-=-, by using a complete relational model with transactions. Although successfulsfor managing relational databases, existing database management systems (DBMS) do not fully satisfy the requirements ofsme...

Citation Context

...00. and/or a fee. SC 09 November 14-20, 2009, Portland, Oregon, USA (c) 2009 ACM 978-1-60558-744-8/09/11 ...$10.00. metadata in such systems, we must seriously ponder a few critical research problems =-=[6, 7]-=- such as “How to efficiently extract useful knowledge from an ocean of data?", “How to manage the enormous number of files that have multi-dimensional or increasingly higher dimensional attributes?", ...

Citation Context

...et alone explicitly, exist in large-scale filessystems.s6.2 Directory-based Subtree PartitioningsSubtree-partitioning based approaches have been widelysused in recent studies, such as Ceph [3], GIGA+ =-=[41]-=-, Farsites[2] and Spyglass [8]. Ceph [3] maximizes the separation between data and metadata management by using a pseudorandom data distribution function to support a scalable andsdecentralized placem...

Citation Context

... body of published work. Spyglass [8] reports that the locality ratios are below 1% in many given traces, meaning that correlated files are contained in less than 1% of the directory space. Filecules =-=[14]-=- reveals the existence of file grouping by examining a large set of real traces where 45% requests from all 11,568,086 requests visit only 6.5% files from all 65,536 files that are sorted by file popu...

Citation Context

... served accurately by Bloom filters.sFigure 9. Average hit rate for point query. 5.4.2 Complex QueriessWe adopt “Recall” as a measure for complex query qualitysfrom the field of information retrieval =-=[35]-=-. Given a query q, wesdenote T(q) the ideal set of K nearest objects and A(q) the actualsneighbors reported by SmartStore. We define recall assrecall =s|T(q) ∩ A(q)|sT(q)s(a) Top-8 NN query.s(b) Range...

Citation Context

...he number of sub-traces replayed concurrently is denoted as the Trace Intensifying Factor (TIF) as shownsin Table 1, 2 and 3. Similar workload scale-up approaches havesalso been used in other studies =-=[28, 32]-=-.sWe compare SmartStore with two baseline systems. The firstsone is a popular database approach that uses a B+ tree [33] to index each metadata attribute, denoted as DBMS that here doessnot take into ...

Citation Context

...loitation of highdimensional attributes of metadata. To put things in perspective, linear brute-force approach uses 0-dimensional correlation while spatial/temporal locality approaches, such as Nexus =-=[11]-=- and Spyglass [8], use 1-dimensional correlation, which can be considered as special cases of our proposed approach that considers higher dimensional correlation. The main benefit of using semantic co...

Citation Context

...acent groups where duplicate copies can be placed together with high probability to narrow the search space and further facilitate fast identification.sOn the other hand, caching [24] and prefetching =-=[25]-=- areswidely used in storage systems to improve I/O performancesby exploiting spatial or temporal access locality. However, theirsperformance in terms of hit rate varies largely from applicationsto app...

Citation Context

...n operations take place within one minute. Semantic correlation can be exploited to optimize system performance. Our research group has proposed metadata prefetching algorithms, Nexus [11] and FARMER =-=[16]-=-, in which both file access sequences and semantic attributes are considered in the evaluation of the correlation among files to improve file metadata prefetching performance. The probability of inter...

Citation Context

... into the same or adjacent groups where duplicate copies can be placed together with high probability to narrow the search space and further facilitate fast identification. On the other hand, caching =-=[24]-=- and prefetching [25] are widely used in storage systems to improve I/O performance by exploiting spatial or temporal access locality. However, their performance in terms of hit rate varies largely fr...

Citation Context

...ing complex queries in large-scale file systems. 1. INTRODUCTION Fast and flexible metadata retrieving is a critical requirement in the next-generation data storage systems serving high-end computing =-=[1]-=-. As the storage capacity is approaching Exabytes and the number of files stored is reaching billions, directory-tree based metadata management widely deployed in conventional file systems [2, 3] can ...

Citation Context

...00. and/or a fee. SC 09 November 14-20, 2009, Portland, Oregon, USA (c) 2009 ACM 978-1-60558-744-8/09/11 ...$10.00. metadata in such systems, we must seriously ponder a few critical research problems =-=[6, 7]-=- such as “How to efficiently extract useful knowledge from an ocean of data?", “How to manage the enormous number of files that have multi-dimensional or increasingly higher dimensional attributes?", ...

Citation Context

...mputing [1]. As the storage capacity is approaching Exabytes and the number of files stored is reaching billions, directory-tree based metadata management widely deployed in conventional file systems =-=[2, 3]-=- can no longer meet the requirements of scalability and functionality. For the next-generation large-scale storage systems, new metadata organization schemes are desired to meet two critical goals: (1...