Abstract

Some previous works show that a web page can be partitioned to multiple segments or blocks, and usually the importance of those blocks in a page is not equivalent. Also, it is proved that differentiating noisy or unimportant blocks from pages can facilitate web mining, search and accessibility. But in these works, no uniform approach or model is presented to measure the importance of different portions in web pages. Through a user study, we found that people do have a consistent view about the importance of blocks in web pages. In this paper, we investigate how to find a model to automatically assign importance values to blocks in a web page. We define the block importance estimation as a learning problem. First, we use the VIPS (VIsion-based Page Segmentation) algorithm to partition a web page into semantic blocks with a hierarchy structure. Then spatial features (such as position, size) and content features (such as the number of images and links) are extracted to construct a feature vector for each block. Based on these features, learning algorithms, such as SVM and neural network, are applied to train various block importance models. In our experiments, the best model can achieve the performance with Micro-F1 79 % and Micro-Accuracy 85.9%, which is quite close to a person’s.

Citations

2569	The Anatomy of a Large-Scale Hypertextual Web Search Engine - Brin, Page - 1998
448	Solving Multiclass Learning Problems via Error-Correcting Output Codes - Dietterich, Bakiri - 1995
180	Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition - Cover - 1965
81	Errorcorrecting output codes: A general method for improving multiclass inductive learning programs - Dietterich, Bakiri - 1991
63	Template Detection via Data Mining and its Applications, WWW - Bar-Yossef, Rajagopalan - 2002
62	X.: Eliminating noisy information in web pages for data mining - Yi, Liu, et al.
59	Discovering Informative Content Blocks from Web Documents - Lin, Ho - 2002
58	DOM-based Content Extraction of HTML Documents - Gupta, Kaiser, et al. - 2003
56	Improving pseudo-relevance feedback in web information retrieval using web page segmentation - Yu, Cai, et al. - 2003
51	W.-Y.Ma, “VIPS: a visionBased page segmentation algorithm”, Microsoft - Cai, Wen - 2003
45	Principles of risk minimization for learning theory - Vapnik - 1992
33	Function-based Object Model Towards Website Adaptation - Chen, Zhou, et al. - 2001
32	Support Vector Machines for Multiclass Classification - Mayoraz, Alpaydin - 1999
19	Task Force on Bias-Free Language. Guidelines for Bias-Free Writing - Schwartz - 1995
6	Recognition of Common Areas in a Web Page Using Visual Information: a possible application in a page classification - Kovacevic, Diligenti, et al. - 2002
6	Automatic Browsing - Liu, Xie, et al. - 2003
2	Page Cleaning for Web Mining through Feature - Yi, Liu, et al. - 2003