Structure based Data Extraction from Hidden Web Sources: A Review
---
paper_title: Automatic Data Records Extraction from List Page in Deep Web Sources

paper_content:

with the explosive growth and popularity of the World Wide Web, a wealth of online e-commerce information resources become available. List pages in these web sites are usually automatically generated from the back-end DBMS using scripts. In order to provide value-added services and convenience for users, it is very necessary to integrate web sources of the same domain. Given the huge number of these web pages, it is difficult and even impossible to use a manual approach to extract data records from these list pages on a large scale. According to characteristics of the template-based list pages, in this paper, we propose a LBDRF algorithm to solve the problem of automatic data records extraction from web pages in Deep Web. Our experimental results show that the proposed method performs well.
---
paper_title: Mining data records in Web pages

paper_content:

A large amount of information on the Web is contained in regularly structured objects, which we call data records. Such data records are important because they often present the essential information of their host pages, e.g., lists of products or services. It is useful to mine such data records in order to extract information from them to provide value-added services. Existing automatic techniques are not satisfactory because of their poor accuracies. In this paper, we propose a more effective technique to perform the task. The technique is based on two observations about data records on the Web and a string matching algorithm. The proposed technique is able to mine both contiguous and non-contiguous data records. Our experimental results show that the proposed technique outperforms existing techniques substantially.
---
paper_title: Automatic Data Records Extraction from List Page in Deep Web Sources

paper_content:

with the explosive growth and popularity of the World Wide Web, a wealth of online e-commerce information resources become available. List pages in these web sites are usually automatically generated from the back-end DBMS using scripts. In order to provide value-added services and convenience for users, it is very necessary to integrate web sources of the same domain. Given the huge number of these web pages, it is difficult and even impossible to use a manual approach to extract data records from these list pages on a large scale. According to characteristics of the template-based list pages, in this paper, we propose a LBDRF algorithm to solve the problem of automatic data records extraction from web pages in Deep Web. Our experimental results show that the proposed method performs well.
---
paper_title: VIPS : a Vision-based Page Segmentation Algorithm

paper_content:

A new web content structure analysis based on visual representation is proposed in this paper. Many web applications such as information retrieval, information extraction and automatic page adaptation can benefit from this structure. This paper presents an automatic top-down, tag-tree independent approach to detect web content structure. It simulates how a user understands web layout structure based on his visual perception. Comparing to other existing techniques such as DOM tree, our approach is independent to the HTML documentation representation. Our method can work well even when the HTML structure is quite different from the visual layout structure. Several experiments show the effectiveness of our method.
---

Title: Structure based Data Extraction from Hidden Web Sources: A Review

Section 1: Introduction
Description 1: Introduce the two main categories of the World Wide Web (Surface web and Hidden web) and the importance of extracting data from the Hidden web.

Section 2: Understanding Data Structures in Web Pages
Description 2: Describe the different types of data structures found in web pages: unstructured, semi-structured, and structured.

Section 3: Layout Based Data Region Finding (LBDRF)
Description 3: Explain the LBDRF system for data region identification in structured web pages, including its components and processes.

Section 4: Extracting the Data Records in the Data Region
Description 4: Discuss the method of extracting data records from identified data regions using the LBDRF algorithm.

Section 5: Mining Data Records in Web Pages (MDR)
Description 5: Introduce the MDR method proposed by Liu and Grossman for mining data records, including steps like building an HTML tag tree and identifying data regions.

Section 6: Building the HTML Tag Tree
Description 6: Detail the process of constructing an HTML tag tree to facilitate data record extraction.

Section 7: Mining Data Regions and Identifying Data Records
Description 7: Describe the steps for identifying and extracting data regions and records from a structured web page using the MDR technique.

Section 8: String Comparison Using Edit Distance
Description 8: Explain the string comparison method using normalized edit distance for identifying similar data regions.

Section 9: Machine Learning Based Approach for Table Detection
Description 9: Explore the machine learning approach proposed by Yalin Wang and Jianying Hu for detecting data-rich tables in web pages.

Section 10: Vision Based Approach for Data Extraction
Description 10: Present the vision-based approach by P. S. Hiremath and Siddu P. Algur for extracting data items using visual clues.

Section 11: Identification and Extraction of Data Regions
Description 11: Detail the method for identifying and extracting the most relevant data regions from web pages using visual-based techniques.

Section 12: Identification of Data Records and Items
Description 12: Explain the techniques for identifying data records and items from the extracted data regions.

Section 13: Conclusion and Future Scope
Description 13: Summarize the paper, discussing the advantages and disadvantages of the reviewed techniques, and propose future directions for data extraction systems.