Webpage duplicate detection using combined POS and sequence alignment algorithm

Mohamed Elhadi*, Amjad Al-Tobi

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Combined syntactical categories and sequence alignment algorithms are implemented and used to weed-out duplicate and near-duplicate web-pages from search engine results. The syntactical structures manifested as POS-tags were pre-processed using a POS tagger converting parts of a webpage's text into a string of tags. The produced string was then subjected into the longest Common Sequence (LCS) techniques (as is commonly done in computational biology), to detect duplicate and near-duplicate webpages. The process of tagging and aligning was based on set of sentences extracted from the web page as a representative of the pages. The query-keywords are used as a basis for sentence extraction. Results obtained from experiments performed have shown that such a combined approach can provide very interesting similarity calculation and re-ranking measure. This can be used with reasonable efficiency to detect duplications on search results generated by search engines such as Google. Similarity measurements obtained can be further used as a basis for text analysis of the search results allowing the detection of duplicate and near duplicates and clustering of documents in general.

Original languageEnglish
Title of host publication2009 WRI World Congress on Computer Science and Information Engineering, CSIE 2009
Pages630-634
Number of pages5
DOIs
Publication statusPublished - 2009
Externally publishedYes
Event2009 WRI World Congress on Computer Science and Information Engineering, CSIE 2009 - Los Angeles, CA, United States
Duration: Mar 31 2009Apr 2 2009

Publication series

Name2009 WRI World Congress on Computer Science and Information Engineering, CSIE 2009
Volume1

Conference

Conference2009 WRI World Congress on Computer Science and Information Engineering, CSIE 2009
Country/TerritoryUnited States
CityLos Angeles, CA
Period3/31/094/2/09

Keywords

  • Copy detection
  • Duplicate
  • LCS
  • Longest common sequence
  • Part-of-speech
  • POS
  • Search engine

ASJC Scopus subject areas

  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems
  • Software

Cite this