An Efficient Smartcrawler for Harvesting Web Interfaces of a Two-Stage Crawler

Volume 5 Issue 4 September - November 2016

Research Paper

An Efficient Smartcrawler for Harvesting Web Interfaces of a Two-Stage Crawler

Nikitha Sharma*, V. Sowmya Devi**

* M.Tech Scholar, Department of Computer Science and Engineering, Gitam University, Telangana, India.

** Assistant Professor, Department of Computer Science and Engineering, Gitam University, Telangana, India.

Sharma. N and Devi. V. S (2016). An Efficient Smart Crawler for Harvesting Web Interfaces of Two-Stage Crawler. i-manager's Journal on Information Technology, 5(4), 20-25. https://doi.org/10.26634/jit.5.4.10334

Abstract

The WWW is an incomprehensible collection of one thousand millions of pages containing tera bytes of information organized in many servers using HTML. The extent of this gathering itself is an imposing snag in recovering fundamental and applicable data. This made web indexes a vital part of our service. The venture expects to make a keen WebCrawler for an idea based semantic based internet searcher. The authors intend to raise the potency of the Concept Based Semantic Search Motor by utilizing the SmartCrawler. They proposed a two phase architecture to be specific SmartCrawler, for smartly collecting incredible web interfaces. On the premier level, SmartCrawler performs site-based crawling for hunting down key pages with the brace of web search tools, abstaining from going by a prodigious amount of pages. To finish more correct answers for a drew in crawl, SmartCrawler position locales to sort out significantly corresponded ones for a devoted topic. In the secondary level, SmartCrawler finishes quick on-site looking by uncovering most relevant associations with associate in nursing versatile association situations. To evacuate incomplete destinations on setting off to some particularly applicable associations in releasing web registries, we plot an association tree data structure to reach a broader degree for a website. The outcomes occur on a game plan of those ranges, which show the adaptability and precision of the proposed crawler structure, which competently recoups significant web interfaces from sizable voluminous-scale neighborhoods and finishes higher rates than other crawler's results.