Probabilistic graphical model for efficient focused web crawling

Focused crawlers aim to search only a subset of the web related to a specific topic whose performance mainly depends on the accuracy of predicting the relevance of a newly seen URL. In this paper a novel focused crawling method based on sequential probabilistic model and reinforcement learning is proposed. The crawled website is modeled as a graph and a model of Conditional Random Fields is trained over it which learns the optimal paths that lead to relevant pages on the websites through exploit a variety of features in the hyperlinks, HTML tags, and page sequence. Then, a reinforcement learning algorithm is used to map each hyperlink on the crawl frontier to a future discounted reward as its priority. Experimental results using a large number of web pages from diverse domains show that our technique provides better performance than traditional focused crawlers.