Recursive Crawling(out-links)

[rica-v3]

Crawling web pages and out-links

• Crawler keeps crawling links to a depth, which is pre-configured by user.
• A depth of a link from an initial link and domain is recored.
• Outlinks in a page are added to the 'CrawlingQueue' ('CrawlingGenerator genearates 'Crawling' with them and provides 'CrawlingSchedueler')

Above procedures are repeated untill the miximum depth of link

[rica-v3]

The algorithm is breadth-first search.
Better to add depth-first search as a option set by user.
Additionally a concept of time-sharing can improve performance of traversing pages.