Project Description
SharkCrawler is a simple web page crawler written in C# for demonstration of how regular crawler is working.
Features
- Crawl from a specified starting page
- Pause and Resume crawling
- Specify how many threads are used to crawl
- Use local file system to store crawled pages
- Track requests and errors
