Multi-threaded crawling

Multi-threading allows WebCopy to download multiple resources simultaneously rather than one at a time. By increasing the number of concurrent requests, you can significantly reduce the time required to complete a crawl, especially when downloading from servers with high latency or when crawling large websites.

Setting the maximum concurrent requests

The Maximum Concurrent Requests setting determines how many download requests are attempted at once whilst downloading content. The default value is 1, which processes requests sequentially. Higher values enable parallel downloading and can improve crawl speed.

To set the maximum number of requests

1. Open the Options dialogue box and select the Advanced category
2. In the Multi-threading group, enter the maximum number of concurrent requests the crawler will support, between 1 and 12

Choosing an appropriate value

When selecting the number of concurrent requests, consider the following factors:

• Server capacity - Most web servers can handle multiple simultaneous connections, but excessive concurrent requests may trigger rate limiting or result in connection errors
• Network bandwidth - Your available bandwidth may limit the practical benefit of very high concurrency levels
• System resources - Each concurrent request consumes memory and CPU resources on your local system
• Crawl politeness - Fewer concurrent requests are more respectful to the target server and reduce the risk of being blocked