The default behaviour of the crawl engine is to ignore errors and continue with the crawl. However, it is possible to instruct the crawler to automatically cancel if a request returns a specific response code, such as 429 (Too Many Requests) or 503 (Service Unavailable).

Configuring abort codes

  1. From the Project Properties dialogue, select the Advanced category
  2. In the Abort HTTP Status Codes group, enter a comma separated list of status codes into the Status codes field

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.10 (buildref #186.15944), last modified 2024-08-18. Generated 2024-08-18 08:00 using Cyotek HelpWrite Professional version 6.20.0