The default behaviour of the crawl engine is to ignore errors and continue with the crawl. However, it is possible to instruct the crawler to automatically cancel if a request returns a specific response code, such as 429 (Too Many Requests) or 503 (Service Unavailable).

Configuring abort codes

  1. From the Project Properties dialog, select the Advanced category
  2. In the Abort HTTP Status Codes group, enter a comma separated list of status codes into the Status codes field

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features