The default behaviour of the crawl engine is to ignore errors and continue with the crawl. However, it is possible to instruct the crawler to automatically cancel if a request returns a specific response code, such as 429 (Too Many Requests) or 503 (Service Unavailable).