HTTP persistent connection, also called HTTP keep-alive, or HTTP connection reuse, is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair. This can improve crawl performance but can sometimes cause requests to fail with a protocol violation error.

To disable keep-alive

  1. From the Project Properties dialog, select the Advanced category
  2. Uncheck the Keep alive option

Note

It is not recommended this option unless you are having specific problems with a given website.

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features