By default, when crawling a web site, WebCopy will send a application defined user agent. As some web sites use user agent sniffing to determine what content to send, or block unrecognised agents, WebCopy allows you to send a pre-defined or custom agent.

To configure the user agent

  1. From the Project Properties dialogue, select the User Agent category
  2. To use the application defined user agent, select Use default user agent
  3. Alternatively, to apply a custom agent, select Use custom user agent and enter an agent string in the space provided or choose from the list

See Also

Configuring the Crawler

Working with local files

Controlling the crawl



Modifying URLs

Creating a site map


Deprecated features

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.10 (buildref #185.15779), last modified 2024-03-31. Generated 2024-03-31 14:04 using Cyotek HelpWrite Professional version 6.19.1