By default, when crawling a web site, Sitemap Creator will send a application defined user agent. As some web sites use user agent sniffing to determine what content to send, or block unrecognised agents, Sitemap Creator allows you to send a pre-defined or custom agent.
- From the Project Properties dialogue, select the User Agent category
- To use the application defined user agent, select Use default user agent
- Alternatively, to apply a custom agent, select Use custom user agent and enter an agent string in the space provided or choose from the list