Cyotek WebCopy HelpBy default, when crawling a web site, WebCopy will send a application defined user agent. As some web sites use user agent sniffing to determine what content to send, or block unrecognised agents, WebCopy allows you to send a pre-defined or custom agent.
To configure the user agent
- From the Project Properties dialogue, select the User Agent category
- To use the application defined user agent, select Use default user agent
- Alternatively, to apply a custom agent, select Use custom user agent and enter an agent string in the space provided or choose from the list
See Also
Configuring the Crawler
Working with local files
- Extracting inline data
- Remapping extensions
- Remapping local files
- Updating local time stamps
- Using query string parameters in local filenames
Controlling the crawl
- Content types
- Crawling multiple URLs
- Crawling outside the base URL
- Downloading all resources
- Including additional domains
- Including sub and sibling domains
- Limiting downloads by file count
- Limiting downloads by size
- Limiting scans by depth
- Limiting scans by distance
- Scanning data attributes
- Setting speed limits
- Working with Rules
JavaScript
Security
- Crawling private areas
- Manually logging into a website
- TLS/SSL certificate options
- Working with Forms
- Working with Passwords
Modifying URLs
Creating a site map
Advanced
- Aborting the crawl using HTTP status codes
- Cookies
- Defining custom headers
- HEAD vs GET for preliminary requests
- HTTP Compression
- Origin reports
- Redirects
- Saving link data in a Crawler Project
- Setting the web page language
- Specifying accepted content types
- Using Keep-Alive