User Agent Category

See Also

Configuring the Crawler

• Configuring the crawler
• Specifying the web site to copy
• Configuring the output location

Working with local files

• Extracting inline data
• Remapping extensions
• Remapping local files
• Updating local time stamps
• Using query string parameters in local filenames

Controlling the crawl

• Content types
• Crawling multiple URLs
• Crawling outside the base URL
• Downloading all resources
• Including additional domains
• Including sub and sibling domains
• Limiting downloads by file count
• Limiting downloads by size
• Limiting scans by depth
• Limiting scans by distance
• Scanning data attributes
• Setting speed limits
• Working with Rules

JavaScript

• Working with JavaScript enabled websites

Security

• Crawling private areas
• Manually logging into a website
• TLS/SSL certificate options
• Working with Forms
• Working with Passwords

Modifying URLs

• Fixing sites using mixed prefixes
• Forcing HTTPS
• Ignoring URL Case
• Transforming URLs

Creating a site map

• Creating a site map

Advanced

• Aborting the crawl using HTTP status codes
• Cookies
• Defining custom headers
• HEAD vs GET for preliminary requests
• HTTP Compression
• Origin reports
• Redirects
• Saving link data in a Crawler Project
• Setting the web page language
• Specifying a User Agent
• Specifying accepted content types
• Using Keep-Alive

Deprecated features

• Configuring domain aliases
• Query String Manipulation
• Specifying default documents
• Using a proxy server