Cyotek WebCopy HelpThe General settings category allows you to configure the website to crawl and how it should be crawled.
- Specifying the web site to copy
- Including sub and sibling domains
- Downloading all resources
- Limiting scans by depth
- Limiting scans by distance
- Extracting inline data
See Also
Configuring the Crawler
Working with local files
- Extracting inline data
- Remapping extensions
- Remapping local files
- Updating local time stamps
- Using query string parameters in local filenames
Controlling the crawl
- Content types
- Crawling above the root URL
- Crawling additional hosts
- Crawling additional root URLs
- Downloading all resources
- Including sub and sibling domains
- Limiting downloads by file count
- Limiting downloads by size
- Limiting scans by depth
- Limiting scans by distance
- Scanning data attributes
- Setting speed limits
- Working with Rules
JavaScript
Security
Modifying URLs
Creating a site map
Advanced
- Aborting the crawl using HTTP status codes
- Defining custom headers
- Following redirects
- HEAD vs GET for preliminary requests
- HTTP Compression
- Origin reports
- Saving link data in a Crawler Project
- Setting cookies
- Setting the web page language
- Specifying a User Agent
- Specifying accepted content types
- Using Keep-Alive