As it is impractical for WebCopy to crawl the Internet, by default the crawling is limited to only the domain of the crawl URL, but supports an easy option to include sub or sibling domains without requiring the use of custom rules.
| Option | Notes |
|---|---|
| Site Only | Only crawls URLs that match the host name specified in the crawl URL |
| Sub domains | Includes any sub domains of the host URL |
| Sibling domains | Includes both sub domains and sibling domains of the host URL |
| Everything | Will crawl any discovered HTTP or HTTPS URL unless excluded via other settings |
Use of the Everything option is not recommended and should only be used on sites which are self contained or where rules are used to explicitly exclude addresses. Use of this option may cause WebCopy to become unstable.