As it is impractical for WebCopy to crawl the Internet, by default the crawling is limited to only the domain of the crawl URL, but supports an easy option to include sub or sibling domains without requiring the use of custom rules.

  1. From the Project Properties dialog, select the General category
  2. Select a mode from the Crawl Mode group
OptionNotes
Site OnlyOnly crawls URLs that match the host name specified in the crawl URL
Sub domainsIncludes any sub domains of the host URL
Sibling domainsIncludes both sub domains and sibling domains of the host URL
EverythingWill crawl any discovered HTTP or HTTPS URL unless excluded via other settings

Important

Use of the Everything option is not recommended and should only be used on sites which are self contained or where rules are used to explicitly exclude addresses. Use of this option may cause WebCopy to become unstable.

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features

In this article