While you may not wish to crawl external sites, it is still possible to download any files directly linked from the site you are crawling without having to change multiple settings. When the Download all resources option is set, WebCopy will automatically download any external file, as long as the reported content type is not text/html. The downloaded file will not be crawled, allowing easy downloading of linked images, sounds and other files.

Enabling the downloading of all resources

  1. From the Project Properties dialogue, select the General category
  2. Check the Download all resources option to always download non-HTML content

Note

Rules can still be used to exclude content even when this setting is enabled.

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.9 (buildref #182.15707), last modified 2024-03-15. Generated 2024-03-15 22:36 using Cyotek HelpWrite Professional version 6.19.1