By default, WebCopy will send a HEAD command to all URL's to check their content types before downloading the content via a GET. You can disable this behavior and always GET if required, however this is not recommended unless not supported by the website you are trying to crawl.
To enable or disable header checking
- From the Project Properties dialog, click the Advanced category
- Check or uncheck the Use header checking option
If header checking is disabled, the full contents of any non-excluded file will always be downloaded regardless of it's content type. Where possible, header checking should be left enabled.
- Customizing Projects
- Specifying the web site
- Specifying the save folder
- Remapping extensions
- Specifying how a website is crawled
- Specifying default documents
- Crawling additional root URL's
- Updating local time stamps
- Scanning custom attributes
- Fixing sites using mixed prefixes
- Extracting inline data