Many web servers can compress data prior to sending it, using a variety of different compression methods. When a client makes a request of a server, it includes the Accept-Encoding header which informs the server which encoding methods the client supports. WebCopy supports the following compression methods

  • Compress (Legacy)
  • Deflate
  • GZip
  • Brotli (Non-standard)
  • BZip2 (Non-standard)
  • Identity (no compression, not directly selectable)

Note

Disabling compression will cause the Identity value to be sent for the Accept-Encoding header, informing the web server not to compress content before serving it.

Important

Encoding options generally only apply to static content, such as HTML, CSS and JavaScript. Other files such as Pdf or Zip are already compressed and normally won't be recompressed by a server.

To enable or disable compression

  1. From the Project Properties dialogue, select the HTTP Compression option group.
  2. Check or uncheck the types of compression methods you wish to support.

Tip

It is recommended to always ensure that at least Deflate and GZip options are enabled as this helps makes downloads of HTML and static content smaller and faster.

Some servers may not support all available compression options. If you receive 406 (Not Accepted) errors when trying to copy a website, specifying an unsupported encoding could be the cause.

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.10 (buildref #186.15944), last modified 2024-08-18. Generated 2024-08-18 08:00 using Cyotek HelpWrite Professional version 6.20.0