Many web servers can compress data prior to sending it, using a variety of different compression methods. When a client makes a request of a server, it includes the
Accept-Encoding header which informs the server which encoding methods the client supports. Sitemap Creator supports the following compression methods
Disabling compression will mean the Accept-Encoding header will not be sent with requests, informing the web server not to compress content before serving it.
To enable or disable compression
- From the Project Properties dialog, select the HTTP Compression option group.
- Check or uncheck the types of compression methods you wish to support.
Some servers may not support all available compression options. If you receive 406 (Not Accepted) errors when trying to copy a website, specifying an unsupported encoding could be the cause.
- Customizing Projects
- Specifying the web site
- Configuring site maps
- Specifying default documents
- Replacing the host name
- Crawling additional root URL's
- Canonical URL's
- Fixing sites using mixed prefixes
Advanced Project Customization
- Advanced Project Customization
- Configuring the behaviour of link maps
- Configuring domain aliases
- Specifying a User Agent
- Modifying page titles
- Disabling header checking
- Query String Manipulation
- Overwriting read only files
- Specifying how a website is crawled
- Following redirects
- Crawling above the root URL
- Aborting the crawl based on HTTP status code
- TLS/SSL certificate options