Every HTTP request is made of one or more headers. Usually, WebCopy will take care of determining the correct set of headers to send. However, you can also define your own headers to include with requests.
Opening the custom headers editor
- From the Project Properties dialogue, expand the Advanced category and select Custom Headers
Adding a new header
- Click the Add button to add an empty header definition
- Enter the header information into the Name and Value fields
- Optionally, if you only wish the header to be sent with certain requests, enter the URL into the URL Expression field. This field supports regular expressions.
Important
If your expression includes any of the ^
, [
, .
, $
, {
, *
, (
, \
, +
, )
, |
, ?
, <
, >
characters and you want them to processed as plain text, you need to "escape" the character by preceding it with a backslash. For example, if your expression was application/epub+zip
this would need to be written as application/epub\+zip
otherwise the +
character would have a special meaning and no matches would be made. Similarly, if the expression was example.com
, this should be written as example\.com
, as .
means "any character" which could lead to unexpected matches.
Editing an existing header
- Select the header to edit from the list
- Modify the header properties using the Name, Value and URL Expression fields
Deleting a header
- Select the header you want to remove from the list
- Click the Delete button to remove the header
See Also
Configuring the Crawler
Working with local files
- Extracting inline data
- Remapping extensions
- Remapping local files
- Updating local time stamps
- Using query string parameters in local filenames
Controlling the crawl
- Content types
- Crawling multiple URLs
- Crawling outside the base URL
- Downloading all resources
- Including additional domains
- Including sub and sibling domains
- Limiting downloads by file count
- Limiting downloads by size
- Limiting scans by depth
- Limiting scans by distance
- Scanning data attributes
- Setting speed limits
- Working with Rules
JavaScript
Security
- Crawling private areas
- Manually logging into a website
- TLS/SSL certificate options
- Working with Forms
- Working with Passwords
Modifying URLs
Creating a site map
Advanced
- Aborting the crawl using HTTP status codes
- Cookies
- HEAD vs GET for preliminary requests
- HTTP Compression
- Origin reports
- Redirects
- Saving link data in a Crawler Project
- Setting the web page language
- Specifying a User Agent
- Specifying accepted content types
- Using Keep-Alive