You can configure your project to automatically apply one or more cookies before crawling commences. This can be used to provide authentication cookies in the event that the built in authentication features of WebCopy are not sufficient.

Important

Cookies stored in a WebCopy project must conform to the Set-Cookie header syntax. The minimum required is <cookie-name>=<cookie-value>. <cookie-value> should be URL encoded if appropriate, WebCopy does not perform any automatic encoding.

Important

If you use Forms, Passwords or Cookies to authenticate with a website, you should consider adding a custom rule to exclude any logout pages. Otherwise, if WebCopy detects this page, eventually it will access it and your session will be logged out, potentially affecting the remainder of the crawl.

To customise cookies

  • From the Project Properties dialogue, expand the Advanced category and select the Cookies sub-category
  1. Click the Add button
  2. Enter the the cookie data into the Data field
  1. Select one or more cookies that you wish to remove
  2. Click the Delete button
  1. Select the cookie to edit from the list. The Data field will be updated to contain the value of the cookie
  2. Enter new value for the cookie data

Reading cookies from an external file

To read cookies from an external file, enter the file name in the Read Cookies From field, or click the Browse button to select a file.

Important

Only cookies in the Netscape cookie file format are supported.

Discarding session cookies

To discard any session cookies from the external file, ensure the Discard session cookies option is checked. When set, any cookies without an expiry date will be skipped.

Writing cookies to an external file

Once a copy operation has complete, WebCopy can optionally write all cookies into a file, using the Netscape cookie format. To write cookies to a file, enter the file name in the Write Cookies To field, or click the Browse button select a file.

Important

Cookies are only written when performing a copy, not when performing a read-only scan.

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.10 (buildref #186.15944), last modified 2024-08-18. Generated 2024-08-18 08:00 using Cyotek HelpWrite Professional version 6.20.0