Adding a new rule via the Rules Editor

  1. Select Rules Rules from the Project menu, or press Control+R to display the rule editor.
  2. Click the Add button to add a blank rule
  3. Enter a search expression into the Expression field
  4. Choose what components the search expression will be matched against in the Compare with field
  5. Select any other options to control how the rule behaves
  6. Click the OK button to save the new rule and close the editor


If your expression includes any of the ^, [, ., $, {, *, (, \, +, ), |, ?, <, > characters and you want them to processed as plain text, you need to "escape" the character by preceding it with a backslash. For example, if your expression was application/epub+zip this would need to be written as application/epub\+zip otherwise the + character would have a special meaning and no matches would be made. Similarly, if the expression was, this should be written as example\.com, as . means "any character" which could lead to unexpected matches.

Compare Options

This table outlines the different compare options available. The example match is based on the following sample address

AuthorityThe URL
Authority, Path, and Query StringThe domain, path and query string of the
Content TypeThe detected content type of the URLtext/html
Entire URLThe complete URL
PathThe path of the URL, including file names if applicablefolder/products
Path and Query StringThe path and query string of the URLfolder/products?sort=name&order=asc
Query StringThe query string of the URLsort=name&order=asc
MatchesSpecifies the rule will be processed if the given input matches the rule expression
Does Not MatchSpecifies the rule will be processed if the given input does not match the rule expression

Rule Options

Enable this ruleSpecifies if the rule is enabled or not. Disabled rules will be ignored
ExcludeSpecifies that the URL should not be downloaded
Exclude, but scan for additional contentSpecifies that although the URL is excluded and will not be permanently saved, its contents should still be scanned (applies to HTML documents only). This means that although a permanent copy of the URL is not downloaded, a temporary copy is still made in order to scan for additional URLs to crawl.
DownloadSpecifies that the URL should be downloaded. This allows you to have a wider rule to exclude content, and then a narrower rule to include specific content.
Download, but do not scan for additional contentSpecifies that although the URL is downloaded, its contents should not be scanned. This means that while a permanent copy of the URL is created, it will not be scanned for additional URLs to crawl. This option applies to HTML documents only
Stop processing more rulesBy default all rules are processed sequentially. You can use this flag to control this process; if set and the rule is matched, no further rules will be processed
Download PriorityAllows the download priority for URLs matching the rule to be changed. High priority will mean the URL will be downloaded immediately, while Low means the URL will be downloaded when all other URLs have been processed1.

1 The Download Priority options is only supported for rules that match against a URL, it is ignored for rules matching against content types.

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.9 (buildref #182.15707), last modified 2024-03-15. Generated 2024-03-15 22:36 using Cyotek HelpWrite Professional version 6.19.1