If there are sections of your site which do not have any inbound links, but which you still wish crawled, you can specify additional seed URLs for the crawl process. The additional list of URLs can also be read from an external file.

To set up multiple URLs for crawling

  • From the Project Properties dialogue, select the Additional URLs category

Defining the URL list within the WebCopy project

  1. Select the Load additional URLs from list option
  2. Enter the URLs to process, one per line

Tip

Blank lines, or lines beginning with # or ; will be ignored

Storing the URL list within an external text file

  1. Select the Load additional URLs from file option
  2. Enter or select the name of the external file name

Note

The filename must be full qualified

Tip

Blank lines in the external file, or lines beginning with # or ; will be ignored

See Also

Configuring the Crawler

Working with local files

Controlling the crawl

JavaScript

Security

Modifying URLs

Creating a site map

Advanced

Deprecated features

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.9 (buildref #182.15707), last modified 2024-03-15. Generated 2024-03-15 22:36 using Cyotek HelpWrite Professional version 6.19.1