Cyotek Sitemap Creator HelpIf set the crawl address for a project to be a nested URL such as https://demo.cyotek.com/features, by default Sitemap Creator will not crawl above this URL. For example, it would process /features/cdn.php, but would skip /html/elements/base.php. In some circumstances it may be desirable to allow this without changing the root URL to be at a higher level.
To enable or disable crawling above the root URL
- From the Project Properties dialogue, select the Advanced category
- Check or uncheck the Crawl above the root URL option
See Also
Configuring the Crawler
Working with local files
- Extracting inline data
- Remapping extensions
- Remapping local files
- Updating local time stamps
- Using query string parameters in local filenames
Controlling the crawl
- Content types
- Crawling additional root URLs
- Including additional domains
- Including sub and sibling domains
- Limiting downloads by file count
- Limiting downloads by size
- Limiting scans by depth
- Limiting scans by distance
- Scanning data attributes
- Setting speed limits
- Working with Rules
JavaScript
Security
- Crawling private areas
- Manually logging into a website
- TLS/SSL certificate options
- Working with Forms
- Working with Passwords
Modifying URLs
Advanced
- Aborting the crawl using HTTP status codes
- Cookies
- Defining custom headers
- Following redirects
- HEAD vs GET for preliminary requests
- HTTP Compression
- Modifying page titles
- Origin reports
- Overwriting read only files
- Saving link data in a Crawler Project
- Setting the web page language
- Specifying a User Agent
- Specifying accepted content types
- Using Keep-Alive