While forms can have myriad purposes, in terms of crawling a website using WebCopy they are normally used to authenticate a user.

WebCopy is able to post forms to websites. As well as specifying values for user supplied parameters, the crawl engine will automatically include other parameters present on the original form, such as session tokens or anti-forgery tokens, without you needing to define these yourself.

Forms can be manually defined, or automatically captured. Once you have defined a form, you can test that it works correctly before initiating a website copy.

Important

WebCopy is unable to automatically log into web sites using two-factor authentication (2FA), multi-factor authentication (MFA), or that require client side JavaScript to be executed as part of the login process. You can use External Authentication or cookies to manually log into a target website prior to copying.

Important

If you use Forms, Passwords or Cookies to authenticate with a website, you should consider adding a custom rule to exclude any logout pages. Otherwise, if WebCopy detects this page, eventually it will access it and your session will be logged out, potentially affecting the remainder of the crawl.

Click the links below to learn more about working with forms.

© 2010-2024 Cyotek Ltd. All Rights Reserved.
Documentation version 1.10 (buildref #186.15944), last modified 2024-08-18. Generated 2024-08-18 08:00 using Cyotek HelpWrite Professional version 6.20.0