Web Site Crawler

Crawl a web site and store information about what was found

Search Other Modules

Rank

Normal

Authors

hdm < hdm [at] metasploit.com >

Development

Similar Modules

Usage Information

$ msfconsole

                ##                          ###           ##    ##
##  ##  #### ###### ####  #####   #####    ##    ####        ######
####### ##  ##  ##  ##         ## ##  ##    ##   ##  ##   ###   ##
####### ######  ##  #####   ####  ##  ##    ##   ##  ##   ##    ##
## # ##     ##  ##  ##  ## ##      #####    ##   ##  ##   ##    ##
##   ##  #### ###   #####   #####     ##   ####   ####   #### ###
                                      ##

msf > use auxiliary/scanner/http/crawler
msf auxiliary(crawler) > set RHOST [TARGET IP]
msf auxiliary(crawler) > run

Module Options

MAX_MINUTES	The maximum number of minutes to spend on each URL (default: 5)
MAX_PAGES	The maximum number of pages to crawl per URL (default: 500)
MAX_THREADS	The maximum number of concurrent requests (default: 4)
Proxies	Use a proxy chain
RHOST	The target address
RPORT	The target port (default: 80)
URI	The starting page to crawl (default: /)
VHOST	HTTP server virtual host
BasicAuthPass	The HTTP password to specify for basic authentication
BasicAuthUser	The HTTP username to specify for basic authentication
HTTPAdditionalHeaders	A list of additional headers to send (separated by \x01)
HTTPCookie	A HTTP cookie header to send with each request
RedirectLimit	The maximum number of redirects for a single request
RequestTimeout	The maximum number of seconds to wait for a reply
RetryLimit	The maximum number of attempts for a single request
SSL	Negotiate SSL for outgoing connections
SSLVersion	Specify the version of SSL that should be used (accepted: SSL2, SSL23, SSL3, TLS1)
UserAgent	The User-Agent header to use for all requests
VERBOSE	Enable detailed status messages
WORKSPACE	Specify the workspace for this module