Web Information Gathering
This cheatsheet provides a comprehensive guide to web information gathering.
- Published on
- Published on
Passive Enumeration Phase
Passive reconnaissance focuses on gathering intelligence about a target’s digital footprint using third-party services and public records without direct interaction with the target’s infrastructure.
Domain and Infrastructure Information
Identifying domain ownership and mapping public netblocks provides potential entry points and technical contacts.
| Command/Link | Description |
|---|---|
whois target.com | Standard query to identify domain owners, registration dates, and assigned name servers. |
whois target.com -h 192.168.50.251 | Queries a specific WHOIS server for registration records. |
| WhoisFreaks | Web service used to access historical WHOIS records and track ownership changes over time. |
| Netcraft | Web interface used to gather historical server and hosting information. |
Subdomain Intelligence (OSINT)
Certificate Transparency (CT) logs and DNS repositories reveal subdomains, such as development or staging environments, that are not publicly linked.
| Command/Link | Description |
|---|---|
| crt.sh | Web-based search for SSL/TLS certificates recorded in public, append-only ledgers. |
curl -s "https://crt.sh/?q=target.com&output=json" | jq -r '.[].name_value' | sort -u | CLI command to extract all unique subdomains for a domain from CT logs. |
curl -s "..." | jq -r '.[] | select(.name_value | contains("dev")) | .name_value' | CLI filter to specifically find subdomains containing the string “dev” in CT logs. |
| Censys | Web search engine for hostnames and globally observed certificates. |
| DNSDumpster | Web service mapping domains and visualizing host relationships using open-source repositories. |
| Farsight DNSDB | Passive DNS data service collected from a global sensor array. |
Google Hacking (Dorking)
Advanced search operators are used to find information leakage, specific file types, and exposed directory listings.
| Command/Link | Description |
|---|---|
site:target.com -www | Google Dork to find subdomains while excluding the main “www” site. |
filetype:pdf site:target.com | Searches for public PDF documents belonging to the target organization. |
ext:php or ext:aspx | Finds pages using specific web technologies or file extensions. |
intitle:"index of" "parent directory" | Discovers exposed directory listings on web servers. |
| Google Hacking DB | Extensive database of advanced search strings for reconnaissance (GHDB). |
Technology Profiling & Web Archives
Analyze the technology stack and content history without sending packets directly to the target server.
| Command/Link | Description |
|---|---|
whatweb http://target.com:54387 | CLI tool to passively identify technologies, versions, and frameworks. |
| Wappalyzer | Browser extension identifying frameworks, CMS, and server-side technologies. |
| BuiltWith | Web-based service that profiles a site’s entire technology stack. |
| Wayback Machine | Accesses historical versions of pages to find deleted content or old parameters. |
| Netcraft Fingerprinting | Web-based tool for fingerprinting web servers passively. |
Credential and Email Gathering
Scrape public data and organizational patterns to prepare for later authentication or social engineering tests.
| Command/Link | Description |
|---|---|
| theHarvester | OSINT tool gathering emails, employee names, and subdomains from multiple sources. |
./breach-parse.sh target.com out.txt | CLI tool to scrape local data breach repositories for credentials associated with a domain. |
| hunter.io | Web service for discovering organizational email patterns and addresses. |
| phonebook.cz | Lists emails and subdomains associated with a specific domain. |
| OSINT Framework | A web directory of various OSINT tools and resources. |
Active Enumeration Phase
Active enumeration requires direct interaction with the target’s services to verify configurations, map paths, and identify vulnerabilities.
DNS Probing & Resolution
Manual DNS queries target authoritative nameservers to resolve specific record types and test for misconfigurations.
| Command/Link | Description |
|---|---|
dig target.com ANY | Attempts to retrieve all available DNS records for the domain. |
dig target.com A / MX / NS / TXT | Retrieves specific DNS record types (IPv4, Mail, Nameserver, or Text). |
dig @1.1.1.1 target.com | Directs a query to a specific DNS server. |
dig +trace target.com | Shows the full path of DNS resolution from root servers. |
dig -x 192.168.1.1 | Performs a PTR lookup to find the hostname associated with an IP. |
dig @10.42.42.42 version.bind chaos txt | Queries a nameserver for its specific BIND software version. |
host www.target.com | Streamlined CLI check for the A record of a hostname. |
host -t mx target.com | Concise check for specific record types like mail exchange. |
for ip in $(cat list.txt); do host $ip.domain; done | Bash loop to resolve multiple hostnames from a wordlist. |
nslookup -type=TXT <host> <NS_IP> | Resolves specific records; commonly used to confirm blind injection. |
DNS Brute Forcing & Zone Transfers
Automated tools perform dictionary attacks to find hidden subdomains and attempt zone transfers.
| Command/Link | Description |
|---|---|
dig axfr @10.129.26.159 target.com | Attempts a full DNS zone transfer from the specified nameserver. |
dnsenum --enum target.com -f wordlist.txt -r | Automated enumeration with recursive subdomain brute forcing. |
dnsrecon -d target.com -t std | Performs a standard enumeration scan for common DNS records. |
dnsrecon -d target.com -D list.txt -t brt | Brute forces subdomains using a provided dictionary. |
dnsrecon -r 192.168.1.0/24 | Performs reverse DNS (PTR) lookups on an IP range. |
dnsrecon -t crt -d sec542.com | CLI tool searching for domains in CT logs via crt.sh. |
nmap --script dns-brute target.com | Uses an NSE script to brute force subdomains via an internal wordlist. |
sublist3r -d target.com -t 100 | Aggregates subdomains from many public sources. |
Host & Service Discovery
Scanning open ports is essential for determining listening services and software versions to identify exploits.
| Command/Link | Description |
|---|---|
nmap -sV target.com | Identifies open ports and probes them for service version detection. |
nmap -A -p80,443,53 target.com | Aggressive scan including OS detection, versioning, scripts, and traceroute. |
nmap -p 443 --script ssl-enum-ciphers <IP> | Enumerates and rates supported SSL/TLS ciphers for security. |
nmap --script http-robots.txt www.target.com | Automatically parses and displays entries in the robots.txt file. |
nc -vz 127.0.0.1 80-1023 | Uses Netcat as a simple, fast port scanner. |
| Shodan | Search engine that facilitates port scanning and device discovery without sending packets. |
gowitness can single -u <URL> --write-db | Captures screenshots of discovered web interfaces for visual profiling. |
Fingerprinting & Application Probing
Direct interaction reveals server software, supported methods, and presence of security controls.
| Command/Link | Description |
|---|---|
curl -I target.com | Banner grabbing: Reads HTTP headers for server/platform info. |
echo -e "GET / HTTP/1.0\r\n" | nc target 80 | Manual Netcat connection to send raw requests and observe responses. |
curl -i -X OPTIONS https://target.com | Probes for allowed HTTP methods via the OPTIONS verb. |
wafw00f target.com | CLI tool to actively probe and identify a Web Application Firewall. |
nikto -h http://192.168.1.23 -Tuning b | Scans for vulnerabilities and identifies software modules. |
| Retire.js | Scans applications for vulnerable or outdated JavaScript libraries. |
Content & Virtual Host Discovery
Fuzzing the Host header identifies hidden virtual hosts, while forced browsing finds unlinked content.
| Command/Link | Description |
|---|---|
ffuf -w list.txt -u http://target.com/FUZZ | Basic directory and file discovery using the FUZZ keyword. |
ffuf -w list.txt -u <URL>/FUZZ -e .php,.txt | Fuzzes for files with specific extensions. |
ffuf -u http://IP/ -H "Host: FUZZ.target.com" -w list.txt | Identifies hidden virtual hosts sharing a single IP. |
ffuf -w list.txt -u <URL>/FUZZ -mc 200 -fs 9 | Matches 200 OK responses and filters out specific response sizes. |
gobuster vhost -u http://IP -w list.txt --append-domain | Identifies virtual hosts by fuzzing the Host header. |
wget -r -e robots=off https://target.com -P /tmp | Recursively spiders a site while ignoring robots.txt instructions. |
cewl https://www.target.com -w wordlist.txt | Spiders a site to create a custom contextual wordlist based on its text. |
| SecLists | Comprehensive collection of high-quality wordlists for fuzzing and discovery. |
API & Web Service Discovery
Modern applications use explicit routes and HTTP methods that require specialized discovery tools.
| Command/Link | Description |
|---|---|
| Kiterunner | Brute forces API endpoints using specialized .kite wordlists. |
| Assetnote Lists | High-quality wordlists built from internet-wide scans for APIs and tech stacks. |
| Postman | Popular platform for building and testing APIs. |
| Bruno | Open-source API client used for testing and seeding requests to proxies. |
Automated Vulnerability Scanning
Template-based and integrated scanners probe for specific misconfigurations and exposed secrets.
| Command/Link | Description |
|---|---|
nuclei -u http://target.com | CLI tool running default vulnerability templates against a target. |
nuclei -u <URL> -t exposures/configs | Limits scanning to specific templates, such as exposed configuration files. |
nuclei -u <URL> -H 'Cookie: JESSIONID=...' | Supplies authentication headers for scanning protected resources. |
| Burp Scanner | Professional version feature performing automated auditing for vulnerabilities. |
| ZAP Active Scan | Integrated scanner in ZAP that actively searches for vulnerabilities. |
Database Metadata Enumeration
If a SQL injection vulnerability is found, automated tools are used to exfiltrate the database structure.
| Command/Link | Description |
|---|---|
sqlmap -u "URL?id=1" --banner | Fingerprints the DBMS and retrieves the version banner. |
sqlmap -u URL --data "id=1" --batch --dbs | CLI tool that enumerates all databases on the backend server. |
sqlmap -u URL -D DB_NAME --tables | Lists all tables within a specific database. |
sqlmap -u URL -D DB_NAME -T TABLE_NAME --dump | Extracts and saves all data from a specific table. |
sqlmap -u URL --file-read=/etc/passwd | Attempts to read local files from the server’s filesystem. |
sqlmap -u URL --os-shell | Attempts to gain an interactive operating system shell. |
sqlmap -u URL -v 6 | Maximum verbosity: shows full HTTP requests and response content. |
Key Files and URI Patterns
Reviewing metafiles and predictable URI patterns provides insight into server management and potential attack vectors.
| File / URI Pattern | Description |
|---|---|
/robots.txt | Lists paths disallowed for crawlers; often reveals sensitive directories like /admin. |
/.well-known/security.txt | Standardized location for security contact information and reporting policies. |
/.well-known/openid-configuration | Discloses OpenID identity provider endpoints and metadata. |
/server-status | Apache status page; leaks server version, uptime, and current active requests. |
/openapi.json | Retrieves OpenAPI/Swagger definitions describing API routes and methods. |
http://169.254.169.254/ | Target URI for AWS EC2 instance metadata (IMDSv1) during SSRF testing. |
http://metadata.google.internal/ | Target URI for Google Cloud instance metadata. |
https://kubernetes.default | Probes for Kubernetes pod information and internal debug services. |
.bak, .old, .tmp, .conf | Common extensions targeted during forced browsing to find exposed sensitive data. |
Disclaimer
This cheatsheet does not ensure that commands and tools are accurate. Make sure to review results and use at your own risk.
Perform pentest with proper authorization only!