Content Discovery: A Complete Guide to the Essential Tools
Content Discovery: A Complete Guide to the Essential Tools
Content discovery is the process of finding hidden files, directories, API endpoints, and other resources on a web server. It is a critical phase in penetration testing and bug bounty hunting, often revealing sensitive files, backup archives, admin panels, and undocumented APIs that expand the attack surface. This guide covers fifteen powerful tools for directory brute‑forcing, web crawling, API endpoint discovery, and internet‑wide asset discovery.
⚠️ Legal and ethical notice: These tools send HTTP requests and sometimes brute‑force paths on target servers. Only use them against systems you own or have explicit written authorization to test. Unauthorized scanning and fuzzing may violate laws and terms of service.
Table of Contents
- Why Content Discovery Matters
- Tool Overview
- Directory & File Brute‑Forcing
- gobuster
- feroxbuster
- dirsearch (Python)
- dirsearch (Go)
- filebuster
- dirstalk
- dirbuster-ng
- recursebuster
- Web Crawling & Spidering
- gospider
- hakrawler
- crawley
- katana
- API Endpoint Discovery
- kiterunner
- vaf
- Internet‑Wide Discovery
- uncover
- Comparison Table
- Best Practices & Workflow
- Final Tips
Why Content Discovery Matters
A web server may expose far more than the links on its homepage. Common findings include:
- Backup files (
.bak,.old,.zip,.tar.gz) - Configuration files (
.env,config.php,web.config) - Admin panels (
/admin,/wp-admin,/phpmyadmin) - API endpoints (
/api/v1/users,/graphql) - Source code repositories (
.git,.svn) - Sensitive documents (PDFs, spreadsheets, internal notes)
Automated content discovery tools use wordlists and crawling to find these resources efficiently. The right tool depends on the task: brute‑forcing for known paths, crawling for linked resources, and API‑specific tools for modern applications.
Tool Overview
| Tool | Language | Primary Purpose | Best For |
|---|---|---|---|
| gobuster | Go | Directory/DNS/VHost busting | Fast, multi‑purpose brute‑forcing |
| feroxbuster | Rust | Recursive content discovery | Deep, recursive scans |
| dirsearch (Python) | Python | Web path scanner | Classic, feature‑rich scanning |
| dirsearch (Go) | Go | Go implementation of dirsearch | Fast, concurrent scanning |
| filebuster | Perl | Web fuzzer | Flexible, high‑performance fuzzing |
| dirstalk | Go | Modern directory brute‑forcer | Multi‑threaded, easy to use |
| dirbuster-ng | C | CLI dirbuster | Lightweight, C‑based scanning |
| recursebuster | Go | Recursive content discovery | Rapid recursive querying |
| gospider | Go | Web spider | Fast crawling and endpoint discovery |
| hakrawler | Go | Web crawler | Simple, pipeline‑friendly crawling |
| crawley | Go | Unix‑way web scraper | Feature‑rich crawling |
| katana | Go | Next‑gen crawling framework | Advanced crawling with headless support |
| kiterunner | Go | API endpoint bruteforcer | API‑specific content discovery |
| vaf | Nim | Advanced web fuzzer | High‑performance fuzzing |
| uncover | Go | Internet‑wide discovery | Finding exposed hosts via APIs |
Directory & File Brute‑Forcing
These tools test a list of paths (from a wordlist) against a target to find existing files and directories. They are the workhorses of content discovery.
gobuster
Directory/File, DNS and VHost busting tool written in Go.
Gobuster is fast, multi‑threaded, and supports three modes: dir (directory/file), dns (subdomain), and vhost (virtual host).
Installation:
go install github.com/OJ/gobuster/v3@latest# orsudo apt install gobuster
Basic Usage:
# Directory brute‑forcegobuster dir -u https://example.com -w /usr/share/wordlists/dirb/common.txt# With extensionsgobuster dir -u https://example.com -w wordlist.txt -x php,html,txt# DNS subdomain brute‑forcegobuster dns -d example.com -w subdomains.txt# VHost discoverygobuster vhost -u https://example.com -w vhosts.txt
Key Features:
- Three modes: dir, dns, vhost
- Multi‑threaded for speed
- Supports custom extensions, status code filtering, and proxies
- Progress display and result saving
When to use: Gobuster is the go‑to tool for quick, reliable directory and DNS brute‑forcing. It is fast and easy to integrate into scripts.
feroxbuster
A fast, simple, recursive content discovery tool written in Rust.
Feroxbuster automatically recurses into discovered directories, making it excellent for deep scans.
Installation:
# Using cargocargo install feroxbuster# Kali Linuxsudo apt install feroxbuster
Basic Usage:
# Basic scanferoxbuster -u https://example.com -w wordlist.txt# Recursive with depth limitferoxbuster -u https://example.com -w wordlist.txt -d 3# With extensions and filteringferoxbuster -u https://example.com -w wordlist.txt -x php,html -C 404
Key Features:
- Recursive by default
- Fast Rust implementation
- Automatic filtering of wildcard responses
- Supports multiple wordlists, proxies, and headers
- JSON output
When to use: Use feroxbuster when you need thorough recursive discovery without manually chaining scans.
dirsearch (Python)
Web path scanner.
Dirsearch is a classic Python tool with a rich feature set and a large user base.
Installation:
git clone https://github.com/maurosoria/dirsearch.gitcd dirsearchpip install -r requirements.txt
Basic Usage:
# Basic scanpython dirsearch.py -u https://example.com -w wordlist.txt# With extensions and threadspython dirsearch.py -u https://example.com -e php,html -t 50# Recursive scanpython dirsearch.py -u https://example.com -r
Key Features:
- Extensive configuration options
- Supports HTTP/HTTPS, proxies, cookies, headers
- Recursive scanning
- Multiple report formats (JSON, XML, HTML)
When to use: Dirsearch is a solid choice when you need a mature, feature‑rich scanner with a familiar interface.
dirsearch (Go)
A Go implementation of dirsearch.
This is a fast, concurrent Go port of the original dirsearch.
Installation:
go install github.com/evilsocket/dirsearch@latest
Basic Usage:
dirsearch -u https://example.com -w wordlist.txtdirsearch -u https://example.com -w wordlist.txt -e php,html -t 100
Key Features:
- Fast, concurrent scanning
- Similar feature set to the Python version
- Single binary, no dependencies
When to use: Use the Go version when speed and portability are priorities.
filebuster
An extremely fast and flexible web fuzzer.
Filebuster is a Perl‑based fuzzer known for its speed and flexibility.
Installation:
git clone https://github.com/henshin/filebuster.gitcd filebustercpan install LWP::UserAgent
Basic Usage:
./filebuster.pl -u https://example.com -w wordlist.txt -e php,html
Key Features:
- Very fast
- Flexible request customization
- Supports multiple wordlists and extensions
- Low resource usage
When to use: Use filebuster when raw speed is critical and you are comfortable with Perl.
dirstalk
Modern alternative to dirbuster/dirb.
Dirstalk is a Go‑based directory brute‑forcer with a modern interface.
Installation:
go install github.com/stefanoj3/dirstalk@latest
Basic Usage:
dirstalk scan https://example.com --dictionary wordlist.txtdirstalk scan https://example.com --dictionary wordlist.txt --threads 50
Key Features:
- Multi‑threaded
- Supports custom headers, cookies, proxies
- Output in JSON and text
- Easy to use
When to use: Dirstalk is a good alternative to gobuster with a slightly different feature set.
dirbuster-ng
dirbuster-ng is a C CLI implementation of the Java dirbuster tool.
This is a lightweight, C‑based reimplementation of the classic DirBuster.
Installation:
git clone https://github.com/gruntjs/dirbuster-ngcd dirbuster-ngmake
Basic Usage:
./dirbuster-ng -u https://example.com -w wordlist.txt
Key Features:
- Lightweight C binary
- Fast
- Basic directory brute‑forcing
When to use: Use dirbuster-ng when you need a minimal, dependency‑free tool for simple scans.
recursebuster
Rapid content discovery tool for recursively querying webservers.
Recursebuster is designed for fast, recursive discovery.
Installation:
go install github.com/C-Sto/recursebuster@latest
Basic Usage:
recursebuster -u https://example.com -w wordlist.txt
Key Features:
- Recursive by design
- Fast
- Handles large scans efficiently
When to use: Use recursebuster when you want a dedicated recursive scanner without the overhead of a larger tool.
Web Crawling & Spidering
Crawlers automatically discover links and endpoints by following links from a starting URL. They are essential for finding content that isn't easily brute‑forced.
gospider
Fast web spider written in Go.
Gospider crawls websites and extracts URLs, JavaScript files, forms, and more.
Installation:
go install github.com/jaeles-project/gospider@latest
Basic Usage:
gospider -s https://example.com -o output_dirgospider -s https://example.com -d 3 -c 10
Key Features:
- Fast, concurrent crawling
- Extracts URLs, JS files, forms, and secrets
- Supports depth control and concurrency
- Outputs to files
When to use: Use gospider for quick, thorough crawling of a target to discover linked resources.
hakrawler
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets.
Hakrawler is pipeline‑friendly and outputs clean lists of URLs.
Installation:
go install github.com/hakluke/hakrawler@latest
Basic Usage:
echo https://example.com | hakrawlerhakrawler -url https://example.com -depth 3 -plain
Key Features:
- Simple, Unix‑philosophy design
- Outputs one URL per line
- Supports depth and concurrency
- Extracts JavaScript files and forms
When to use: Use hakrawler when you need a lightweight crawler that integrates easily with other tools via pipes.
crawley
Fast, feature‑rich Unix‑way web scraper/crawler written in Golang.
Crawley is a versatile crawler with many options.
Installation:
go install github.com/s0rg/crawley@latest
Basic Usage:
crawley -u https://example.com -d 3
Key Features:
- Rich feature set
- Supports depth, concurrency, and custom headers
- Outputs in various formats
- Unix‑friendly
When to use: Use crawley when you need a feature‑rich crawler with extensive configuration options.
katana
A next‑generation crawling and spidering framework.
Katana is a powerful crawler from ProjectDiscovery that supports headless browsing, JavaScript parsing, and advanced filtering.
Installation:
go install github.com/projectdiscovery/katana/cmd/katana@latest
Basic Usage:
katana -u https://example.com -o urls.txtkatana -u https://example.com -d 3 -jc -o urls.txt
Key Features:
- Headless browser support (
-headless) - JavaScript crawling (
-jc) - Advanced filtering and scope control
- JSON output
- Integrates with other ProjectDiscovery tools
When to use: Use katana for modern web applications that rely heavily on JavaScript. It is the most advanced crawler in this list.
API Endpoint Discovery
Modern applications expose APIs, often with undocumented endpoints. These tools specialize in finding API routes.
kiterunner
Fast API endpoint bruteforcer and content discovery tool for modern web applications.
Kiterunner uses a custom wordlist of API routes and can detect the API framework.
Installation:
go install github.com/assetnote/kiterunner@latest
Basic Usage:
kr scan https://example.com -w routes.kitekr brute https://example.com -w wordlist.txt
Key Features:
- API‑specific wordlists
- Framework detection
- Fast, concurrent scanning
- Supports custom headers and authentication
When to use: Use kiterunner when targeting REST or GraphQL APIs. It is far more effective than generic directory brute‑forcing for API discovery.
vaf
Vaf is a cross‑platform very advanced and fast web fuzzer written in Nim.
Vaf is a high‑performance fuzzer with advanced features.
Installation:
git clone https://github.com/karimhabush/vafcd vafnim c -d:release vaf.nim
Basic Usage:
./vaf -u https://example.com -w wordlist.txt
Key Features:
- Extremely fast (Nim compiled)
- Advanced fuzzing options
- Supports multiple protocols
- Low resource usage
When to use: Use vaf when you need maximum speed and are comfortable compiling Nim.
Internet‑Wide Discovery
uncover
Uncover is a Go wrapper using APIs of well‑known search engines to quickly discover exposed hosts on the internet.
Uncover queries Shodan, Censys, FOFA, and other search engines to find hosts matching a query.
Installation:
go install github.com/projectdiscovery/uncover/cmd/uncover@latest
Basic Usage:
uncover -q "apache" -o results.txtuncover -q "country:US port:443" -e shodan
Key Features:
- Queries multiple search engines
- Supports Shodan, Censys, FOFA, Hunter, etc.
- Outputs IPs and ports
- Integrates with other tools
When to use: Use uncover for internet‑wide asset discovery based on technology, port, or country.
Comparison Table
| Tool | Type | Language | Recursive | API Focus | Speed |
|---|---|---|---|---|---|
| gobuster | Dir/DNS/VHost | Go | No | No | Fast |
| feroxbuster | Dir/File | Rust | Yes | No | Very fast |
| dirsearch (Python) | Dir/File | Python | Yes | No | Medium |
| dirsearch (Go) | Dir/File | Go | Yes | No | Fast |
| filebuster | Fuzzer | Perl | No | No | Very fast |
| dirstalk | Dir/File | Go | No | No | Fast |
| dirbuster-ng | Dir/File | C | No | No | Fast |
| recursebuster | Dir/File | Go | Yes | No | Fast |
| gospider | Crawler | Go | Yes | No | Fast |
| hakrawler | Crawler | Go | Yes | No | Fast |
| crawley | Crawler | Go | Yes | No | Fast |
| katana | Crawler | Go | Yes | No | Fast |
| kiterunner | API | Go | No | Yes | Fast |
| vaf | Fuzzer | Nim | No | No | Very fast |
| uncover | Discovery | Go | No | No | API‑dependent |
Best Practices & Workflow
- Start with crawling. Use katana or gospider to find linked resources.
- Follow with directory brute‑forcing. Use feroxbuster or gobuster to find unlinked files.
- Target APIs separately. Use kiterunner for API endpoint discovery.
- Use appropriate wordlists. SecLists provides excellent wordlists for directories, files, and APIs.
- Filter noise. Exclude common false positives (e.g., 404 pages, wildcard responses) with status code and size filters.
- Respect rate limits. Use concurrency controls to avoid overwhelming targets.
- Combine results. Merge findings from multiple tools for comprehensive coverage.
- Automate. Chain tools in scripts for repeatable workflows.
Example Workflow
# 1. Crawl for linked resourceskatana -u https://example.com -d 3 -jc -o crawled.txt# 2. Directory brute‑force with feroxbusterferoxbuster -u https://example.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -o dirs.txt# 3. API endpoint discovery with kiterunnerkr scan https://example.com -w routes.kite -o api.txt# 4. Merge and deduplicatecat crawled.txt dirs.txt api.txt | sort -u > all_endpoints.txt# 5. Probe for live endpointscat all_endpoints.txt | httpx -silent -o live.txt
Final Tips
- Wordlists are key. A good wordlist (e.g., SecLists) dramatically improves results.
- Extensions matter. Add
-x php,html,js,jsonto find more files. - Recursion is powerful. Tools like feroxbuster and recursebuster automatically explore subdirectories.
- API discovery is different. Use kiterunner or vaf for API‑heavy applications.
- Crawl before you brute‑force. Crawling often finds endpoints that wordlists miss.
- Stay authorized. Always obtain written permission before scanning any system.
- Keep tools updated. The content discovery landscape evolves quickly.
Official Resources:
- SecLists: github.com/danielmiessler/SecLists
- gobuster: github.com/OJ/gobuster
- feroxbuster: github.com/epi052/feroxbuster
- dirsearch: github.com/maurosoria/dirsearch
- katana: github.com/projectdiscovery/katana
- kiterunner: github.com/assetnote/kiterunner
- uncover: github.com/projectdiscovery/uncover
Remember: Content discovery is a powerful reconnaissance technique. Use it responsibly, stay within scope, and always obtain proper authorization before testing any system.
Author
Discussion · 0
No comments yet — start the discussion.
Sign in to join the discussion.