Links: A Complete Guide to URL, Endpoint, and JavaScript Link Discovery Tools
Links: A Complete Guide to URL, Endpoint, and JavaScript Link Discovery Tools
Link discovery is a cornerstone of modern web reconnaissance. JavaScript files, archived URLs, and hidden endpoints often contain the keys to understanding an application's attack surface. Whether you are hunting for API routes, secrets, parameters, or forgotten admin panels, the tools in this guide will help you extract, filter, and organize links from a wide variety of sources.
⚠️ Legal and ethical notice: These tools interact with web applications, fetch archived data, and query public APIs. Only use them against systems you own or have explicit written authorization to test. Unauthorized reconnaissance may violate laws and terms of service.
Table of Contents
- Why Link Discovery Matters
- Tool Overview
- JavaScript Endpoint Extraction
- Archive & Historical URL Tools
- General Link Extraction & Crawling
- Advanced Discovery & Wordlist Generation
- GitHub Endpoint Discovery
- Comparison Table
- Best Practices & Workflow
- Final Tips
1. Why Link Discovery Matters
A web application’s visible surface is only a fraction of its true attack surface. JavaScript files often contain hardcoded API endpoints, hidden parameters, and sometimes secrets. Archived URLs from the Wayback Machine or Common Crawl can reveal old, forgotten, or deprecated endpoints that still work. GitHub repositories may leak internal paths. Extracting and analyzing these links helps you:
- Discover undocumented API endpoints
- Find hidden parameters and query strings
- Identify forgotten or staging environments
- Uncover secrets and API keys
- Build a comprehensive target wordlist
The tools below automate this process across JavaScript files, web archives, and code repositories.
2. Tool Overview
| Tool | Language | Primary Purpose | Best For |
|---|---|---|---|
| LinkFinder | Python | Extract endpoints from JS files | Classic JS endpoint discovery |
| JS-Scan | PHP | Scrape URLs and info from JS | PHP-based JS scanning |
| LinksDumper | Python | Extract and filter links from responses | Response link extraction |
| GoLinkFinder | Go | Fast JS endpoint extractor | High-speed JS parsing |
| BurpJSLinkFinder | Java | Burp extension for passive JS scanning | Burp Suite integration |
| urlgrab | Go | Spider a website for links | General web crawling |
| waybackurls | Go | Fetch URLs from Wayback Machine | Historical URL discovery |
| gau | Go | Fetch URLs from OTX, Wayback, Common Crawl | Multi-source URL aggregation |
| getJS | Go | Fast retrieval of JS files | JS file collection |
| linx | Go | Reveal invisible links in JS | Hidden endpoint discovery |
| waymore | Python | Find more from Wayback Machine | Deep archive mining |
| xnLinkFinder | Python | Discover endpoints, params, wordlists | Advanced link & param discovery |
| URLFinder | Go | Passive URL gathering | High-speed asset discovery |
| github-endpoints | Go | Find endpoints via GitHub search | Code repository leaks |
| jsleak | Python | Find secrets, paths, links in JS | Secret and path extraction |
| jsfinder | Python | Find JS file URLs in HTML | JS file enumeration |
| jsluice | Go | Extract URLs, paths, secrets from JS | Context-aware JS analysis |
3. JavaScript Endpoint Extraction
JavaScript files are goldmines for endpoint discovery. These tools parse JS and extract URLs, paths, and parameters.
LinkFinder
A Python script that finds endpoints in JavaScript files.
Installation:
git clone https://github.com/GerbenJavado/LinkFinder.gitcd LinkFinderpip install -r requirements.txt
Basic Usage:
python linkfinder.py -i https://example.com/app.js -o results.htmlpython linkfinder.py -i ./js/ -o cli
Key Features:
- Extracts endpoints from JS files
- Supports single files, directories, and URLs
- Outputs in HTML or CLI format
- Uses regex to find URLs and paths
When to use: Classic choice for extracting endpoints from JavaScript. Works well as a standalone tool or within pipelines.
🔗 GitHub
JS-Scan
A .js scanner, built in PHP, designed to scrape URLs and other info.
Installation:
git clone https://github.com/zseano/JS-Scan.gitcd JS-Scan
Basic Usage:
php js-scan.php -u https://example.com/app.js
Key Features:
- PHP-based, lightweight
- Scrapes URLs and other information from JS
- Simple CLI interface
When to use: When you prefer a PHP-based tool or need a lightweight scanner.
🔗 GitHub
GoLinkFinder
A fast and minimal JS endpoint extractor.
Installation:
go install github.com/0xsha/GoLinkFinder@latest
Basic Usage:
GoLinkFinder -u https://example.com/app.js
Key Features:
- Very fast (Go)
- Minimal dependencies
- Extracts endpoints from JS files
When to use: When speed is critical and you want a simple, focused tool.
🔗 GitHub
BurpJSLinkFinder
Burp Extension for passive scanning of JS files for endpoint links.
Installation: Download the JAR from the releases page and load it in Burp Suite (Extender → Extensions → Add).
Basic Usage: Passive scanning — automatically processes JS files as you browse.
Key Features:
- Integrates directly into Burp Suite
- Passive scanning of JS files
- Displays endpoints in the Burp UI
When to use: When you are already using Burp Suite and want passive JS endpoint discovery.
🔗 GitHub
getJS
A tool to fastly get all JavaScript sources/files.
Installation:
go install github.com/003random/getJS@latest
Basic Usage:
getJS -url https://example.com -output js_files.txt
Key Features:
- Fast retrieval of JS file URLs
- Can resolve and save JS files
- Supports crawling
When to use: To collect all JS files from a target for further analysis.
🔗 GitHub
linx
Reveals invisible links within JavaScript files.
Installation:
go install github.com/riza/linx@latest
Basic Usage:
linx -u https://example.com/app.js
Key Features:
- Finds hidden or obfuscated links in JS
- Fast Go implementation
- Outputs discovered endpoints
When to use: When you suspect hidden endpoints in JS that standard regex might miss.
🔗 GitHub
jsfinder
A tool that scans web pages to find JavaScript file URLs linked in the HTML source code.
Installation:
git clone https://github.com/Threezh1/JSFinder.gitcd JSFinderpip install -r requirements.txt
Basic Usage:
python JSFinder.py -u https://example.com
Key Features:
- Finds JS file URLs in HTML
- Can scan entire sites
- Outputs a list of JS files
When to use: To enumerate all JavaScript files referenced by a web page.
🔗 GitHub
jsluice
Extracts URLs, paths, secrets, and other interesting bits from JavaScript files.
Installation:
go install github.com/BishopFox/jsluice/cmd/jsluice@latest
Basic Usage:
jsluice urls https://example.com/app.jsjsluice secrets https://example.com/app.js
Key Features:
- Context-aware extraction (not just regex)
- Extracts URLs, paths, secrets
- Multiple subcommands for different data types
When to use: When you need intelligent, context-aware JS analysis rather than simple regex.
🔗 GitHub
jsleak
A tool to find secrets, paths, or links in JavaScript files or source code.
Installation:
git clone https://github.com/ChaitanyaHaritash/jsleak.gitcd jsleakpip install -r requirements.txt
Basic Usage:
python jsleak.py -u https://example.com/app.js
Key Features:
- Finds secrets, paths, and links
- Simple Python implementation
- Can scan local files or URLs
When to use: For quick secret and link extraction from JS.
4. Archive & Historical URL Tools
Archived URLs often reveal old endpoints, parameters, and forgotten assets.
waybackurls
Fetch all the URLs that the Wayback Machine knows about for a domain.
Installation:
go install github.com/tomnomnom/waybackurls@latest
Basic Usage:
echo example.com | waybackurls > urls.txt
Key Features:
- Fetches URLs from Wayback Machine
- Fast and simple
- Outputs one URL per line
When to use: To quickly get historical URLs for a domain.
🔗 GitHub
gau
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
Installation:
go install github.com/lc/gau/v2/cmd/gau@latest
Basic Usage:
gau example.comgau --threads 5 example.com > urls.txt
Key Features:
- Aggregates from multiple sources
- Fast and concurrent
- Supports filtering and output options
When to use: When you want a broader set of historical URLs from multiple sources.
🔗 GitHub
waymore
Find way more from the Wayback Machine!
Installation:
git clone https://github.com/xnl-h4ck3r/waymore.gitcd waymorepip install -r requirements.txt
Basic Usage:
python waymore.py -i example.com -o results/
Key Features:
- Deep mining of Wayback Machine
- Downloads and analyzes archived content
- Extracts URLs, parameters, and more
When to use: When you need a thorough, deep archive analysis.
🔗 GitHub
5. General Link Extraction & Crawling
These tools extract links from responses and crawl websites.
LinksDumper
Extract (links/possible endpoints) from responses & filter them via decoding/sorting.
Installation:
git clone https://github.com/0x3curity/LinksDumper.gitcd LinksDumperpip install -r requirements.txt
Basic Usage:
python linksdumper.py -u https://example.com
Key Features:
- Extracts links from HTTP responses
- Filters and decodes links
- Sorts and deduplicates
When to use: When you have raw HTTP responses and need to extract links.
🔗 GitHub
urlgrab
A golang utility to spider through a website searching for additional links.
Installation:
go install github.com/IAmStoxe/urlgrab@latest
Basic Usage:
urlgrab -u https://example.com -d 3
Key Features:
- Spidering with depth control
- Fast Go implementation
- Outputs discovered links
When to use: For general-purpose link crawling.
🔗 GitHub
URLFinder
A high-speed tool for passively gathering URLs, optimized for efficient web asset discovery without active scanning.
Installation:
go install github.com/projectdiscovery/urlfinder/cmd/urlfinder@latest
Basic Usage:
urlfinder -d example.com -o urls.txt
Key Features:
- Passive URL gathering
- High-speed
- Integrates with ProjectDiscovery ecosystem
When to use: For passive, high-speed URL discovery across many domains.
🔗 GitHub
6. Advanced Discovery & Wordlist Generation
xnLinkFinder
A Python tool used to discover endpoints, potential parameters, and a target-specific wordlist for a given target.
Installation:
git clone https://github.com/xnl-h4ck3r/xnLinkFinder.gitcd xnLinkFinderpip install -r requirements.txt
Basic Usage:
python xnLinkFinder.py -i https://example.com -o results/
Key Features:
- Discovers endpoints and parameters
- Generates target-specific wordlists
- Supports deep crawling and JS analysis
When to use: When you want to build a custom wordlist based on a target's own links and parameters.
🔗 GitHub
7. GitHub Endpoint Discovery
github-endpoints
This Go tool performs searches on GitHub and parses the results to find endpoints of a given domain.
Installation:
go install github.com/gwen001/github-endpoints@latest
Basic Usage:
github-endpoints -d example.com -t YOUR_GITHUB_TOKEN
Key Features:
- Searches GitHub for domain references
- Extracts endpoints from code
- Requires GitHub token for API access
When to use: To find endpoints leaked in public GitHub repositories.
🔗 GitHub
8. Comparison Table
| Tool | Source | Language | Output | Best For |
|---|---|---|---|---|
| LinkFinder | JS | Python | HTML/CLI | Classic JS endpoint extraction |
| JS-Scan | JS | PHP | Text | PHP-based JS scanning |
| LinksDumper | Responses | Python | Text | Extracting links from responses |
| GoLinkFinder | JS | Go | Text | Fast JS endpoint extraction |
| BurpJSLinkFinder | JS | Java | Burp UI | Passive JS scanning in Burp |
| urlgrab | Web | Go | Text | General crawling |
| waybackurls | Archive | Go | Text | Quick historical URLs |
| gau | Archive | Go | Text | Multi-source historical URLs |
| getJS | JS | Go | Text | Collecting JS files |
| linx | JS | Go | Text | Hidden links in JS |
| waymore | Archive | Python | Files | Deep archive mining |
| xnLinkFinder | Web/JS | Python | Files | Endpoint & wordlist generation |
| URLFinder | Web | Go | Text | Passive URL gathering |
| github-endpoints | GitHub | Go | Text | Code repository leaks |
| jsleak | JS | Python | Text | Secrets, paths, links |
| jsfinder | HTML | Python | Text | Finding JS file URLs |
| jsluice | JS | Go | Text | Context-aware JS analysis |
9. Best Practices & Workflow
- Start with passive sources. Use
waybackurls,gau, andwaymoreto gather historical URLs. - Collect JavaScript files. Use
getJSandjsfinderto enumerate all JS files. - Extract endpoints from JS. Use
LinkFinder,GoLinkFinder,jsluice, orlinx. - Check for secrets. Use
jsleakorjsluice secretsto find exposed credentials. - Crawl the live site. Use
urlgraborkatanato discover current links. - Check GitHub. Use
github-endpointsto find leaked endpoints in code repositories. - Build a wordlist. Use
xnLinkFinderto generate a target-specific wordlist from discovered links and parameters. - Filter and deduplicate. Combine all results and remove duplicates:
cat wayback.txt gau.txt js_endpoints.txt | sort -u > all_links.txt
- Probe for live endpoints. Pipe the list into
httpxto find live URLs:
cat all_links.txt | httpx -silent -o live.txt
Example Pipeline
# 1. Historical URLswaybackurls example.com > wayback.txtgau example.com > gau.txt# 2. JS filesgetJS -url https://example.com -output js_files.txt# 3. Extract endpoints from JScat js_files.txt | while read url; dopython linkfinder.py -i "$url" -o clidone > js_endpoints.txt# 4. GitHub endpointsgithub-endpoints -d example.com -t $GITHUB_TOKEN > github_endpoints.txt# 5. Combine and deduplicatecat wayback.txt gau.txt js_endpoints.txt github_endpoints.txt | sort -u > all_links.txt# 6. Probe live URLscat all_links.txt | httpx -silent -o live.txt# 7. Generate wordlistpython xnLinkFinder.py -i https://example.com -o wordlist/
10. Final Tips
- Use multiple sources. No single tool finds everything. Combine archives, JS analysis, and code search.
- Filter aggressively. Raw link lists are noisy. Use
grep,sed, andawkto filter for interesting patterns (e.g.,api,admin,token). - Look for parameters. Query strings often reveal hidden parameters that can be fuzzed later.
- Check for secrets. JS files sometimes contain API keys, tokens, and passwords.
- Keep tools updated. The web evolves, and so do these tools.
- Stay authorized. Always obtain written permission before scanning any system.
- Document findings. Keep a record of discovered endpoints for your report.
Official Resources:
- LinkFinder: github.com/GerbenJavado/LinkFinder
- GoLinkFinder: github.com/0xsha/GoLinkFinder
- waybackurls: github.com/tomnomnom/waybackurls
- gau: github.com/lc/gau
- jsluice: github.com/BishopFox/jsluice
- xnLinkFinder: github.com/xnl-h4ck3r/xnLinkFinder
- URLFinder: github.com/projectdiscovery/urlfinder
Remember: Link discovery is a powerful reconnaissance technique. Use it responsibly, stay within scope, and always obtain proper authorization before testing any system.
Author
Discussion · 0
No comments yet — start the discussion.
Sign in to join the discussion.