So, I have seen various articles about subdomain enumeration and decided to make one in detail without confusing everyone with various tools.
Without any further ado, let’s jump right in.
Prerequisites:
What things do we need before performing a great enumeration?
- API keys of Passive DNS source
- 100% accurate open public DNS resolvers
- A VPS (Virtual Private Server)
1. API keys for Passive DNS data 🔑
What is Passive DNS data?
Whenever a domain is alive on the internet, to access it, a DNS query needs to be made to the DNS resolver. With special probes activated on the DNS resolver, it is possible to record these queries into a database. This doesn’t record which client made the request but, just the fact that at some point a domain has been associated with a specific DNS record.
Hence, we can know, what were the subdomains of a particular root domain that once existed in DNS records. These subdomains in the present time may be alive or dead. (we need further find out which are the valid ones). Along with these various companies have their internet crawlers which continuously keep on crawling the whole internet and discover new assets.
There are a number of services/sources(https://securitytrails.com/, https://censys.io/, https://spyse.com/, https://www.shodan.io/, https://www.whoisxmlapi.com/, https://www.farsightsecurity.com/tools/dnsdb-scout/)
Configuring API keys:
There are 2 types of passive DNS sources:-
- Allow querying their Datasets freely:
A number of sources allow users to freely query their DNS datasets. Check out which sources allow to freely query their dataset here. (we don’t need to care about these sources as our subdomain enumeration tools like https://github.com/OWASP/Amass, https://github.com/projectdiscovery/subfinder, https://github.com/tomnomnom/assetfinder will query them and do the work for us 😋).
2. Need to generate API keys to query Datasets:
Also, a number of sources require you to signup on to their platform and generate a unique API key for yourself so that you are authorized to query and retrieve their historical datasets.
Problems with obtaining free API keys of good passive sources:
- Good passive sources provide API keys for a limited period. (7 days/20 days).
- They provide a limited amount of API query quota. (50 per day/1000 per month ).
- Limited query results (2 pages of data).
Is it worth making API keys?
Yes, absolutely, given below is the comparison between running Subfinder with API keys configured and without.
You can clearly see the difference that using API keys gave me 20000 more subdomains as shown in the below images.
Further, this passive data would be used to generate permutation/alterations which eventually would give us more subdomains.
WITHOUT API KEYS:
WITH API KEYS:
How much time does it takes to signup and obtain API keys?
- There are in total 19 services on which you can sign up and obtain API keys.
- I have created a detailed excel sheet about which sources to signup on for, what’s the validity of API key, their API key quota, rate limits, etc.
- Depending on your consumption of API queries and the validity of API keys, you need to make keep making new accounts at a regular interval of time in order to get the maximum results.
2. 100% accurate open public DNS resolvers
What is a DNS resolver?
A DNS(Domain Name System) resolver is a service that manages “name to IP address” translations. The process of DNS resolution involves converting a hostname (such as www.example.com) into a computer-friendly IP address (such as 192.168.1.1). In short, we get to know whether a domain is alive or not, when we query a DNS resolver.
Why do we need a public DNS resolvers list?
During various subdomain enumeration techniques like bruteforcing or resolution of a large number of domains, we use a base tool called MassDNS.
MassDNS is a simple high-performance tool that is used to check whether a given domain is valid or not. For this purpose, MassDNS needs to be provided with a list of public DNS resolvers. These public resolvers perform DNS queries and check for validation of a domain. Hence, more the number of public resolvers provided, the more concurrent DNS queries are made and thus quicker the output.
How can we create a list of public resolvers?
Dnsvalidator (https://github.com/vortexau/dnsvalidator) is a tool that helps us to generate a valid list of open public DNS resolvers.
https://public-dns.info/ is a website that includes a list of around 5.7k open public DNS resolvers. But some of these wouldn’t be working. So, dnsvalidator helps us to verify only the valid open public DNS resolvers. Dnsvalidator takes this list of 5.7k public resolvers and queries for its resolution using trusted resolvers like Google DNS(8.8.8.8), Cloudflare DNS(1.1.1.1), Quad9(9.9.9.9).
git clone https://github.com/vortexau/dnsvalidator.git
cd dnsvalidator/
python3 setup.py install
dnsvalidator -tL https://public-dns.info/nameservers.txt -threads 100 -o resolvers.txt
3. A VPS (Most Preferable)
What is a VPS?
VPS(Virtual Private Server) can be called as your own dedicated virtual machine in the cloud.
Benefits of a VPS?
VPS tends to have higher bandwidth and better DNS resolution capabilities as compared to your local home system which uses ISP’s DNS resolver which is slow.
A VPS helps while performing various bandwidth-intensive tasks such as DNS resolution & brute-forcing. Alongside when performing such tasks on a local system generally blows up your wifi making it unusable for other users connected on the same network.
Also, you can perform tasks 24/7; 365 days in a VPS, unlike your local system.
How to get one?
There are various cloud providers that provide free 100$ credits like https://www.digitalocean.com/, https://www.linode.com/, https://www.vultr.com/ .
When you signup for the first time. (CC is required).
