Although WHOIS data can be obtained by using a dedicated Internet protocol, this isn’t as straightforward as it sounds. WHOIS data is stored and updated by registrars, but many providers pose limitations on the how the data may be accessed. Organizations such as Whois XML API, INC. specializes in collecting and organizing WHOIS data within a central database. WHOIS data is of fundamental importance in numerous fields. Here we will explore what this involves and how it is used.
Whatever the field or market, domain names are becoming increasingly important. A domain name is essentially a synonym of a brand, and it’s the fundamental basis of SEO (Search Engine Optimization).
ICANN (Internet Corporation for Assigned Named and Numbers) keeps information on the root of Domain Name Systems. The organization runs a specialized website where anyone can access detailed information on WHOIS data.
When connecting to the Internet, devices have a unique identification number called the IPv4 address. These are assigned a name in order to searchable on the Internet. The names are organized hierarchically to domains. The assignment between IP addresses and names constitutes a DNS (Domain Name System), which is implemented and distributed by the Domain Name Servers. These servers operate on zones: subsets, often a single domain of the hierarchical domain name structure. The information about a zone is contained in so-called zone files.
Top-level domain information is kept and managed by ICANN. There are of two kinds of Top-Level domains: country-code top-level domains (ccTLDs, such as .us, .uk, etc.), and generic top level domains (GTLDs, such as .com, org.) Originally, gTLDs involved extensions such as .com, .org, etc. From 2012 onwards, the set of gTLDs started to significantly extend thanks to the new GTLD program announced by ICANN. These gTLDs are termed as new GTLDs (nGTLDs).
The registration ccTLDs subdomains (as well as lower level domains) are carried out by registrars. These registrars usually consist of competing companies complying with ICANN’s regulations. Although basic contact information is documented when someone registers a domain, but how can we verify the information is accurate? This is where WHOIS (with its distributed database and dedicated protocol) comes into play.
Registrars usually maintain WHOIS servers. This involves detailed information on how, when, and who registered a given domain. This also involves the name and contact data of the registrar as well as the registrant (who registered the domain), update and expiry dates, the address of the primary domain name servers, and the date of the last update of the given record.
Domain branding and marketing research
Having an online presence is currently considered one of the most important aspects of a business, regardless the field or product it sales. An adequate domain name is crucial to help potential clients whatever they’re looking for online. Domain names have a vital role in assisting search engines with queries. In other words, domain names can greatly assist search engines connect customers to products. This is referred to as SEO (Search Engine Optimization).
WHOIS data allows companies to determine if relevant domain names are available. When they’re not available, it provides information on whom a domain belongs to, how that person or company may be contacted, when that registration would expire, and other technical info.
When considering domain names for businesses, searching for domain reputation is also common practice. When buying a domain name, it’s important to consider if the domain name has any conflicting history, or if there are other companies using similar domain names. Even when a domain name does not have a bad track record attached to it, it’s important to keep track of possible trademark violations. This happens when other domains use similar or deriving names (e.g. “amaz0n.com” infringing on “amazon.com”).
There are also circumstances when a domain you previously owned becomes available. Even if you don’t care about that business anymore, the domain can be purchased by a illegitimate company (a porn website, for instance), which may eventually connect back to you.
As such, domain databases can be useful marketing tools. With this, one can identify potential clients, collaborators or competitors, and check a domains history. One can also follow business trends in any given market.
Domain name registration is also a business in itself. This is referred to as “domaining”, and it involves buying domain names that may be sold for a profit to interested companies. This requires following market trends and keeping up with ownership and expiration information.
In conclusion, bulk and historical WHOIS data is a crucial aspect of domain branding, domain protection, and marketing research.
In the field of economic research there are vast amounts of scientific publications in which WHOIS data has been used.
For instance, WHOIS data includes information about a company’s dynamic structure, which helps understand how its SEO functions. For instance, NESTA, a global innovation foundation based in the UK, recently published research that pointed out that most businesses today start through domain registration rather than company registration. The research was carried out by accessing data through Whois database download . Such data allowed the study to understand entrepreneurial trends, including information on the business and market sectors involved, as well as Geolocation and Geo-marketing analysis.
IT security: practice
Cybersecurity agencies are continually facing new challenges. New forms of cyber attacks are constantly being devised, which makes it difficult for Cybersecurity agencies to keep up with.
Malicious agents frequently collect data by phishing or other methods. These are often targeted at the infrastructure of a company.
In a phishing mail campaign, for instance, attackers send emails with the purpose of obtaining private data. These emails are usually from websites that collect data or contain viruses. In such cases, WHOIS data serves identify the individuals or companies behind such websites.
WhoisXMLAPI allows users to access information and reputation scoring of domain names and websites. When having a full WHOIS database, one can develop custom approaches to this problem.
The MITRE corporation, for instance, has developed a pivot table tool for analysts and researchers of cybersecurity to work with WHOIS data obtained from WhoisXMLAPI. Their tool named WhoDat is freely available under the General Public License and can be used for a variety of tasks.
IT security: research
WHOIS data is also a fundamental aspect of IT security research. Data from WhoisXMLAPI has served an important source for various such studies. Protection against malicious websites is an important task within cybersecurity. A common way of identifying these sites is through the use of blacklists that contain large sets of URLs that are considered dangerous.
There are various techniques for compiling such lists. For instance, researchers of the University of Calabria, Italy have recently proposed and demonstrated an efficient machine-learning approach that is based on WHOIS data. This machine-learning approach is able to identify and compile a “blacklist” of malicious websites.
In addition to the aforementioned fields, there are several other areas where WHOIS data bears fundamental importance.
For example, WHOIS data is often used within the field of bank transaction fraud, where banks and payment processors need to identify physical entities associated with IP addresses. These organizations also use geolocation information to identify suspicious transactions. Criminal investigators and lawyers also depend on WHOIS data for identifying criminal and fraudulent activities. Geo-fencing can also be used in such circumstances.
How to obtain BULK WHOIS data?
WHOIS data may be accessed via a dedicated Internet protocol. As long as you need data concerning a few domains, you should not encounter any issues.
However, if you require data on more than a few domains, one can easily encounter issues as many operators pose limits on the frequency WHOIS data may be accessed.
Organizations such as WHOIS XML API provide up-to-date data through restful interfaces. The data is provided in standard formats including JSON or XML.