deep web

deep web, a part of the Internet that extends beyond the reach of search engines such as Google, Yahoo!, and Bing because it includes unindexed sites, fee-for-service (FFS) content, and private databases such as password-protected e-mail accounts. As such, it is a widely effective source of mass media. The term “deep web” was coined in 2001 by computer scientist Michael K. Bergman, who differentiated it from the “surface web,” where openly viewable and retrievable content resides. The deep web is also known as the “invisible web” or “hidden web,” but it should not be confused with the “dark web,” where encrypted content with hidden IP addresses resides. Called “dark” because it is accessible with anonymity and only through certain networks and software such as Tor, this part of the Internet represents a small fraction of the overall Web.

The deep web represents a vast array of data and content. In fact, it is likely some 500 times larger than the surface web and may contain as much as 96 percent of online content. Much of this content is in the form of databases that are accessible only by password or subscription or for a fee.

How the deep web works

Content that resides on the surface web is accessible because software robots called “spiders” or “crawlers” capture and index it, and search engines assign it rankings. These systems typically scan websites that contain .com, .org., .net, or a similar domain as well as some data and posts at social media sites. As these spiders capture Web pages, they follow embedded links to uncover additional content. Later, when people search for specific content, the results appear in a search engine such as Google, where the content is then openly viewable.

With only about 4 percent of all online content freely accessible (making up the surface web), the remainder is tucked away in the deep web. This means there is no easy, direct way for the general public to search this vast amount of unindexed content. In some cases, websites use various methods to block spiders and prevent indexing. These methods include using CAPTCHAs, multiple IP addresses for the same content, non-HTML content or data that spiders cannot pick up, password protection, and unlinked content.

Types, viewability, and risks of deep web content

There are numerous types of deep web content. These include websites with registration requirements; fee-based video and media on-demand services such as Netflix, HBO Max, Apple TV+, Amazon Prime, and Spotify; and various password-protected entities. Among these are e-mail systems, legal and medical databases, corporate intranets, document libraries, image archives, financial records, scientific databases, and police and government resources as well as gaming sites, cloud storage services such as Dropbox or iCloud, and private sites and services that are not registered and thus do not appear in search engines. However, university researchers and commercial search services such as Google and Microsoft have explored ways to index deep web content, and law enforcement agencies have attempted to develop deep web crawlers that can spot illicit activities, including drug dealing, sex trafficking, and terrorist activity. The success of these efforts has been limited.

There is no inherent risk associated with searching the surface web or entering the deep web. In fact, both acts are common and part of daily life and business today. However, some websites contain digital dangers such as malware, viruses, spyware, and keyloggers (a kind of surveillance software that can monitor and record every keystroke on a computer). It is also possible to find sensitive data on the deep web that can be stolen or abused, and, of course, the possibility of encountering individuals who engage in cybercrime and unethical, even harmful, activities is always a risk in the online world.