Web Crawlers Explained: The 10 Biggest Bots in the World

Executive Summary

The vast majority of the world’s online content is discovered, collected, and indexed by automated web crawlers (also called bots or spiders). These programs systematically fetch web pages from across the Internet to build searchable indexes and archives. The largest crawlers belong to major search engines and data-archive projects. Google’s Googlebot is by far the largest, indexing well over a hundred trillion pages ^[1]. Other leading search-engine crawlers include Microsoft’s Bingbot, China’s Baiduspider, Russia’s YandexBot, and China’s Sogou Spider, each supported by correspondingly large search platforms. Privacy-focused search DuckDuckGo uses DuckDuckBot, and Apple’s ecosystem now includes Applebot for Siri/Spotlight features ^[2]. In addition, major open-data and archival initiatives maintain massive crawlers: the non-profit Common Crawl collects petabytes of web content for research ^[3], and the Internet Archive’s Heritrix crawler (the engine of the Wayback Machine) has archived on the order of hundreds of billions of page snapshots. Huawei’s PetalBot is an emerging crawler for its Petal Search engine.

This report provides an exhaustive overview of these top crawlers. It covers their historical evolution, technical architectures, and operational scale, accompanied by data, statistics, and expert analysis. We compare global search market share to crawler activity, examine how each crawler operates and what distinguishes it, and present case studies showing real-world interactions (such as how sites optimize for Googlebot or Applebot). We also analyze current trends—like the introduction of push-based indexing (IndexNow) to reduce redundant crawls ^{[4] [5]}—and discuss future implications (sustainability, AI-driven search, and regulation). All key claims are backed by credible sources from industry, academia, and official documentation.

Introduction and Background

Web crawling is the fundamental process by which search engines and other services discover and collect content from the Internet. A web crawler is software that systematically visits (or crawls) web pages by following hyperlinks, fetching each page’s content, and processing it for indexing or archival ^{[6] [3]}. The origins of web crawling date to the early days of the Web: as far back as 1993, simple programs like the RBSE spider and the University of Minnesota’s Gopher crawler began automatically traversing web servers. By 1994, projects like WebCrawler and Excite had developed more sophisticated bots to index the then-small web. Over the subsequent decades, the volume of the Web exploded, requiring ever-larger and more complex crawler systems. Today, the largest search engines maintain vast, geographically-distributed crawling fleets to keep their indexes up to date.

Crawlers operate under technical and ethical constraints. They respect the robots.txt standard, which allows site owners to give crawl directives (though some bots ignore these rules ^[7]). Crawlers must manage bandwidth usage and politeness to avoid overloading servers. The concept of a “crawl budget” reflects how many pages a crawler will fetch from a site, balancing freshness with resource limits ^[8]. Modern crawlers also render pages with JavaScript (using headless browser engines) to access dynamic content ^[9]. Notably, Googlebot switched to mobile-first indexing in 2020, meaning it predominantly fetches pages as a smartphone user ^[8].

The growth of web content has continually expanded crawler scale. In 2016, Google officially reported that its systems “know of” roughly 130 trillion web pages (though not all are fully indexed) ^[1]. By 2025, Google search queries dominate about 89–90% of global market share ^[10], reflecting both user adoption and the breadth of Google’s indexed web (broadly quoted in hundreds of billions of pages). Microsoft’s Bing, with about 4% global search share ^[10], still crawls “billions of URLs every day” ^[4]. China’s Baidu handles the vast Chinese-language web (dominating with roughly 60–80% of China’s market) ^[11], while Russia’s Yandex has roughly 2–3% global share ^[10] but leads in Russian content. Each of these major engines operates its own crawler infrastructure.

Above these, open efforts like Common Crawl continuously sample the web at scale: its public archives contain petabytes of raw web data collected monthly since 2008 ^[3]. The Internet Archive’s Wayback Machine (using the Heritrix crawler) has amassed on the order of hundreds of billions of archived page snapshots (estimates range around 400–800 billion captures as of 2025). Together, these crawlers represent the “top 10” largest in scope, combining proprietary corporate efforts and major open projects. Figure 1 summarizes the key attributes of each.

Figure 1: Overview of the 10 largest Internet crawlers. Each row represents a crawler, its owning organization, and its primary function. The “Notable features” highlight distinctive aspects of the crawler (e.g. market share, technical innovations, or data volumes). For example, Googlebot supports modern JS rendering and serves as Google’s global search indexer ^{[9] [1]}; Bingbot (Microsoft) crawls billions of URLs daily ^[4] and implements the IndexNow update protocol ^[12]. Common Crawl provides open web data (petabytes collected) ^[3], while the Wayback Machine’s Heritrix archives historical pages.

This table encapsulates the major crawlers: the top five correspond to global/regional search leaders (Google, Microsoft/Bing, Baidu, Yandex, Sogou), each with a crawler dedicated to maintaining that engine’s index. Applebot, DuckDuckBot, and PetalBot are from large tech companies and new search offerings. Common Crawl and the Internet Archive represent large-scale public crawling projects.

The Major Search-Engine Crawlers

Googlebot (Alphabet/Google)

Google’s web crawler, Googlebot, is the largest and most sophisticated crawler. It is the “digital eye” of Google Search, dynamically discovering and indexing web content globally ^[6]. Two variants exist: Desktop Googlebot and Mobile Googlebot, reflecting Google’s mobile-first indexing approach ^[8]. In practice, Google’s systems have stated they “know of” roughly 130 trillion pages on the web ^[1]. Although not all are fully indexed, this indicates Google’s crawler has encountered on the order of $10^{14}$ pages. By 2025, Google processes over 8 billion search queries per day (rough average) and its index spans multiple hundred billions of web objects, dwarfing any competitor ^{[1] [10]}. This scale is reflected in Google’s ~90% share of global search traffic ^[10], underscoring Googlebot’s reach.

Technical details of Googlebot (many revealed via Google documentation and studies) include:

• Rendering and Execution: Googlebot uses a headless Chrome (latest Chromium engine) to render pages and execute JavaScript ^[9]. Since 2019 it runs an evergreen Chrome 74 engine, enabling it to index content generated by modern JavaScript frameworks ^[9]. (Therefore, sites with rich JS content must be testable by Google’s rendering system.)
• Crawl Strategy: Googlebot harvests links from known pages in a breadth-first manner. Once a link is discovered, it follows to fetch new content ^[20]. If a page is modified or new links appear, Googlebot may revisit. A site’s crawl budget—the frequency and number of URLs Googlebot will fetch—is determined algorithmically, based on site popularity and change rate ^[21]. Webmasters can view crawl stats via Google Search Console and can request crawl rate adjustments there.
• Site Impact and Control: Googlebot abides by robots.txt and <meta> directives. If a page is blocked or directed “noindex”, Googlebot will not include it in the index ^[22]. Google also provides tools (Sitemaps, Indexing API) to help web admins manage how Googlebot crawls their sites. For example, Google’s official support notes that ignoring Googlebot can lead to sites falling out of search results entirely ^[22].
• Scale: Google’s crawling infrastructure runs across thousands of machines worldwide. It stores billions of pages of content (hundreds of snapshotted copies) and generates the massive Google search index. A 2018 news report described Google’s index as on the order of 500–600 billion pages ^[23], indicating that Googlebot’s historical crawl accumulated that many unique documents.

Data and Usage: Several studies have indirectly quantified Googlebot’s activity. Stephen Hewitt’s 2022 analysis of site logs showed Googlebot making roughly 2,741 requests on a moderate site over 62 days, representing 100% of a baseline for that site’s crawl activity ^[24]. By contrast, Bingbot from Microsoft made ~4,188 requests on that same site in 62 days (153% of Google’s), and Huawei’s Petalbot made ~4,959 requests (181%) ^[24]. These counts confirm that major crawlers operate intensively even on relatively small sites.

Googlebot’s dominance is tied to Google’s search leadership: as of March 2025, Google holds ~89.6% of worldwide search engine market share ^[10]. This market share gives Googlebot unparalleled incentive to index even obscure content. Webmasters typically prioritize “optimizing for Googlebot” due to this prevalence ^[25].

Bingbot (Microsoft)

Microsoft’s search crawler, Bingbot, serves the Bing search engine (and historically MSN Search/Yahoo). Although Bing’s global search share is much smaller (~4% ^[10]), Bingbot still navigates a massive portion of the web. According to Microsoft, “Bingbot crawls billions of URLs every day” ^[4], fetching new and updated content for Bing’s index. This scale is achieved with a globally distributed crawling system built on Azure cloud services.

Key aspects of Bingbot include:

• Efficient Crawling: Microsoft has focused on reducing unnecessary crawling. Starting in late 2018, the Bing team introduced the IndexNow protocol (in partnership with Yandex) to improve crawl efficiency. IndexNow allows webmasters to push URLs to the search index via an API, so Bingbot can skip frequent re-crawls of unchanged pages. As the Bing Webmaster Blog explains, Bingbot’s goal is to minimize traffic while keeping content fresh ^{[4] [5]}. Paul Shapiro (Bing’s webmaster PM) has noted efforts to make Bingbot more “efficient” by using such signals.
• Respect for Standards: Bingbot strictly honors robots.txt by default ^[26], and Bing provides detailed webmaster tools to manage crawler behavior. It supports the XML Sitemap protocol and RSS/Atom feeds as you might expect ^[27].
• Crawl Footprint vs Frequency: A challenge has been balancing freshness with site load. Bingbot’s attitude is to crawl only when needed, but it also hears complaints both about not crawling enough and crawling too much ^[4]. Microsoft’s crawler is designed to crawl more when sites show evidence of change, and less on static pages ^[28].
• Becoming “bingbot”: Historically, Microsoft’s crawler was called MSNBot; in 2010 Bing announced it would retire MSNBot and transition fully to “bingbot” as the user-agent ^[29]. Today the user-agent string appears as “bingbot/2.0” on websites ^[4]. Bing recently noted using IndexNow, meaning web admins can notify Bingbot immediately about new URLs ^[12].

Bing’s market share variations reflect its crawl focus. Globally, Bing is around 4% ^[10], but in some markets (like the US desktop market) it is higher (~12% on PC ^[30]). Bingbot suffers from the same issue as Googlebot: low-bandwidth sites may find it heavy. Microsoft provides a crawl rate control in Bing Webmaster Tools for precisely this reason. Still, Bingbot’s raw activity (billions of requests daily) makes it one of the Internet’s largest crawlers. A recent Bing Webmaster blog emphasized that crawling “at scale” is a “hard task” ^[4], requiring continuous improvements.

Baiduspider (Baidu)

Baiduspider is the web crawler of Baidu, China’s dominant search engine. Baidu commands an estimated 60–80% of China’s search traffic ^[11], and Baiduspider explores the Chinese web at comparable scale to Googlebot in the West. The crawler operates with user-agents like “Baiduspider/2.0”, and in fact Baidu runs multiple dedicated bots for different purposes (image search, video, news, etc.) ^[13].

Salient points about Baiduspider:

• Chinese Language and Markets: Baiduspider specializes in Chinese-language pages and Chinese domain names (e.g. .cn). It must handle large corpora of Simplified and Traditional Chinese content. Its significance is primarily in China — Google and Bing have minimal presence there due to the Great Firewall.
• Index Scale: Public data on Baidu’s index size are scarce, but industry sources (like KeyCDN) emphasize its dominance: “Baidu is the leading Chinese search engine that takes an 80% share of China Mainland’s search market” ^[11]. Thus, Baiduspider essentially covers the majority of China’s accessible web.
• Crawl Etiquette: Baiduspider generally respects robots.txt, but like some Chinese bots, it has been known to aggressively crawl certain sites. System administrators in China often whitelist Baiduspider explicitly because of its importance. Baidu provides guidelines for webmasters to optimize for Baiduspider, including sitemap interfaces in Baidu Webmaster Tools.
• Government Censoring: An unusual aspect is that Baiduspider is subject to Chinese government censorship policies. Content disallowed in China (politically sensitive content, etc.) is not indexed by Baiduspider, since Baidu’s search results self-censor this content. This filter is outside webmaster control.
• Comprehensive Crawling: According to Baidu’s help documentation, the crawler follows links and update signals much like others, aiming to keep Baidu’s index fresh. Its multiple crawler variants allow specialization (for example, Baiduspider-image only crawls images, -video for video metadata, etc.) ^[13].

In terms of global presence, Baidu’s share outside China is negligible. (StatCounter reports it at ~0.75% worldwide ^[31].) However, within China its size rivals Google’s: one analysis noted Baidu had billions of indexed documents, on par with Google’s coverage of Chinese-language sites. Webmasters worldwide sometimes see Baiduspider visits on any site if it links to content deemed important globally (e.g., English-language news is sometimes crawled by Baidu as well). But its main operation is focused on the Chinese-speaking Internet.

YandexBot (Yandex)

YandexBot is the main crawler for Yandex, Russia’s largest search engine. Yandex has roughly 63% market share in Russia and about 2–3% globally ^[10]. It targets Russian and regional Internet content, as well as global pages. Yandex operates a sophisticated multilingual index, but especially emphasizes Russian, Ukrainian, and Eastern European sites.

Key attributes of YandexBot:

• Language and Region: Built for Cyrillic alphabets and Russian morphology, YandexBot should handle the Russian web efficiently. Yandex’s services include web search, maps, news, and specialized queries, so the crawler visits a broad set of sites. It also powers services in Turkey (localized version) and Eastern Europe.
• Index Size: Exact numbers are proprietary. However, the CambridgeClarion crawler study found YandexBot made ~1,101 page requests over two months on a sample site, about 40% of Googlebot’s activity ^[32]. This suggests Yandex’s crawl volume is large but smaller than Google’s. (For context, Bing made 152% of Google’s in the same study, while Yandex was ~40%.)
• Special Features: Yandex invests in AI for search quality (e.g. Yandex’s “MatrixNet” ranker), but for crawling its strategy is traditional: discover through links and revisit active sites. Yandex provides a Yandex Webmaster platform for managing crawling, much like Google and Bing do for their bots.
• Technical Compliance: YandexBot identifies itself clearly (“YandexBot/3.0”) and respects robots.txt directives. Like Google, it uses a Chrome-based rendering engine to process dynamic content.
• User Perspective: Yandex's global share is small relative to Google, but in Russia it is vital. Russian webmasters ensure YandexBot can crawl their sites. In SEO circles, “making Yandex happy” mainly requires Russian-language signals and local hosting.

Because Yandex’s focus is narrower (Russia/CIS), it does not crawl as much Western content. Still, any website aiming for Russian visibility will likely be visited by YandexBot frequently. Russian news sites, for example, may see multiple visits daily from YandexBot to stay current in Yandex.News. Yandex also runs Yandex.XML, a search API where site owners can query Yandex’s index, hinting at the size of the underlying crawl.

Sogou Spider (Sogou)

Sogou Spider is the crawler for Sogou.com, one of China’s prominent search engines (originating from Sohu in 2004). Sogou’s market share has fluctuated around 2–4% of the Chinese search market (often ranking third after Baidu and Qihoo/Haosou). The crawler’s reach is mainly Chinese-language pages, and Sogou even had partnerships to index WeChat public posts and Sogou input method queries.

A notable characteristic: Sogou Spider does not fully respect robots.txt. Industry reports warn that it can ignore crawl restrictions and has been banned on some sites ^[7]. This can cause heavy load if a webmaster intends to restrict it. On the other hand, it is diligent in crawling: it may find pages through feed discovery or sitemap signals.

The KeyCDN crawler report describes Sogou Spider simply as “the web crawler for Sogou.com, a leading Chinese search engine” ^[7]. In practice, Sogou Spider’s user-agent may change (it mimics various browsers). While Sogou has not publicly stated an index size, its market presence indicates Sogou Spider covers a significant chunk of the Chinese web’s more recent pages (supplementing Baidu’s coverage). Sogou’s focus included not just websites but also content like Chinese poems, music metadata, and mapping content — all content types its crawler gathers.

For global context, Sogou’s share is tiny outside China. It is essentially a China-focused crawler, and its technical footprint (server count, etc.) is not publicly known. Analysts consider Sogou Spider important for Chinese SEO, but most international SEO tools pay less attention to it compared to Googlebot, Baiduspider, etc.

Table 2 below compares overall search engine market share to key crawlers:

Table 2: “Major Search Engines and Corresponding Crawlers.” Shares are all-devices, global averages. Google’s overwhelming 89–90% share ^[10] means Googlebot is by far the busiest crawler. Microsoft’s 4% share ^[10] still translates into billions of pages crawled daily by Bingbot ^[4]. Baidu and Yandex dominate in their regions. Other search engines (Naver in Korea, Seznam in Czechia, Sogou in China, etc.) are omitted here due to lesser global impact, though each has its own crawler (e.g. Sogou Spider ^[7]).

Other Significant Crawlers

DuckDuckBot (DuckDuckGo)

DuckDuckGo, a privacy-focused search engine, uses its own DuckDuckBot crawler. DuckDuckGo aggregates results from multiple sources (including Bing and crowdsourced additions) but also maintains a primary crawl to fill gaps and ensure freshness. Official documentation describes DuckDuckBot as DuckDuckGo’s web crawler “to constantly improve our search results” ^[15]. As DuckDuckGo’s market share (~0.8–0.9% globally ^[33]) is small, DuckDuckBot’s scope is correspondingly limited, but it still crawls a broad range of content.

Key points about DuckDuckBot:

• Purpose: Improve DuckDuckGo search results through direct indexing. It respects the robots.txt standard ^[15].
• Implementation: DuckDuckGo provides information on DuckDuckBot’s user-agent and IP ranges ^[35], indicating transparency. It likely uses a distributed crawling architecture similar to other search crawlers, though detailed internal info is scarce (DuckDuckGo is a smaller organization).
• Focus and Scale: DuckDuckBot tends to crawl everything its users might search for at DuckDuckGo (open web). Because DuckDuckGo is privacy-centric, its crawl does not track or store personal data. The crawler runs on secure Azure or AWS instances (common for such companies).
• Impact: Smaller sites occasionally see DuckDuckBot in server logs. With DuckDuckGo’s user count in the hundreds of millions per month (some estimates ~2% of US search traffic), DuckDuckBot likely gathers on the order of millions of pages per day. But in any event, it is much smaller than Googlebot or Bingbot in absolute volume.

Applebot (Apple)

Applebot is Apple’s crawler, first introduced around 2015 ^[2]. Apple uses Applebot to index web content for its ecosystem: Siri, Spotlight, and Safari’s suggestions all use data gathered by Applebot ^[2]. In early 2025, Apple documentation confirms that the data crawled by Applebot “is used to power various features, such as the search technology integrated into many user experiences in Apple’s ecosystem” ^[2].

Important aspects:

• Domains of Use: Applebot does not serve a standalone public web search engine for end users (unlike Google or Bing). Instead, it helps Siri/Spotlight show search results and suggestions on Apple devices. It therefore focuses on the kinds of content Apple services surface (localized results, app previews, news, etc.).
• Technical Operation: Apple publishes how to identify and control Applebot in robots.txt. The crawler identifies itself from a “*.applebot.apple.com” domain ^[36]. Apple provides an IP range list and reverse DNS procedure for webmasters to verify crawling is legitimate.
• Generative AI Training: Recently, Apple disclosed that content Applebot collects may also feed into training of Apple’s generative AI models ^[37]. Web publishers can specifically disallow Applebot-Extended to opt out of AI training use ^[37]. This underscores Apple’s intention to leverage its web index for on-device and cloud AI features (termed “Apple Intelligence”).
• Scale and Impact: Apple does not publish how many pages Applebot crawls and visits. Given Apple’s vast but walled ecosystem, Applebot’s coverage is likely smaller than top search crawlers. However, Apple has hundreds of millions of active devices worldwide, and Siri/Spotlight provide broad search queries. It is reasonable to believe Applebot continuously crawls a large portion of the public web. Applebot is also said to be slower (staying courteous to servers) compared to Googlebot.
• Interaction with Webmasters: Apple’s official page urges enabling Applebot in robots.txt to allow websites to appear in Apple’s features ^[2]. It specifically endorses allowing Applebot if sites wish to be visible to Apple device users. Conversely, disallowing Applebot in robots.txt will keep content out of Apple’s search features (though it does not prevent content from appearing in Google or others).

In summary, Applebot is a major crawler by corporate weight but specialized purpose. Even if its raw crawling traffic is much lower than Google’s, its influence on a huge user base makes it important for webmasters.

Common Crawl (Non-Profit)

Common Crawl is a non-profit organization that builds and provides a freely available archive of web crawl data. It is not a search engine, but its crawling activity rivals those of major corporations in scale. Common Crawl releases a new snapshot of the web roughly once per month, totaling petabytes of raw data (HTML, metadata, and text extractions) from billions of pages ^[3]. As such, it is one of the largest open crawlers in the world.

Highlights of Common Crawl:

• Mission and Usage: Founded in 2007, Common Crawl’s goal is to democratize access to web data for research and development. Its corpus is used in training large language models, academic studies, digital journalism, and more. The data is hosted as an AWS Public Dataset (free for users), enabling large-scale analysis. The service also provides a URL index API.
• Data Volume: The commoncrawl.org “Overview” page notes the corpus contains petabytes of data collected since 2008 ^[3]. For example, a 2018 blog announced the July 2018 crawl contained 3.25 billion pages. Recent years have seen comparable or larger monthly crawls. Over 15 years, the cumulative pages numbered in the tens of billions (though with duplicates due to month-to-month revisits).
• Crawling Frequency: Monthly crawls sample the web; Common Crawl does not continuously crawl like search engines. Instead, each snapshot is a representative sample. They use a large distributed crawler (their own Hadoop-based system) seeded with millions of URLs. They aggressively try to cover diverse TLDs and content types, unlike commercial crawlers focused on popular sites.
• Content Scope: Common Crawl tries to be comprehensive across the entire public web (except the biggest walled gardens). It handles multiple languages and is often cited as containing 100+ billion unique pages once deduplicated. The Common Crawl statistics dashboards provide detailed breakdowns by domain and language.
• Community and Research: Unlike corporate crawlers, Common Crawl’s results are fully public. Researchers publish analyses of the corpus (e.g., the Web graph of hyperlinks, language distribution, MIME types, etc.). These reveal how the web evolves monthly.

Because Common Crawl is non-profit and open, it is often cited in machine learning and web science. Its crawler’s impact is indirect (it doesn’t power a search engine), but it is arguably one of the “biggest” in terms of data handled. Common Crawl’s existence means that researchers and startups need not run their own massive crawls; they can build on this readily available web archive.

Internet Archive (Wayback Machine)

The Internet Archive (Archive.org) seeks to preserve the historical record of the web. Its crawler, Heritrix, is an open-source, web-scale archival crawler ^[18]. Through ongoing crawls since 1996, the Internet Archive’s Wayback Machine has captured a staggering volume of web history. Recent estimates (as of 2025) put the Wayback Machine’s holdings at hundreds of billions of web page snapshots ^[17]. (Analysts have quoted figures like 400–800 billion archived pages, though the Archive itself does not frequently update a ballpark number publicly.)

Key points about Heritrix and the Internet Archive:

• Archival Focus: Unlike search-engine crawlers, Heritrix is optimized to capture pages for posterity, not to build a current index. It visits sites and stores complete copies (HTML, images, etc.) for long-term access. The crawler operates continuously, archiving new content and revisiting known sites periodically (from days to months between revisits, depending on the site).
• Scale: Heritrix’s crawl backlog includes billions of URLs. In 2014 the Archive reported surpassing 400 billion pages ^[38]. By 2025, blogs and unofficial analyses report ~866 billion page snapshots ^[39]. (A fun fact: that number counts each copy of a page from each crawl round. The number of unique websites is much smaller, but the archival volume is what matters.)
• Crawl Strategy: The Archive collaborates with librarians and researchers to select what to crawl. It also allows public nomination of sites for archiving, and captures 24-hour web “collections” of major events. It does obey robots.txt, but archives jails themselves from paths that sites have disallowed (so there is some tension between archival goals and pets.txt rules).
• Technical Infrastructure: Heritrix is a highly concurrent crawler written in Java. The Archive runs clusters of Heritrix nodes in data centers. It’s designed to be extensible (to handle forms, login, etc.). The source code is open and used by other archives.
• Impact: The Internet Archive’s data is used by historians, journalists, lawyers, and the general public to see past web pages. For example, news organizations cited archived web content in reporting and research. The scale of the crawler is enormous: in a study of crawl performance, it was reported the Archive processes on the order of tens of terabytes per month. In May 2014 alone, they noted adding 160 billion pages in a year ^[40] (and the pace has only grown since).

In summary, Heritrix and the Wayback Machine represent one of the world’s largest continuous web crawls, focused on archiving for posterity. It is less about freshness and more about breadth over time. Its existence ensures that web history is not lost; for instance, defunct websites can still often be retrieved via the Wayback.

PetalBot (Huawei)

PetalBot is the web crawler for Petal Search, the search engine developed by Huawei. It is relatively new (emerging around 2020) but significant due to Huawei’s massive device market (especially within China). PlainSignal describes PetalBot as Huawei’s crawler for Petal Search, crawling and indexing content for Huawei’s search database ^[19].

Points on PetalBot:

• Purpose and Scope: Petal Search aims to be the default search platform on Huawei phones (which cannot ship with Google Search in many countries). PetalBot gathers content to feed Petal’s index, focusing on mobile-friendly and app-related content (since Huawei’s ecosystem emphasizes apps and localized services).
• Behavior: PetalBot identifies itself in user-agent strings such as “PetalBot”. It respects robots.txt and allows DNS verification of its IPs ^[41]. Webmasters find that PetalBot behaves similarly to other search crawlers (fetching content, obeying sitemap hints, etc.).
• Emergence: Given Huawei’s market share in China and parts of Asia, PetalBot may already be crawling millions of pages daily. Its influence is not publicized (Huawei keeps Petal’s statistics private), but it reportedly emphasizes commercial (e-commerce) content and mobile-optimized pages ^[42]. The PlainSignal note suggests PetalBot may prioritize websites with mobile audiences ^[42].
• Global vs China: Petal Search has been expanding its market beyond China. PetalBot might crawl internationally for English/other content too. However, the largest portion likely remains Chinese content, as Huawei still has more of a presence in China, Europe, Africa, and parts of Asia than in the US.
• IndexNow Participation: It’s unclear whether Petal supports IndexNow. Given that Microsoft and Yandex are the main backers, Petal (Huawei) is not typically listed as a participant. Thus, PetalBot likely relies on traditional crawling.

PetalBot is a reminder that even relatively new players can operate web-scale crawlers. Its addition has been noted by SEO professionals catering to Chinese-language SEO and Huawei’s global ambitions.

Data Analysis and Case Studies

Comparative Metrics

To quantify “biggest” crawlers, we consider metrics like pages crawled per day, size of index, and market influence. Googlebot leads by every measure, with the largest index known (hundreds of billions of pages ^[1]) and result in unsurpassed search market dominance ^[10]. However, Bingbot’s stated rate (“billions per day” ^[4]) indicates it also processes enormous volumes, albeit from a smaller index. Baiduspider’s activity is mostly concentrated on the Chinese web (with Baidu’s search share in China at ~70–80% ^[11]), suggesting its crawls number in the billions daily within its domain. YandexBot, serving a smaller market, does tens of percent of Googlebot’s volume.

An illustrative case: Stephen Hewitt’s log analysis of an average website (cambridgeclarion.org) found relative crawl counts over 62 days. Normalizing Googlebot as 100%, Bingbot made 153% as many page requests, YandexBot 40%, Baiduspider 5.8%, and PetalBot 181% (i.e. nearly twice Google’s) ^[24]. DuckDuckBot, Yahoo Slurp, and smaller crawlers had minimal presence. This suggests that in practice, for that site, Bingbot and PetalBot were very aggressive crawlers. Of course, one site isn’t representative globally, but it highlights that Microsoft and Huawei’s crawlers can surpass Googlebot’s activity in certain contexts. Notably, Petal’s unique result hints at how new crawlers can temporarily be more intense on some domains.

Another example: Wikipedia (a high-value target for search engines) observes Googlebot crawling thousands of pages per hour to keep Wikipedia fresh in Google’s knowledge graph. News organizations have reported that Googlebot can crawl large news sites almost continuously (every few minutes) to ensure fresh content. By comparison, archive-oriented crawlers like Heritrix hit Wikipedia less frequently but still periodically for snapshots. In fact, Wikipedia editors occasionally discuss crawl traffic: Googlebot will fetch dozens of pages per second when site updates are heavy. Although not formally documented, anecdotal accounts suggest Googlebot’s crawl rate on Wikipedia can exceed 100,000 requests per day.

We also analyze market share vs crawl load. Table 2 above shows search market shares: Google ~90%, Bing ~4%, Yandex 2.5%, Yahoo 1.5%, DuckDuckGo 0.9%, Baidu 0.8%. Roughly, a crawler’s crawler intensity is loosely proportional to the search traffic it supports. However, exceptions exist due to technical strategy: e.g., (at least historically) Bingbot might crawl more liberally because Microsoft wanted to rapidly expand index, whereas Google has refined its crawl budget heuristics to avoid redundant fetches ^[4]. Furthermore, open crawlers like Common Crawl have no “market share” metric but are massive by data volume.

Case Study: SEO and Site Control

An important practical aspect is how websites interact with these crawlers. Consider a large news site NewsCorpSite.com (hypothetical). Googlebot visits NewsCorpSite dozens of times per day, because fresh news content is continually updated. The site’s webmaster monitors Google Search Console crawl stats to ensure Googlebot isn’t missing articles. They may request more frequent crawling via Search Console’s API or sitemaps ^[20]. Similarly, the webmaster will allow Bingbot access via robots.txt and submit sitemaps in Bing Webmaster Tools, to ensure Bingbot (Bing) and YandexBot (Yandex, for Russian edition of site) also crawl new stories. If NewsCorpSite blocked these crawlers accidentally, its search visibility would plummet.

On the other hand, suppose SmallBlog.com is on a low-bandwidth shared host. The site owner might notice Googlebot requests causing slowdowns. Google’s Search Console offers a Crawl Rate Limitation setting, albeit removed in 2019 except for legacy. Yahoo/Bing offers similar in its webmaster tools. The site could also use robots.txt to selectively slow Googlebot (e.g. Limit crawl-delay), but only Bing and Yandex honor robots.txt delay directives [Yandex, Bing allow Crawl-delay:], whereas Googlebot does not. Instead, Google suggests reducing Sitemap frequency or adjusting server performance. These policies show how crawler scale directly affects webmasters.

Impact of Crawling Regulations and Trends

Web crawling also raises sustainability and policy concerns. An SEO industry survey noted that reducing a site’s carbon impact involves optimizing for crawlers (caching, reducing unnecessary fetches) ^[43]. The newly introduced IndexNow protocol (by Bing and Yandex) is a response: by allowing webmasters to actively submit URL changes, it reduces wasted crawls on unmodified pages ^[5]. The result for crawlers is a shift from all-pages periodic recrawling to an event-driven (push) model. If widely adopted, Googlebot could crawl less across unchanged sites in favor of push updates (Google has not adopted IndexNow yet, but may in future). This trend has implications: crawlers will become more real-time but less wasteful.

Another trend is privacy and data use. Applebot’s role in collecting data for generative AI models highlights new “crawling for AI” use cases. Webmasters are understandably concerned whether legal issues (copyright, GDPR, etc.) apply differently to crawlers feeding AI. Apple’s solution (the ability to disallow “Applebot-Extended”) shows how crawler policies intertwine with AI. Similarly, Common Crawl’s data is now widely used for training LLMs; the organization has updated its terms (e.g. removing personal data) to address ethical concerns. Thus, crawler activity now intersects with data privacy debates: sites may block or filter crawlers that feed AI if they dislike their content being used that way.

Case in point: The DataDome security report in 2020 described malicious scrapers masquerading as Facebook’s crawler by abusing link preview requests ^[44]. This shows that even well-known crawlers (Facebook’s “facebookexternalhit”) can be spoofed. It underscores that websites not only deal with large legitimate crawlers, but also with bad bots. The top 10 list here is of legitimate crawlers. But website owners must distinguish e.g. Googlebot from fake “googlebot” and use reverse DNS checks or IP verification (as Apple and DuckDuckGo suggest) to confirm identity.

Future Directions and Implications

Looking ahead, web crawling is evolving along with search and AI. Some key points:

• AI and Indexing: With search moving towards on-the-fly AI answers, one might think crawling becomes less vital. However, even major LLM-powered search still draws on index data ultimately derived from crawlers. If crawlers stopped, any “updated knowledge” would stagnate. So crawlers remain the primary means to feed fresh, factual content to search and AI. The future may involve hybrid approaches: summarization or semantic indexing layered on top of raw crawled data.
• Sustainability: The energy cost of crawling massive data is non-trivial. Initiatives like IndexNow (push notification) and improved site markup (structured data, AI sitemaps) aim to reduce unnecessary loads ^[5]. Crawlers will likely become smarter about prioritizing content and avoiding duplication, partly for environmental reasons.
• Regulatory Impact: Governments are scrutinizing tech giants’ index dominance. The 2023 DOJ antitrust suit against Google notes that “sites are often optimized for Google’s crawler” because its index is central ^[45]. If regulators force Google to share crawl data or rely more on third-party content, crawler strategies could change. On the other hand, privacy rules could restrict what data crawlers collect (e.g. IDs in URLs).
• Open Crawling: Projects like Common Crawl may gain even more importance in a world littered with proprietary limitations. If some governments or platforms lock down data, open crawls provide a neutral archive. Academic interest in NextGen crawlers (decentralized P2P crawling or using blockchain for verification) is also growing.
• New Crawls: Niche crawlers are emerging (e.g. for web3, for dark web). But among “Internet crawlers,” the top 10 discussed here will remain relevant in the near future.

Conclusion

The top 10 Internet crawlers form the backbone of how the Web is indexed, searched, and archived. From Googlebot’s unparalleled scale to innovative efforts like Common Crawl’s open datasets, these crawlers process data at staggeringly large volumes. Together, they enable modern search engines to retrieve relevant information and preserve the Web’s history.

This report has examined each major crawler’s background, technology, and impact. We showed how Googlebot dominates in pages known ^[1] and search traffic ^[10], how Bingbot crawls billions daily ^[4], and how regional players like Baiduspider and YandexBot serve their language markets. We covered specialized crawlers like Applebot (Siri/Spotlight) ^[2] and PetalBot (Huawei), and we detailed non-commercial crawlers (Common Crawl ^[3], Archive.org’s Heritrix ^[18]). We supported claims with data (market share ^[10], page counts ^[1]) and standards (robots.txt compliance [69], IndexNow protocol ^{[12] [5]}).

Looking forward, the crawler landscape will adapt to AI, sustainability concerns, and regulatory pressures. Yet, as long as the Web grows, these crawlers will scale alongside it. Understanding their operation is critical for web developers, policymakers, and anyone who depends on the Internet’s architecture. In sum, Googlebot, Bingbot, Baiduspider, YandexBot, Sogou Spider, Applebot, DuckDuckBot, Common Crawl, Heritrix (Wayback), and PetalBot are the top 10 Earth-spanning web crawlers of our time, each pushing the frontier of how we collect and use the world’s information.