About Applebot

Learn about Applebot, the web crawler for Apple.

Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot. It respects customary robots.txt rules and robots meta tags, and it originates in the 17.0.0.0 net block.

Identify Applebot

The user-agent string contains “Applebot” together with additional agent information. Here's an example:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

Customize rules for Applebot

Applebot respects standard robots.txt directives that are targeted at Applebot. In this example, Applebot doesn't try to crawl documents that are under /private/ or /not-allowed/:

User-agent: Applebot         # apple
Disallow: /private/          # disallow this directory

User-agent: *                # any robot
Disallow: /not-allowed/      # disallow this directory

Applebot supports robots meta tags in HTML documents. To specify robots rules in meta tags, put the tags in the <head> section of the document, like this:

<html><head>
<meta name="robots" content="noindex"/>
...
</head>
<body>...</body>
</html>

Applebot also supports the following directives:

noindex: Applebot won't index this page, and it won't appear in Spotlight or Siri Suggestions.
nosnippet: Applebot won't generate a description or web answer for the page. Any suggestions to visit this URL will only include the page's title.
nofollow: Applebot won't follow any links on the page.
none: Applebot won't index, snippet or follow links on the page, as described above.
all: Applebot provides the document for suggestions and snippets the contents so that a short description of the page can appear next to a representative image. Applebot may follow links on the page to provide more suggestions.

To put multiple directives in a single meta tag, use a comma-separated list or multiple meta tags. Here are some examples:

<meta name="robots" content="nosnippet, noindex”>

And:

<meta name="robots" content=“noindex">
<meta name="robots" content=“nosnippet">

Contact us

If you have questions or concerns, please contact us at applebot@apple.com.

Information about products not manufactured by Apple, or independent websites not controlled or tested by Apple, is provided without recommendation or endorsement. Apple assumes no responsibility with regard to the selection, performance, or use of third-party websites or products. Apple makes no representations regarding third-party website accuracy or reliability. Risks are inherent in the use of the Internet. Contact the vendor for additional information. Other company and product names may be trademarks of their respective owners.

Published Date: March 25, 2019