About Applebot

Find out about Applebot, the web crawler for Apple.

The data crawled by Applebot is used to power various features, such as the search technology that is integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products.

Applebot accesses many kinds of resources from web servers, including but not limited to robots.txt, sitemaps, RSS feeds, HTML, sub resources needed to render pages (such as JavaScript), Ajax requests, images and more.

Identifying Applebot

Traffic coming from Applebot is generally identified by using reverse DNS in the *.applebot.apple.com domain.

Another way is to match the IP address with a CIDR prefix contained in the following JSON file: Applebot IP CIDRs.

Reverse DNS

The host command can be used to determine if an IP address is part of Applebot. These examples show the host command and its result:

$ host 17-58-101-179.applebot.apple.com 17-58-101-179.applebot.apple.com has address 17.58.101.179.

The host command can also be used to verify that the DNS points to the same IP address:

$ host 17.58.101.179 179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.

User agents

A user agent helps webmasters identify crawler traffic, so that they can get accurate access log reports of crawler activity and control access to the site via robots.txt.

Applebot powers several user agents, including Search and Podcasts.

Search

For search web crawling and rendering, Applebot uses the following format:

The user agent string contains “Applebot” and other information. The following is the general format:

Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko)Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot)

Example for desktop:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15(KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Example for mobile:

Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

Occasionally, Applebot will update the browser version that it advertises while remaining in the general format above.

Apple Podcasts

iTMS traffic may also come from applebot.apple.com hosts, and will be identified by the following user agent:

User-Agent: iTMS

The iTMS user agent does not follow robots.txt, as it is not a general search crawler. It only crawls URLs associated with registered content on Apple Podcasts.

Customising robot.txt rules

Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot. In this example, Applebot doesn’t try to crawl documents that are under/private/ or /not-allowed/:

User-agent: Applebot Allow: / Disallow: /private/ User-agent: * Disallow: /not-allowed/

If robots instructions don’t mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.

Rendering and robot rules

Applebot may render the content of your website within a browser. If JavaScript, CSS and other resources are blocked via robots.txt, it may not be able to render the content properly. This includes XHR, JS and CSS that the page may require.

In order for Applebot to index the best content for the page, make sure everything needed for a user to render the page is available to Applebot. Alternatively, make sure the website renders cleanly, even if all of the resources are not available. This is often referred to as graceful degradation.

Customising indexing rules for Applebot

Applebot supports robots meta tags in HTML documents. To specify robots rules in meta tags, put the tags in the

... ...

Applebot also supports the following directives:

• noindex: Applebot won’t index this page, and it won’t appear in Spotlight or Siri Suggestions.

• nosnippet: Applebot won’t generate a description or web answer for the page. Any suggestions to visit this URL will only include the page’s title.

• nofollow: Applebot won’t follow any links on the page.

• none: Applebot won’t index, snippet or follow links on the page, as described above.

• all: Applebot provides the document for suggestions and snippets the contents so that a short description of the page can appear next to a representative image. Applebot may follow links on the page to provide more suggestions.

To put multiple directives in a single meta tag, use a comma-separated list or multiple meta tags.

Example:

Controlling data usage

In addition to following all robots.txt rules and directives, Apple has a secondary user agent, Applebot-Extended, that gives web publishers additional controls over how their website content can be used by Apple.

With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s foundation models powering generative AI features across Apple products, including Apple Intelligence, Services and Developer Tools.

You can add a rule in robots.txt to disallow Applebot-Extended, as follows:

User-agent: Applebot-Extended Disallow: /private/

Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.

Allowing Applebot-Extended will help improve the capabilities and quality of Apple’s generative AI models over time.

About search rankings

Apple Search may take the following factors into account when ranking web search results:

• Aggregated user engagement with search results

• Relevancy and matching of search terms to web page topics and content

• Number and quality of links from other pages on the web

• User location-based signals (approximate data)

• Web page design characteristics

Search results may use the factors above with no (pre-determined) importance of ranking. Users of Search are subject to the privacy policy in Siri Suggestions, Search & Privacy.

Contact us

If you have any questions or concerns, please contact us at applebot@apple.com.