Applebot – Apple’s new web crawler


May 24th, 2015

With the announcement by Apple regarding the Applebot, the long rumored discussion about the existence of Apple’s web crawling service has gotten a direction. Apple has published a support document. The document provided details about Applebot’s operations. Siri and Spotlight, which offer web search capabilities, will be served with an Apple’s web crawler Applebot. Up till now it was thought that Google or Bing search engines were used by Siri and Spotlight in IOS and OS X platforms. Google’s Googlebot and Apple’s Applebot have similar functionalities. In cases where Applebot does not find Applebot-specific instruction, it follows Googlebot’s instructions. Applebot will follow customary robots.txt rules and robots Meta logs.

Applebot is a web crawler from Apple, which had existence for some time now; however official support document is released in February, which was updated with proper instructions last month. Let’s first understand what a web crawler is. A web crawler also called as a Web Scutter is used for systematically browsing World Wide Web for creating an index of the web or the internet pages scanned. A web crawler is basically a script that scans the internet pages and creates an index of the data. Web crawlers are used to collect information on the internet. For example, search engines mostly use crawlers to collect the information. With the vast number of data added over the internet, web crawlers help search engines to keep their database up to date.

Web crawlers also known as spiders, scan internet pages and check what words they contain and where those words are used. The crawler stores its findings into a giant index. The index is basically a list words and scanned internet pages. When we type a keyword in search engine, the search engine checks its indexes and returns the list of web pages. These web crawlers keep scanning regularly and make sure their index database is in sync with the latest data on web. Google bot works on the same concept. It scans internet pages and passes it on to Google indexer. Google indexer maintains an index of alphabetically sorted web pages in its database.

Previously Apple had a partnership with Google and Bing to provide search results in products like Siri. Google and Bing have their own search engines which are at the moment the top search engines used by everyone. However with the launch of Applebot, Apple can consider launching its own search platform. Apple has not confirmed anything on that part yet though.

Apple has mentioned on Apple Insider that Applebot will be used in products like Siri and Spotlight suggestions in IOS and OSX. It was known that Siri and Spotlight use Bing and Wolfram Alpha or its searches. Applebot will be used for Siri and Spotlight suggestions. Apple has mentioned the details about how Applebot operates. These are just very few details. It operates on netblock. It is also mentioned on Apple Insider that if robots do not find instructions of Applebot it will follow Googlebot instructions. Applebot will follow customary robots.txt rules and robots Meta logs.

Applebot was first noticed by developer Jan Moesen, who found that there are large number of web hits from a crawler indexing from a netblock This IP block is wholly owned and operated by Apple. This information along with the news about the Apple and Google search engine deal for Safari led to a lot of discussions about what is going to be next search engine for Safari.

With the news from Apple about Applebot new speculations have now started about Apple’s own search engine. It is still not clear that this web crawler from Apple will be the base for setting up its own search engine brand or just trying to act as an additional support for other platforms for Apple’s Siri and Spotlight services.

Is Apple Actually planning to develop its own search Engine? The question remains unanswered. There are still a lot of discussions and speculations in the market about Apple planning to launch its own search engine. With the wordings mentioned on Apple Insider, it does give a clear picture. If Applebot is designed to improve the existing search by managing it’s indexing on its own without having to be managed by Google or Bing. Currently, it seems that Applebot is an add-on to existing search engines to make sure Siri and Spotlight are delivering best results. Currently Google and Bing are in competition for Default Search Engine on Apple. However, you never know if Apple has plans to Add to this competition with its own search engine.


Leave a Reply

© 2006-2017 Security Magazine.