Microsoft Bing and Yandex new search protocol to fast crawl
Called IndexNow, the protocol uses an API to allow websites to easily notify search engines whenever content is created, updated, or deleted. Once the search engines are notified of updates they can quickly crawl and reflect the website changes in their index and search results.
It can take days to weeks for search engines to discover that the content has changed, as search engines don’t crawl every URL often. With IndexNow, search engines know immediately the “URLs that have changed, helping them prioritize crawl for these URLs and thereby limiting organic crawling to discover new content.”
“Ensuring timely information is available for searchers is critical,” Microsoft explained in its Bing Blog.
“Yet historically,” it continued, “one of the biggest pain points for website owners has been to have search engines quickly discover and consider their latest website changes. It can take days or even weeks for new URLs to be discovered and indexed in search engines, resulting in loss of potential traffic, customers and even sales.”
Microsoft maintained that IndexNow is an initiative for both a more efficient and open internet.
It explained that by telling search engines whether an URL has been changed, website owners provide a clear signal helping search engines to prioritize their crawl of those URLs. That limits the need for exploratory crawls to test if the content has changed.
Additionally, search is opened up because by notifying one search engine, a website notifies all search engines that have adopted IndexNow.
Lack of Standards
“IndexNow is a good idea because it simplifies the process of getting new content indexed,” observed Greg Sterling, vice president of market insights at Uberall, maker of location marketing solutions based in Berlin.
“It also ensures that new content will be indexed quickly — or immediately”.
Currently there’s no standard for updating search engines, explained Jon Alexander, vice president of product management at Akamai Technologies, a content delivery network service provider in Cambridge, Mass.