The first-ever Impact Week was organized by Cloudflare in the midst of the hottest summer on record. With an emphasis on environmental, social, and governance projects, we unveiled a range of products and initiatives that seek to improve the Internet and our planet. We’re pleased to provide an update on Crawler Hints, a project that was introduced at Impact Week. A service called Crawler Hints increases the efficiency of the roughly 45% of Internet traffic that originates from web crawlers and bots.
By guaranteeing that crawlers receive information about what they have already crawled and whether it makes sense to crawl a website again, Crawler Hints delivers this efficiency boost.
We’re thrilled to share two Crawler Hints updates with you today:
The first: Crawler Hints now supports IndexNow, a new protocol that allows websites to notify search engines whenever content on their website content is created, updated, or deleted. By collaborating with Microsoft and Yandex, Cloudflare can help improve the efficiency of their search engine infrastructure, customer origin servers, and the Internet at large.
The second: Crawler Hints is now generally available to all Cloudflare customers for free. Customers can benefit from these more efficient crawls with a single button click. If you want to enable Crawler Hints, you can do so in the Cache Tab of the Dashboard.
What issue is resolved by Crawler Hints?
The Internet operates with the aid of crawlers. Crawlers are automated services that search the Internet for, well, anything their programming instructs them to seek for. Search engines and similar services run vast networks of bots that scour the Internet to find the content that is most pertinent to a user query in order to power experiences that depend on indexing content from across the web. Search engine crawlers, however, have a Sisyphean chore because web material is constantly changing and there is no single clearinghouse for when these changes occur on websites. They must constantly browse the Internet, speculating how frequently they ought to examine a specific site for content changes.
Companies that operate search engines have worked hard to optimize the process, pushing infrastructure efficiency and crawl cadence to new heights. But there is still one area that is obviously wasteful: excessive creep.
We have spent the last year researching how frequently these bots return to a page that hasn’t changed since they last saw it at Cloudflare, where we see traffic from all the major search crawlers. Each of these trips is a waste of time. Sadly, based on our assessment, 53% of this crawler traffic is useless.
We hope that by giving the individuals who operate these crawlers an additional heuristic, Crawler Hints will help to make this chore a little bit more manageable. Instead of depending on preferences or past modifications that might not accurately reflect the change cadence for a site, this will enable them to know when material has been added to or updated on a site. In order to improve customer experience and decrease the need for repeated crawls, Crawler Hints tries to raise the proportion of relevant crawls and restrict crawls that don’t find fresh content.
Because of its special positioning on the Internet, Cloudflare can help inform spiders when to revisit a site. When Cloudflare can alert your crawler when it’s the best moment to index fresh or altered material, there’s no need to check a website’s door every 30 seconds to see if anything has changed. That’s a brief summary of Crawler Hints.
Collaborating with leaders in the industry
Being able to have a significant chunk of the Internet proxied behind us puts Cloudflare in a special position. We can so identify patterns in how bots access web pages. Because of this visibility, we can be proactive in indicating which crawls are necessary and which are not. To make these insights valuable to our clients, we are eager to collaborate with partners. The most important elements in this equation are search engines. With Microsoft, Bing, and Yandex, we are pleased to work together and share this vision of a more effective Internet. With some early achievements, we have been testing our interactions with Bing and Yandex through IndexNow for months.
This is only the start. To increase Internet efficiency more broadly, Crawler Hints is a continual effort that will involve collaborating with an increasing number of partners. While this may require some time and involvement from other significant industry players, we are willing to work with any interested party whose user experiences rely on crawling.
“The cache data from CDNs is a really valuable signal for content freshness. Cloudflare, as one of the top CDNs, is key in the adoption of IndexNow to become an industry-wide standard with a large portion of the internet actually using it. Cloudflare has built a really easy 1-click button for their users to start using it right away. Cloudflare’s mission of helping build a better Internet resonates well with why I started IndexNow i.e. to build a more efficient and effective Search.”
– Fabrice Canel, Principal Program Manager
“Yandex is excited to join IndexNow as part of our long-term focus on sustainability. We have been working with the Cloudflare team in early testing to incorporate their caching signals in our crawling mechanism via the IndexNow API. The results are great so far.”
– Maxim Zagrebin, Head of Yandex Search
“DuckDuckGo is supportive of anything that makes search more environmentally friendly and better for end users without harming privacy. We’re looking forward to working with Cloudflare on this proposal.”
– Gabriel Weinberg, CEO and Founder
How do Cloudflare users gain from this?
Not just search engines gain from crawler hints. Crawler Hints will guarantee that search engines and other bot-powered experiences always get the most recent version of your material, which will lead to happier users and eventually have an impact on search rankings for our clients and origin owners. Less traffic hitting your origin thanks to crawler hints will improve resource use. Additionally, since bots won’t be competing for your real clients, the performance of your website will also be enhanced.
What about Internet users? Because Cloudflare has informed the owners of the bots when to update their results, whether you interact with bot-fed experiences like pricing tools or search engines—which we all use on a daily basis whether we realize it or not—these will now provide more helpful results from crawling data.
What is IndexNow?
The IndexNow standard was created by the Yandex and Microsoft search engines. The standard seeks to give search engines and other crawlers an effective way to communicate when they should crawl information. IndexNow is now supported by Crawler Hints on Cloudflare.
In its most basic form, IndexNow is just a ping to let search engines know when a URL and its contents have been added, changed, or removed. This allows search engines to update their search results right away.
Visit www.indexnow.org
When you enable Crawler Hints on your website, Cloudflare will take care of notifying these search engines using the IndexNow protocol whenever your content changes. Nothing further needs to be done.
What does this imply for those who run search engines? You can receive a pushed feed of change events for Cloudflare websites in almost real-time using Crawler Hints (that have opted in). As a result, both the quality of your findings and the energy economy of running your bots will be significantly improved.
How can I enable Crawler Hints for my website in Cloudflare?
Crawler Hints is free to use for all Cloudflare customers and promises to revolutionize web efficiency. If you’d like to see how Crawler Hints can benefit how your website is indexed by the worlds biggest search engines, please feel free to opt-into the service:
- Sign in to your Cloudflare Account.
- In the dashboard, navigate to the Cache tab.
- Click on the Configuration section.
- Locate the Crawler Hints sign up card and enable. It’s that easy.
When you enable it, we’ll start giving search engines cues about when to crawl specific pages of your website. Crawler Hints has the potential to significantly increase the effectiveness of the Internet.
What’s next?
We’re excited to work with market leaders Microsoft, Bing, and Yandex to integrate IndexNow into Crawler Hints and make the service generally available to a large audience. In order to increase the effectiveness of this procedure for the entire Internet, we look forward to cooperating with new businesses that operate crawlers.