google-site-verification: googlebc47d07320294fb4.html

Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies

Google is exploring alternatives or supplemental ways of controlling crawling and indexing beyond the 30 year standard of the robots.txt protocol. “We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases,” Google wrote.

Engaging with the community. Google said it is inviting members from the web and AI communities to discuss in a new protocol. Google said it is “kicking off a public discussion,” with a “broad range of voices from across web publishers, civil society, academia and more fields from around the world.

Timing. Google said these discussions are happening over ” to join the discussion, and we will be convening those interested in participating over the “the coming months.” So nothing is happening too soon and nothing is changing tomorrow.

Paywalled content issue. Recently, Open AI disabled the browse with Bing feature in ChatGPT after it was able to access paywalled content without publisher permission. This is one of the many reasons why maybe Google is looking for alternatives to the robots.txt protocol.

Why we care. We have all become accused to allowing bot access to our websites by using robots.txt and other forms of newer structured data. But we may be looking at new methods in the future. What those methods and protocols may look like is unknown right now but the discussion is happening.

The post Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies appeared first on Search Engine Land.

Original source: https://searchengineland.com/google-to-explore-alternatives-to-robots-txt-in-wake-of-generative-ai-and-other-emerging-technologies-429123

12 thoughts on “Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies”

  1. It’s truly very complex in this full of activity life to listen news on TV, thus I only use world wide web for that purpose, and get the most up-to-date news.

  2. Hi there, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of spam responses? If so how do you reduce it, any plugin or anything you can advise? I get so much lately it’s driving me mad so any assistance is very much appreciated.

  3. We are a group of volunteers and starting a new scheme in our community. Your site provided us with valuable information to work on. You have done an impressive job and our whole community will be grateful to you.

Leave a Reply

Your email address will not be published. Required fields are marked *

+ +