Microsoft has confirmed that Reddit has blocked access to alternative search engines. Updates to Reddit’s robots.txt file stopped Bing from crawling the site. This follows Reddit’s deal with Google.
Explanation from Microsoft on Reddit’s blocking of Bing
A recent study has revealed that alternative search engines are no longer able to display Reddit results. This follows a comprehensive agreement between Google and the social media platform.
Microsoft has confirmed that Bing is indeed being blocked by Reddit. In a statement to Search Engine Land, a Microsoft representative said: “We respect the robots.txt standard. Bing stopped crawling Reddit on July 1 after Reddit implemented an updated robots.txt file.”
The robots.txt protocol allows websites to configure which pages a web crawler can or cannot access. These changes are non-binding and actually work based on an honor system.
When a website updates its robots.txt file so that certain web crawlers and search engines cannot access it, there are no technical barriers preventing those crawlers from ignoring this request. Microsoft agreed to Reddit’s request in this case, but it was recently revealed that a crawler linked to the Perplexity AI search engine ignored a publisher’s requests.
OpenAI and Reddit have taken a significant step in the AI world by integrating Reddit content into ChatGPT. Here are the details...
Reddit’s request came a month after the company first confirmed its plans to update the robots.txt file. The company said it had experienced an increase in “clearly commercial entities scraping Reddit”. Reddit’s change follows a deal it struck with Google earlier this year that would allow the platform to use its user-generated content for AI training.
In a statement to The Verge, a Reddit representative said the decision to block alternative search engines had nothing to do with the Google deal: “This decision is not related to our recent collaboration with Google. We have been in discussions with many search engines and were unable to reach an agreement with all of them because some were unwilling or unable to make enforceable promises about the use of Reddit content.”
What do you think about the situation? Please share your thoughts with us in the comments section below.
