A new open licensing protocol called the RSL Standard is giving web publishers more control over how AI companies use their content. Backed by Reddit, Quora, Yahoo, and others, the initiative aims to reshape the economics of online content in the AI era.
RSL Standard adds licensing to robots.txt
For decades, websites have used the robots.txt file to tell bots where they can and can’t go. The RSL Standard builds on that idea but adds a critical layer: licensing terms. Now, site owners can require payment from AI companies that crawl their pages for training data.
They can choose from several pricing models, such as pay-per-crawl, monthly subscriptions, or even per-inference charges when AI models generate text based on a publisher’s content.
Major platforms join the push for AI accountability
The RSL Collective, the group behind the standard, includes influential media and tech brands. Its founders, Eckart Walther and Doug Leeds, say they designed the system to offer a scalable way for any site, not just giants like The New York Times, to monetize AI access.
Current supporters include:
- Quora
- Medium
- wikiHow
- O’Reilly
- Ziff Davis
- Yahoo
- People Inc.
This group hopes that collective pressure will encourage AI companies to follow the rules rather than ignore them.
Fastly to help enforce the RSL Standard
To ensure AI crawlers follow the rules, the RSL Collective is working with Fastly, a content delivery network. If a bot hasn’t agreed to the licensing terms, Fastly can block it from accessing the site. This adds technical enforcement to what was once just a polite request in robots.txt.
Sites that don’t use Fastly can still declare their terms, but they’ll need additional tools to fully enforce them.
The RSL Standard may define a new web economy
While legal enforcement remains uncertain, especially with AI scraping laws still evolving, the RSL Collective believes its system will shift the balance. It functions like a digital rights group for the web, similar to how ASCAP protects music creators.
By combining technical controls, collective pressure, and standardized licensing, the RSL Standard gives online publishers a long-overdue tool: the power to say, “You can’t use our work for free.”
