Cloudflare accused AI search engine Perplexity of scraping websites, defying site-specific methods to block it. The incident sparked a debate over whether AI agents should be treated like bots or humans in web access contexts. Cloudflare provided evidence of Perplexity's actions by setting up a new domain with a robots.txt file blocking the AI's crawling bots, yet Perplexity accessed site content. Cloudflare's CEO likened the actions to hacking, while defenders of Perplexity argued that accessing a site on behalf of a user should be considered acceptable, equating it to human browser access.
Cloudflare's test case involved setting up a new website with a new domain that had never been crawled by any bot, setting up a robots.txt file that specifically blocked Perplexity's known AI crawling bots, and then asking Perplexity about the website's content. This demonstrated that Perplexity was able to respond even when it was explicitly blocked from accessing the site.
Matthew Prince, Cloudflare's CEO, stated that 'Some supposedly
Collection
[
|
...
]