Cloudflare has published findings that Perplexity, an AI tool, is not respecting website requests to refrain from content crawling. Perplexity is accused of spoofing its identity to bypass robots.txt file directives meant to indicate website indexing preferences. This behavior demonstrates a shift from ethical, voluntary agreements on the web to aggressive business practices prioritizing commercial goals over moral considerations. According to cybersecurity expert Eerke Boiten, the fading ethos of cooperation online indicates a loss of respect for ethical data acquisition by major AI companies.
Cloudflare claims Perplexity, an AI-powered "answer engine," is overriding website requests not to crawl their content by spoofing its identity to hide that the requests are coming from an AI company.
The fundamental idea was simple: Act ethically and morally. If someone asked you to stop doing something, you stopped—or at least considered it.
Eerke Boiten states, "The code of honor around crawling and robots.txt files is a charming remnant from when the web was collaborative and based on community standards."
Boiten believes the sense of ethical cooperation online is fading fast, noting that many large AI companies show little regard for where or how they obtain their training data.
Collection
[
|
...
]