Some people are defending Perplexity after Cloudflare ‘named and shamed’ it
Cloudflare accused Perplexity of stealthily scraping websites that explicitly blocked its crawler by masquerading as Chrome and ignoring robots.txt. Perplexity and its supporters argue its AI agents merely fetch public content on users’ behalf—like a browser—not as rogue bots. Cloudflare champions a new Web Bot Auth standard to differentiate good AI traffic, while the debate highlights a growing clash between AI assistants’ access to information and websites’ control over their content.