An AI Agent Is Now a Normal Visitor. Here's What I Changed on My Sites.2026-09-18
Google-Agent ignores robots.txt, so the server log is the only place you'll see it. I pulled 14 days of logs from both my sites, found 30,000 AI crawler hits and zero Google-Agent, and changed five things anyway.