An AI Agent Is Now a Normal Visitor. Here's What I Changed on My Sites.

Full disclosure: this post was drafted by AI from my own server logs and notes, then checked and published under my name. The numbers are from my two sites' nginx logs for September 4–18, 2026. Nothing in here is estimated.

Andrew Yan, ex-Google Search and co-founder of AthenaHQ, posted a breakdown of something Google did quietly in March: it gave AI agents their own identity. When a person asks an AI on Google infrastructure to open a web page, the request now arrives as Google-Agent, with its own user agent string and its own published IP range file.

Three things make it different from Googlebot:

It only fires when a human asks. No background crawling.
It has a separate IP range file from the rest of Google.
It ignores robots.txt. Google's own docs say user-triggered fetchers "generally ignore robots.txt rules" because a person requested the fetch. OpenAI's ChatGPT-User works the same way. Anthropic's Claude-User, for what it's worth, does respect robots.txt.

Andrew's advice was five things: filter your logs for Google-Agent, allow the published IP ranges in your CDN or firewall, audit your forms so an agent can actually use them, publish an llms.txt, and tighten your Schema.org data. He calls the category ASO, agentic search optimization, after SEO, AEO and GEO.

I run two sites, this one and wateradvantage.org, on one Linux box with nginx in front. So I did the five things. Here is what the logs said first.

What 14 days of logs actually show

wateradvantage.org had 74,628 requests. Forty percent of them were AI training crawlers:

ClaudeBot (Anthropic): 17,044 hits from 10 IPs
GoogleOther (Google's research and AI crawler): 11,363
GPTBot (OpenAI): 983
Bytespider (ByteDance): 725

Real people asking an AI to open a page, the user-triggered kind: 326. ChatGPT-User made 260 of those, from 198 different IP addresses. That's 260 moments where someone typed a question into ChatGPT and it went and read one of my water reports for them. DuckAssistBot made 65. Claude-User made one, and that one turned out to be a developer running Claude Code on their own laptop, which identifies itself as Claude-User but doesn't come from Anthropic's IP ranges.

Google-Agent: zero. Not one hit on either site. It's been in Google's crawler list since March. It is not a wave yet.

jacobthorwolf.com had 17,931 requests, zero user-triggered agents, and the usual training crawlers in the low hundreds.

Two things I didn't expect:

On September 6, ClaudeBot hit wateradvantage.org at close to 590 requests a minute for three hours straight, mostly on the JSON API behind the report pages. The site returned 27 errors that afternoon, the only errors it returned in two weeks. Nobody noticed, because nobody was watching that log.
Six "Googlebot" requests came from IPs that are not in Google's published ranges. Fake Googlebot. Small, but I now know it's there.

What I changed

1. The logs are watched now. Neither of my analytics tools could see any of this. GoatCounter drops known bots on purpose, and the tracker script never runs inside a fetcher. So I wrote a classifier that reads the nginx logs every hour, sorts every hit into user-triggered agent, AI search indexer, AI training crawler, or search engine, and checks each one's IP against the range file its vendor publishes: Google's, OpenAI's, Anthropic's, Bing's. It writes a JSON per site, and both sites' admin dashboards read it. The day Google-Agent shows up, I'll see it that hour, verified or spoofed.

2. Nothing blocks them. There's no CDN and no bot wall in front of either site, so there was nothing to whitelist. The one trap I run bans IPs that probe for PHP files and WordPress admin paths, which no legitimate agent has ever hit. Rate limits exist only on the chat endpoint and the data API. I did add one thing in the other direction: a crawl-delay for ClaudeBot on wateradvantage.org, which Anthropic honors, so the next 590-a-minute afternoon doesn't happen.

3. The forms work without JavaScript. Both contact forms were React forms that posted JSON with fetch. A person in a browser never noticed. A fetcher like ChatGPT-User, which doesn't run JavaScript, saw a form it couldn't submit. Now every field has a proper name and autocomplete attribute, the form has a real action and method, and the API accepts a plain form post and redirects back with the result. Same form for humans, JavaScript or not.

4. llms.txt, rewritten, plus the full text. Both sites had an llms.txt from April that was already out of date: wrong project names, wrong system counts, missing whole sections of the site. They're current now. This site also serves /llms-full.txt, the About page, the project list and every published post as one text file, so an assistant that wants everything can get it in one request instead of sixty.

5. Structured data that describes one person, not five. The site had a Person entity on the home page, another on the about page, another on the contact page, none of them linked. Now there's one Person with a stable identifier, an Organization for Bottleless Nation, and a WebSite node, and every blog post's BlogPosting points at that Person as author. WaterAdvantage got a SearchAction so an agent knows the search URL pattern, and links between the two sites' Organization and Person records.

What I think this means

The numbers say training crawlers are the traffic and user-triggered agents are the signal. Thirty thousand hits from ClaudeBot and GoogleOther are a data center reading my site for its own reasons. Three hundred hits from ChatGPT-User are three hundred people who asked a question and got an answer sourced from a page I built. That second number is the one I care about, and it's the one that was invisible until this week.

Andrew's framing is right: this is the next optimization frontier, and most of it is unglamorous. Check your logs. Make your forms work without JavaScript. Tell the machines what the site is. The agents are coming to your site whether you've done that or not; the only question is whether they can do anything useful when they get there.