Who’s Crawling Your WordPress Site? Hands On with Known Agents

AI bots are browsing websites at an incredible rate. From search engine indexing bots and LLM training scrapers to autonomous shopping agents, non-human traffic now accounts for a massive chunk of daily web visits.

If you’ve just kicked off a trial with the Known Agents plugin (formerly Dark Visitors), you’re about to get a clear, real-time look into exactly who—or what—is inspecting your content.

What Known Agents Tracks on Your Site

The plugin connects your WordPress site to the Known Agents database—tracking thousands of distinct bots across 17 categories—to give you full visibility into server activity:

• AI Data Scrapers: Crawlers like GPTBot or ClaudeBot hoovering up content to train future AI models.
• AI Search & Retrieval Agents: Crawlers from Perplexity, ChatGPT, Gemini, or Meta AI fetching live content to answer immediate search queries.
• Task & Shopping Agents: Autonomous bots navigating page layouts, inspecting products, or completing automated user journeys.
• Bad Actor Spoofers: Rogue scrapers attempting to disguise themselves as official crawlers like Googlebot or Bingbot.
• LLM Referral Traffic: Tracks real human visits originating from citations inside platforms like ChatGPT, Claude, and Perplexity (Generative Engine Optimisation / GEO).

What to Check During Your Trial

• Identify Your Top Bot Destinations: Check your dashboard to see which specific URLs or documentation pages are attracting the heaviest AI traffic.
• Monitor Server Performance: Watch out for unannounced crawling spikes that might be consuming unnecessary server bandwidth and slowing down human visitors.
• Manage Dynamic Access: Use the automatic robots.txt generator to block entire categories of bots (like aggressive scrapers) without affecting search engine indexing.
• Track AI Referral Conversions: Review which AI chat platforms are citing your content and driving click-through traffic from real people.

I’m David Saunders and I’m on a trial with Known Agents. I added the plug in to my WordPress website I will let you know in a month and / or as the exciting stuff happens! I did note Bingbot was crawling away! Wow!

Top Tip: Avoid blanket-blocking all AI bots straight away! While training scrapers take without giving back, AI search crawlers actively cite your pages and send human readers your way.