Advanced Search

SEO for AI User Agents New
By
27 December 2024 (Edited )
Share this article:

Behaviors characteristic of AI user agents: what to know

SEO for AI User Agents

A new blog post by web platform Vercel reveals some characteristic behaviors of AI user agents (web crawlers) like OpenAI's GPTBot and Anthropic's Claude, based on 900M+ requests by these agents across their network.

In aggregate, requests by these AI agents represent about 28% of Googlebot's total volume.

Key findings:

  • Major AI agents don't render JavaScript.
  • They tend to be inefficient, spending 30+% of their requests on 404 pages.
  • They prioritize HTML and images over other content.
  • Their operations are heavily US-biased.

Implications for site owners:

  • Because AI agents don't render JavaScript, they don't see anything that's rendered client-side. Render critical content server-side.
  • Use static pages where possible.
  • Make sure navigation, meta information and main content (including images) are present in HTML when pages open.
  • Use descriptive ALT text for images.
  • Use correct header tag hierarchy: H1,H2, etc.
  • Keep your sitemap updated.
  • Do redirects correctly: no excessively long or broken chains.
  • Use a logical, consistent URL structure.
  • Stay on top of, and fix, 404 errors.

Comment: Happily, none of these recommendations conflict with what you should be doing for Google. Implement now.

Share this article:

And if you have questions or comments, you can easily send them to me with the Quick Reply form, below, or send me an e-mail.


David Boggs MS    - David
David@DavidHBoggs.com
View David Boggs's profile on LinkedIn

Google Certifications - David H Boggs
View my profile on Quora
Subscribe to my blog

External Article: https://vercel.com/blog/the-rise-of-the-ai-crawler


Subhead Behaviors characteristic of AI user agents: what to know
Rating
5/5 based on 1 vote.
Show Individual Votes
Related Listings

Sorry, you don't have permission to post comments. Log in, or register if you haven't yet.

Please login or register.

Members currently reading this thread: