SEO for Free / AI search visibility
AI search visibility
Can AI assistants see your website?
More people now ask an AI assistant before they search. If its crawler is blocked, or your pages are hard to read, you are not in the answer. SEO for Free shows which AI crawlers your site lets in, page by page, and what to change.
The AI crawler matrix
Your robots.txt is read the way each crawler reads it, including the groups and wildcards that catch people out. You see, for every AI crawler we track, whether it may read your home page and your key pages, and which rule decided it.
- Search and answer crawlers: OAI-SearchBot, PerplexityBot, Google’s and Bing’s
- Training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot and others
- The exact rule that allows or blocks each one
Being understood, not just seen
AI assistants quote pages they can parse. SEO for Free checks for a clear title and description, headings that say what the page is about, structured data that names your business, and an llms.txt file: a short guide to who you are and your key pages. It can draft the llms.txt for you.
Search and AI are separate choices
You may want to appear in AI answers but not be used for model training, or the reverse. The matrix separates the two, so you can allow search crawlers and refuse training ones on purpose rather than by accident.
Questions
How do I know if ChatGPT can access my site?
Type your address on our home page. The reading lists OAI-SearchBot and GPTBot among the AI crawlers, with whether your robots.txt allows each one and the rule that decides it.
What is llms.txt?
A plain text file at /llms.txt that tells AI systems who you are, what you offer and which pages matter most. SEO for Free checks for it and drafts one from your own pages.
Will blocking GPTBot remove me from ChatGPT search?
Not necessarily. OpenAI uses OAI-SearchBot for search and GPTBot for training. You can allow one and block the other; the matrix shows each separately.
Related
See your site the way Google does.
Free reading of any website with no sign-up; free trial of the console.