CabanaBot
CabanaBot is the crawler for SEO for Free, an SEO service. It reads public web pages so their owners can see how to improve them.
How to recognise it
Mozilla/5.0 (compatible; CabanaBot/1.0; +https://seoforfree.online/bot)
The robots.txt token is CabanaBot.
When it visits
- Sites whose owners connected them to SEO for Free, about once a week on free plans and daily on paid plans, plus when the owner asks.
- A few pages of sites that rank for an SEO for Free customer’s keywords, or that a customer names as a rival, to compare them like for like.
- Up to eight pages of any site when someone reads it with the free reading on our home page. robots.txt is fetched first and obeyed.
It reads HTML, robots.txt and sitemaps. It never submits forms, signs in, runs scripts or downloads large files.
Everything it requests
So nothing in your logs is a surprise, a reading asks for:
/robots.txt, then the home page and a handful of pages chosen from your menu and sitemap;- your sitemaps at the usual places (
/sitemap.xml,/sitemap_index.xml,/wp-sitemap.xml…) and a sample of the addresses they list; /llms.txt,/.well-known/security.txtand/ads.txt, which are small and say a lot;- the four addresses of your home page (http and https, with and without
www), to see where each ends up; - one address that does not exist, such as
/cabana-engine-check-x7k2q9, to see whether your site answers 404 as it should; - the headers of your preview image, to check it loads.
If a firewall refuses CabanaBot, SEO for Free reports that, names the firewall, and stops. It does not retry in disguise.
Controlling it
CabanaBot obeys robots.txt rules for CabanaBot, then for *. To keep it out entirely:
User-agent: CabanaBot
Disallow: /
Questions or problems: hello@seoforfree.online.