AI Web Crawlers | To Crawl or Not to Crawl?
A new frontier has emerged in the realm of SEO with the introduction of dedicated AI web crawlers. In August 2023, OpenAI rolled out GPTBot, a specialised crawler that can be managed via robots.txt—much like how one might restrict Googlebot from accessing certain areas of a website. Recent studies indicate that nearly half of websites in some sectors have taken advantage of this capability. Website…