OpenAI’s New Web Crawler

Search Engine Land » SEO » GPTBot – OpenAI’s new web crawler
OpenAI has published information about its new web crawler named GPTBot. You can read the documentation on GPTBot over here.
What is GPTBot. GPTBot is OpenAI’s web crawler, used by OpenAI to crawl the web, consume knowledge for its AI features, such as ChatGPT, and use that to provide AI-generaterd answers to your questions.
Useragent. GPTBot’s User agent token is “GPTBot” and its full user-agent string: is “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)”.
Robots.txt. You can use your robots.txt to block GPTBot from accessing all or parts of your website. To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:
To allow GPTBot to access your only parts of your site you can add the GPTBot token to your site’s robots.txt like this:
GPTBot IP ranges. OpenAI also published the IP ranges that GPTBot uses over here, it currently lists one, but I suspect they will add more over time.
Why we care. If you do not want GPTBot crawling your site and/or using your content for its purposes, then you can disallow GPTBot from crawling your site. This is the same protocol you would use to block GoogleBot, BingBot or other web crawlers.
Related stories
New on Search Engine Land
About the author
Related topics
Get the daily newsletter search marketers rely on.
See terms.
Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.
March 15-16, 2023: SMX Munich
Online Aug. 16-17: SMX Master Classes
Online Nov. 14-15: SMX Next
Start training now: SMX Advanced
Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.
April 15-17, 2020: San Jose
Build High-Impact Customer Experiences with Actionable Consumer Insights
SEO Reporting Demystified: Winning Over Executives and Influencing Strategies
Don’t Underestimate the Power of Content Management
Enterprise Marketing Attribution and Performance Management Platforms: A Marketer’s Guide
Enterprise Account-Based Marketing Platforms: A Marketer’s Guide
Enterprise Marketing Work Management Platforms: A Marketer’s Guide
2023 State of Email Workflows Report
Receive daily search news and analysis.
Topics
Our events
About
Follow us
© 2023 Third Door Media, Inc. All rights reserved.
Third Door Media, Inc. is a publisher and marketing solutions provider incorporated in Delaware, USA, with an address 88 Schoolhouse Road, PO Box 3103, Edgartown, MA 02539. Third Door Media operates business-to-business media properties and produces events. It is the publisher of Search Engine Land the leading Search Engine Optimization digital publication.

source