by GelFuzion
Home / SEO & Meta / Robots.txt Generator
05.03SEO & Meta

Robots.txt Generator

Set a default policy, add path rules, decide which AI crawlers get in, then test any URL against the result using the same longest-match logic Google uses.

Updated
Default for all crawlers
Bing and Yandex honor this. Google ignores it.
Path rules

Use * for any sequence of characters and $ to anchor the end, e.g. /*.pdf$.

AI crawlers to block
    robots.txt 
    

            
    
              Test a URL against these rules
              
    
                
    
              
    
                
                
    
              
    
            
    
          
    
        
    
      
    
      
    
    
    

  
    
    
    
      

      

      
    
        
    How robots.txt works
    
        
    A robots.txt file is a plain text file at the root of your domain that tells well-behaved crawlers which URLs they may request. It is grouped by User-agent: each group names one or more crawlers and lists Allow and Disallow rules for them. A crawler looks for the group that names it most specifically, and if none does, it falls back to the User-agent: * group.
    
        
    That fallback catches people out. If you add a group for Googlebot, Googlebot stops reading the * group entirely. Any rule you want it to follow has to be repeated inside its own group.
    

        
    Longest match wins
    
        
    When several rules match a URL, the most specific one (the longest path pattern) decides. If an Allow and a Disallow are the same length, Allow wins. That is why this classic WordPress pair works:
    
        
    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
    
        
    /wp-admin/admin-ajax.php matches both rules, but the Allow is longer, so front-end features that call it keep working for crawlers that render your pages. Try it in the tester above. Two wildcards are supported by Google, Bing and the major AI crawlers: * matches any run of characters and $ pins the end of the URL. Disallow: /*?sort= blocks any URL containing that parameter; Disallow: /*.pdf$ blocks PDFs but not /pdf-guide/.
    

        
    AI crawlers: training vs. search vs. user fetches
    
        
    The AI section above is where most people make the wrong call, because "AI bots" are really three different jobs:
    
        
      
          
    • Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent) collect pages in bulk to train future models. Blocking them keeps your content out of training sets going forward. It does not remove anything already collected, and it has no effect on search rankings.
      • 
          
    • Control tokens (Google-Extended, Applebot-Extended) are not separate crawlers at all. Googlebot and Applebot still crawl your site for search. These tokens only tell Google and Apple whether that content may be used for their AI models. Blocking them is the safe way to opt out of Gemini or Apple Intelligence training without touching your Google or Siri visibility.
      • 
          
    • Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes that AI assistants cite when they answer questions with links. Block these and your pages stop appearing as sources in ChatGPT search, Claude or Perplexity answers. For most businesses that is traffic worth keeping.
      • 
          
    • User-triggered fetchers (ChatGPT-User, Claude-User) visit a page only because a person pasted the link or asked the assistant to look at it. They act like a browser on someone's behalf, and some vendors note these fetches may not be governed by robots.txt the same way.
      • 
        
    
        
    A sensible default for a business site is the Training only button: opt out of model training, stay visible in AI search. Publishers who license content may want everything blocked; documentation sites that want to be quoted may want nothing blocked.
    

        
    What robots.txt cannot do
    
        
    It is a request, not a lock. Reputable crawlers honor it; scrapers do not, so private files belong behind authentication. It also does not deindex pages. A disallowed URL can still show up in Google with no description if other sites link to it. To keep a page out of results, allow crawling and add <meta name="robots" content="noindex">. And never disallow your CSS or JavaScript folders: Google renders pages like a browser, and blocking assets can make a page look broken to it.
    

        
    Deploying
    
        
    Upload the file as robots.txt to your site root so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own. After uploading, check the robots.txt report in Google Search Console to confirm Google fetched the new version.
    

        
    Frequently asked questions
    
        
    
          
    Does Disallow in robots.txt remove a page from Google?
    No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To remove a page, let Google crawl it and add a noindex meta tag or X-Robots-Tag header.
    
          
    Will blocking GPTBot or ClaudeBot hurt my Google rankings?
    No. Those are separate crawlers run by OpenAI and Anthropic. Google Search uses Googlebot, and Google-Extended only controls whether your content is used for Gemini, not whether you rank.
    
          
    Where does robots.txt go?
    At the root of each host, for example https://example.com/robots.txt. Subdomains need their own file, and so does each protocol and port combination.
    
          
    Do all bots obey robots.txt?
    Reputable crawlers do. Scrapers and malicious bots ignore it, so never rely on robots.txt to protect private content. Use authentication or server-level blocking for that.