Before an AI assistant can mention your company, it needs to be able to read your website. This sounds like it should just work, but there are several ways you might be accidentally blocking AI systems from accessing your content.
Some companies block AI crawlers intentionally because they are worried about their content being used to train AI models. That is a valid concern, but if you block these crawlers, you are also preventing AI assistants from learning about your company and recommending you to potential customers.
Check your robots.txt file
Your website has a file called robots.txt that tells web crawlers which parts of your site they can access. You can see yours by going to yoursite.com/robots.txt.
AI companies use specific crawler names (called user agents) to identify themselves. Common ones include GPTBot for OpenAI, ClaudeBot for Anthropic, and PerplexityBot for Perplexity.
Look through your robots.txt file and see if any of these are listed under "Disallow." If they are, those AI crawlers are being blocked from your site.
Here is what it might look like if you are blocking AI crawlers:
User-agent: GPTBot
Disallow: /
That means OpenAI's crawler cannot access anything on your site. If you want AI assistants to be able to mention you, you should remove these rules.
Check if your content is actually in your pages
Some websites are built in a way where the content only appears after JavaScript runs in the browser. This is common with single-page applications and certain modern web frameworks.
The problem is that not all crawlers run JavaScript. If they visit your page and your content is not in the initial HTML, they will not see it.
To check this, visit one of your pages and right-click, then select "View Page Source" (not "Inspect"). This shows you the raw HTML that gets sent to crawlers. Look for your actual content. Is the text there? Or is it just empty containers waiting to be filled by JavaScript?
If your main content is not in the page source, you might want to consider server-side rendering for your important pages. This means the server sends complete HTML to the browser instead of having the browser build the page from scratch.
Check for bot blocking
If you use Cloudflare, a Web Application Firewall, or any kind of bot protection, you might be accidentally blocking legitimate AI crawlers.
These tools often try to distinguish between real users and bots, and they sometimes block bots that are actually useful. Check your security settings and see if there are options to allow known AI crawlers.
Some services have specific settings for this. Cloudflare, for example, has options related to AI bot access that you can configure.
A simple test
Ask an AI assistant about your company or your specific product. Try ChatGPT, Claude, and Perplexity with browsing enabled. Ask something like "What does [your company name] do?" or "Is [your company name] good for [your use case]?"
If the AI can give you accurate, up-to-date information about your company, it can probably access your site. If it gives outdated information or says it does not know anything about you, something might be blocking it.
This is not a perfect test since AI assistants also rely on their training data, but it can give you a rough sense of whether they can find current information about you.