AI Scoring Methodology & Retrieval Flow
Learn the mathematical equations, query routing diagrams, and crawler rules that determine your brand\'s visibility score inside language models.
Scoring Formula & Weight Equations
The final Visibility Score is calculated dynamically on client-side requests:
Score = (CrawlStatus * 0.40) + (SchemaCompleteness * 0.30) + (SemanticDensity * 0.30)
CrawlStatus (40% Weight): Direct parsing of robots.txt exclusions. Blocking GPTBot or ClaudeBot triggers an immediate 0/40 crawl factor penalty.
SchemaCompleteness (30% Weight): Validates sameAs links and Organization graph declarations. Evaluates if search crawlers can resolve entity identity connections.
SemanticDensity (30% Weight): Measures structural landmarks (H1/H2 tags), data tables, lists, and copy depth.
Retrieval-Augmented Generation (RAG) Flow
The chronological routing map of how conversational AI retrieves and cites web content during user prompts:
[User Prompt Input]
↓
[Vector Embedding Query]
↓
[Database / Search Index Lookup] ---> [Excludes Disallowed Crawl URLs]
↓
[Re-ranking & Segment Selection] ---> [Prioritizes Schema Entity Matches]
↓
[Response Synthesizer]
↓
[Footnote & Citation Card Generation]
AI Crawler Bot Exclusion Reference
Verified crawler parameters and block behaviors for generative search agents:
| User-Agent | Publisher | Robots.txt Directive | Drift / Block Behavior |
|---|---|---|---|
| GPTBot | OpenAI | User-agent: GPTBot | Disallowing blocks future model generations |
| OAI-SearchBot | OpenAI | User-agent: OAI-SearchBot | Disallowing blocks immediate conversational citations |
| ClaudeBot | Anthropic | User-agent: ClaudeBot | Disallowing restricts context window retrieval |
| PerplexityBot | Perplexity AI | User-agent: PerplexityBot | Disallowing excludes footnotes placement eligibility |
| Google-Extended | User-agent: Google-Extended | Does not block standard AI Overview searches |
Check Your Site's AI Citations Compatibility
Run a real-time crawl audit and review immediate structured schema recommendation logs.
Explore Topic Cluster
Audit entity maps and ranking checkups within our cluster.