`/`<h1>`. 3. **Site Architecture & Link Equity (15%):** Click depth distribution, internal PageRank (`LinkScore`), orphan page detection. 4. **Performance & TTFB (15%):** Server response latency and HTML payload efficiency. 5. **Structured Data & Schema.org (10%):** Embedded `application/ld+json` entity validation. 6. **AI Search & GEO Readiness (15%):** Live bot parity across `Googlebot`, `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, and `PerplexityBot`, SSR raw HTML word count, and `/llms.txt` + `/llms-full.txt` grounding presence. ## 3. Cloud & Local MCP Server Integration - **Discovery Endpoint:** `GET https://gocrawl.tech/api/mcp` - **Tools List Endpoint:** `GET https://gocrawl.tech/api/mcp/tools` - **JSON-RPC 2.0 Endpoint:** `POST https://gocrawl.tech/api/mcp` - **Local Stdio Mode:** `go-crawl -mcp -db gocrawl.db` ## 4. Complete Directory of 20 Interactive Diagnostic Question Pages ### 1. Is ChatGPT Blocked From Crawling My Website? - **URL:** https://gocrawl.tech/q/is-chatgpt-blocked-from-my-website - **Category:** AI Search & Bots - **Summary:** Test live if OpenAI's GPTBot and OAI-SearchBot can crawl your website or if Cloudflare, WAF rules, or robots.txt are blocking ChatGPT Search citations. - **Direct Answer:** ChatGPT relies on two user-agents—OAI-SearchBot (for live search citations) and GPTBot (for model grounding)—to read your site. If your CDN, WAF, or robots.txt returns HTTP 403 or Disallow to these bots while allowing Googlebot, your pages will be omitted from ChatGPT Search answers. ### 2. Why Is Perplexity or Claude Not Citing My Website? - **URL:** https://gocrawl.tech/q/why-is-perplexity-not-citing-my-website - **Category:** AI Search & Bots - **Summary:** Diagnose why PerplexityBot and ClaudeBot fail to cite your domain. Check AI crawler parity, /llms.txt presence, and structured entity grounding. - **Direct Answer:** Perplexity and Claude omit websites primarily due to three technical barriers: edge firewall blocks (HTTP 403/503 on PerplexityBot or ClaudeBot), lack of a machine-readable /llms.txt directory, or thin client-side JavaScript pages that lack server-rendered text and JSON-LD schema. ### 3. Is Cloudflare or My Firewall Blocking AI Search Bots? - **URL:** https://gocrawl.tech/q/is-cloudflare-blocking-ai-bots - **Category:** AI Search & Bots - **Summary:** Run a simultaneous multi-agent probe across Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot to detect WAF and CDN bot blocking. - **Direct Answer:** Many CDN and WAF configurations—including Cloudflare Bot Fight Mode—treat AI search crawlers as unverified scrapers, serving JS challenges or HTTP 403 Forbidden responses to GPTBot, ClaudeBot, and PerplexityBot even when Googlebot passes cleanly. ### 4. What Should My robots.txt Look Like for AI Search Engines? - **URL:** https://gocrawl.tech/q/robots-txt-for-ai-search-engines - **Category:** AI Search & Bots - **Summary:** Audit your live robots.txt for AI search crawler access and generate a clean, spec-compliant robots.txt for Googlebot, OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot. - **Direct Answer:** An optimal robots.txt for AI search explicitly allows retrieval agents (Googlebot, OAI-SearchBot, PerplexityBot, ClaudeBot) on public content while blocking private app routes (/admin, /api, /checkout) and referencing both your XML sitemap and /llms.txt. ### 5. Does My Website Have a Valid /llms.txt File? - **URL:** https://gocrawl.tech/q/does-my-website-have-llms-txt - **Category:** llms.txt & Grounding - **Summary:** Check if your domain serves a valid /llms.txt file for AI search agents, verify markdown section formatting, and test linked URLs for 200 OK status. - **Direct Answer:** The /llms.txt standard places a concise, machine-readable Markdown index at the root of your domain (https://yourdomain.com/llms.txt) so LLMs and AI retrieval agents can immediately understand your product identity, core verticals, and canonical documentation links. ### 6. How Do I Generate an /llms.txt File for My Website? - **URL:** https://gocrawl.tech/q/how-to-generate-llms-txt - **Category:** llms.txt & Grounding - **Summary:** Automatically crawl your top pages and generate a 3-file AI grounding pack: /llms.txt, /llms-full.txt, and a zero-JS semantic entity HTML page. - **Direct Answer:** You can generate a production-ready /llms.txt file by crawling your domain's indexable 200 OK pages, ranking them by internal PageRank (LinkScore), grouping them by directory vertical, and formatting their titles and meta descriptions into clean Markdown. ### 7. Does My Website Have Valid JSON-LD Schema for AI Search? - **URL:** https://gocrawl.tech/q/does-my-website-have-json-ld-schema - **Category:** llms.txt & Grounding - **Summary:** Inspect your page for structured JSON-LD schema.org entities (Organization, WebSite, SoftwareApplication, FAQPage) and generate missing structured data. - **Direct Answer:** JSON-LD (<script type="application/ld+json">) provides explicit machine-readable entity definitions (Organization, WebSite, Product, FAQPage) that Google Knowledge Graph and AI answer engines use to disambiguate your brand, pricing, and authorship. ### 8. Can AI Crawlers Read My React, Next.js, or SPA Website Without JavaScript? - **URL:** https://gocrawl.tech/q/can-ai-crawlers-read-my-javascript-site - **Category:** llms.txt & Grounding - **Summary:** Test what non-JS AI crawlers (GPTBot, ClaudeBot, PerplexityBot) actually see in your raw server HTML response versus client-side JavaScript. - **Direct Answer:** Unlike Googlebot—which runs a headless Chromium rendering queue—most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) fetch only the raw initial HTTP HTML response. If your React, Vue, or Next.js app renders an empty <div id="root"></div> shell without SSR or SSG, AI bots see zero content. ### 9. Will My Page Title or Meta Description Get Cut Off in Google Search? - **URL:** https://gocrawl.tech/q/will-my-title-get-cut-off-in-google - **Category:** On-Page & Keywords - **Summary:** Measure the exact pixel width and character length of your live <title> and <meta name="description"> tags and preview your Google SERP snippet. - **Direct Answer:** Google truncates page titles based on pixel width (roughly 580px–600px on desktop, or 30–60 characters) rather than raw character count alone, and truncates meta descriptions around 920px (70–160 characters). Wide uppercase letters like W and M consume significantly more pixels than narrow characters. ### 10. Are My Page Keywords Aligned With My Title, H1, and Meta Description? - **URL:** https://gocrawl.tech/q/are-my-keywords-aligned-with-title-and-h1 - **Category:** On-Page & Keywords - **Summary:** Analyze 1-word, 2-word, and 3-word N-gram keyword frequency on your page and verify semantic alignment across Title, H1, H2, and Meta Description. - **Direct Answer:** Semantic keyword alignment measures whether the dominant 1-word, 2-word, and 3-word phrases in your visible body copy actually appear in your highest-weight HTML signals: <title>, <h1>, <h2>, and <meta name="description">. ### 11. Why Is Google Ignoring or Rewriting My Page Title? - **URL:** https://gocrawl.tech/q/why-is-google-rewriting-my-page-title - **Category:** On-Page & Keywords - **Summary:** Check the 5 technical triggers that cause Google to rewrite your <title> tag in search results: pixel overflow, H1 mismatch, keyword stuffing, or boilerplate. - **Direct Answer:** Google rewrites over 60% of page titles that suffer from one of five defects: pixel width exceeding 600px, severe mismatch between the <title> and the visible <h1> heading, generic placeholder text ('Home' or 'Untitled'), repetitive keyword stuffing, or missing brand/topic alignment. ### 12. Is My Page Indexable by Google and AI Search Engines? - **URL:** https://gocrawl.tech/q/is-my-page-indexable - **Category:** On-Page & Keywords - **Summary:** Verify HTTP status code, meta robots directives, X-Robots-Tag headers, and self-referencing canonical tags to confirm whether your URL can be indexed. - **Direct Answer:** For a page to be indexable by Google and AI search engines, it must return HTTP 200 OK, contain no 'noindex' directive in <meta name="robots"> or the X-Robots-Tag HTTP header, and specify a self-referencing <link rel="canonical"> URL. ### 13. Does My URL Have a Redirect Chain or Redirect Loop? - **URL:** https://gocrawl.tech/q/does-my-url-have-a-redirect-chain - **Category:** Redirects & Crawlability - **Summary:** Trace every HTTP 301, 302, 307, and 308 redirect hop from source to destination, measure per-hop latency, and generate a flattened 1-hop server rule. - **Direct Answer:** A redirect chain occurs when URL A redirects to URL B, which then redirects to URL C (2 or more hops). Each additional hop adds network latency, wastes crawl budget, and risks Googlebot or AI crawlers abandoning the request before reaching the final 200 OK destination. ### 14. Does My http:// and www Redirect in a Single Hop to https://? - **URL:** https://gocrawl.tech/q/check-http-to-https-www-redirects - **Category:** Redirects & Crawlability - **Summary:** Test all 4 canonical domain variants (http://, http://www., https://, https://www.) to ensure 3 variants redirect cleanly in 1 hop to your canonical origin. - **Direct Answer:** Every domain has four protocol/subdomain variants: http://domain.com, http://www.domain.com, https://domain.com, and https://www.domain.com. Exactly one should return 200 OK as your canonical origin, while the other three should 301 redirect to it in a single hop. ### 15. Does My Website Have Broken Links or Crawl Errors? - **URL:** https://gocrawl.tech/q/does-my-page-have-broken-links - **Category:** Redirects & Crawlability - **Summary:** Run a live multi-page crawl on your domain to detect 404 broken links, 5xx server errors, redirect chains, and orphan pages. - **Direct Answer:** Internal 404 broken links act as dead ends for both human visitors and search engine crawlers, leaking internal PageRank (LinkScore) and signaling poor content maintenance to ranking algorithms. ### 16. Is My sitemap.xml Valid and Free of Redirecting or Noindex URLs? - **URL:** https://gocrawl.tech/q/is-my-xml-sitemap-clean - **Category:** Redirects & Crawlability - **Summary:** Check your domain's indexability health and learn how to purge 3xx redirects, 404 errors, and noindex pages from your XML sitemap. - **Direct Answer:** A clean XML sitemap must contain only canonical, indexable URLs that return HTTP 200 OK. Including 301 redirecting URLs, 404 pages, or noindex URLs in sitemap.xml sends conflicting signals to Google Search Console and wastes crawl budget. ### 17. How Does My Website's SEO and AI Readiness Compare to My Competitor? - **URL:** https://gocrawl.tech/q/compare-website-seo-and-ai-readiness - **Category:** Competitor & Migration - **Summary:** Run a live head-to-head Technical SEO and AI Search Readiness comparison between your domain and any competitor domain. - **Direct Answer:** Comparing your technical SEO and AI Search readiness side-by-side against a direct competitor reveals whether they outrank or out-cite you due to faster response times, cleaner indexability, superior keyword alignment, or deployed /llms.txt grounding. ### 18. How Do I Map 301 Redirects When Redesigning or Migrating a Website? - **URL:** https://gocrawl.tech/q/how-to-map-301-redirects-for-site-migration - **Category:** Competitor & Migration - **Summary:** Audit your domain before a site migration and generate ready-to-deploy 301 redirect rules for Nginx, Caddyfile, Apache .htaccess, and Cloudflare. - **Direct Answer:** To prevent traffic loss during a website redesign or CMS migration, every legacy indexable URL must be mapped via a permanent 301 redirect to its closest semantic equivalent on the new site—matched by URL slug, H1 heading, and page title similarity. ### 19. What Is My Website's Technical SEO and AI Search Health Score? - **URL:** https://gocrawl.tech/q/website-seo-and-ai-search-score - **Category:** Competitor & Migration - **Summary:** Calculate your website's 0-100 executive health score and letter grade across Crawlability, Indexability, On-Page SEO, Link Graph, Speed, and AI Search Readiness. - **Direct Answer:** GoCrawl evaluates websites across six weighted architecture pillars: Crawlability & Status Health (25%), Indexability & Canonicals (20%), On-Page & Content Structure (20%), Internal Link Graph Authority (15%), Speed & Payload Efficiency (10%), and AI Search / GEO Readiness (10%). ### 20. How Do I Audit a Client or Prospect Website for an SEO Agency Pitch? - **URL:** https://gocrawl.tech/q/how-to-audit-prospect-website-for-seo-pitch - **Category:** Competitor & Migration - **Summary:** Generate an executive Technical SEO & AI Search teardown report for any prospect domain to win agency retainers with real crawl data. - **Direct Answer:** The fastest way to win an SEO or GEO agency retainer is to send the prospect a personalized, data-backed teardown showing their exact letter grade, AI bot blocking status, missing /llms.txt manifest, and top 3 high-impact technical fixes.

# GoCrawl (gocrawl.tech) — Full Technical Knowledge Base & AI Reference > Machine-readable single-file compilation for LLMs, AI search engines (ChatGPT Search, Perplexity, Claude, Google AI Overviews), and MCP agents. Source: https://gocrawl.tech --- ## 1. Platform Overview - **Product Name:** GoCrawl - **Canonical Origin:** https://gocrawl.tech - **Core Capabilities:** High-speed Go concurrent website crawler, 6-Dimension Technical SEO & AI Search (GEO) Grader, 3-file `/llms.txt` grounding pack generator, 20 interactive diagnostic question tools, 24/7 HTTP/AI-bot Watchdog, and native Model Context Protocol (MCP) server. - **Supported Platforms:** Web (`https://gocrawl.tech`), Windows (`GoCrawl-Windows-amd64.exe`), macOS (`GoCrawl-macOS-universal.zip`), Linux (`GoCrawl-Linux-amd64`), and MCP (`https://gocrawl.tech/api/mcp`). ## 2. Six-Pillar Site Scorecard Methodology GoCrawl grades every domain on a 0–100 scale (`A+` through `F`) weighted across six technical pillars: 1. **Crawl & Indexability (25%):** HTTP 200 OK ratio, broken 4xx/5xx links, redirect chains/loops, canonical consistency. 2. **On-Page & Keyword Alignment (20%):** Title pixel width (<=580px), H1 presence, meta description, and semantic n-gram alignment between visible body copy and ``/`<h1>`. 3. **Site Architecture & Link Equity (15%):** Click depth distribution, internal PageRank (`LinkScore`), orphan page detection. 4. **Performance & TTFB (15%):** Server response latency and HTML payload efficiency. 5. **Structured Data & Schema.org (10%):** Embedded `application/ld+json` entity validation. 6. **AI Search & GEO Readiness (15%):** Live bot parity across `Googlebot`, `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, and `PerplexityBot`, SSR raw HTML word count, and `/llms.txt` + `/llms-full.txt` grounding presence. ## 3. Cloud & Local MCP Server Integration - **Discovery Endpoint:** `GET https://gocrawl.tech/api/mcp` - **Tools List Endpoint:** `GET https://gocrawl.tech/api/mcp/tools` - **JSON-RPC 2.0 Endpoint:** `POST https://gocrawl.tech/api/mcp` - **Local Stdio Mode:** `go-crawl -mcp -db gocrawl.db` ## 4. Complete Directory of 20 Interactive Diagnostic Question Pages ### 1. Is ChatGPT Blocked From Crawling My Website? - **URL:** https://gocrawl.tech/q/is-chatgpt-blocked-from-my-website - **Category:** AI Search & Bots - **Summary:** Test live if OpenAI's GPTBot and OAI-SearchBot can crawl your website or if Cloudflare, WAF rules, or robots.txt are blocking ChatGPT Search citations. - **Direct Answer:** ChatGPT relies on two user-agents—OAI-SearchBot (for live search citations) and GPTBot (for model grounding)—to read your site. If your CDN, WAF, or robots.txt returns HTTP 403 or Disallow to these bots while allowing Googlebot, your pages will be omitted from ChatGPT Search answers. ### 2. Why Is Perplexity or Claude Not Citing My Website? - **URL:** https://gocrawl.tech/q/why-is-perplexity-not-citing-my-website - **Category:** AI Search & Bots - **Summary:** Diagnose why PerplexityBot and ClaudeBot fail to cite your domain. Check AI crawler parity, /llms.txt presence, and structured entity grounding. - **Direct Answer:** Perplexity and Claude omit websites primarily due to three technical barriers: edge firewall blocks (HTTP 403/503 on PerplexityBot or ClaudeBot), lack of a machine-readable /llms.txt directory, or thin client-side JavaScript pages that lack server-rendered text and JSON-LD schema. ### 3. Is Cloudflare or My Firewall Blocking AI Search Bots? - **URL:** https://gocrawl.tech/q/is-cloudflare-blocking-ai-bots - **Category:** AI Search & Bots - **Summary:** Run a simultaneous multi-agent probe across Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot to detect WAF and CDN bot blocking. - **Direct Answer:** Many CDN and WAF configurations—including Cloudflare Bot Fight Mode—treat AI search crawlers as unverified scrapers, serving JS challenges or HTTP 403 Forbidden responses to GPTBot, ClaudeBot, and PerplexityBot even when Googlebot passes cleanly. ### 4. What Should My robots.txt Look Like for AI Search Engines? - **URL:** https://gocrawl.tech/q/robots-txt-for-ai-search-engines - **Category:** AI Search & Bots - **Summary:** Audit your live robots.txt for AI search crawler access and generate a clean, spec-compliant robots.txt for Googlebot, OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot. - **Direct Answer:** An optimal robots.txt for AI search explicitly allows retrieval agents (Googlebot, OAI-SearchBot, PerplexityBot, ClaudeBot) on public content while blocking private app routes (/admin, /api, /checkout) and referencing both your XML sitemap and /llms.txt. ### 5. Does My Website Have a Valid /llms.txt File? - **URL:** https://gocrawl.tech/q/does-my-website-have-llms-txt - **Category:** llms.txt & Grounding - **Summary:** Check if your domain serves a valid /llms.txt file for AI search agents, verify markdown section formatting, and test linked URLs for 200 OK status. - **Direct Answer:** The /llms.txt standard places a concise, machine-readable Markdown index at the root of your domain (https://yourdomain.com/llms.txt) so LLMs and AI retrieval agents can immediately understand your product identity, core verticals, and canonical documentation links. ### 6. How Do I Generate an /llms.txt File for My Website? - **URL:** https://gocrawl.tech/q/how-to-generate-llms-txt - **Category:** llms.txt & Grounding - **Summary:** Automatically crawl your top pages and generate a 3-file AI grounding pack: /llms.txt, /llms-full.txt, and a zero-JS semantic entity HTML page. - **Direct Answer:** You can generate a production-ready /llms.txt file by crawling your domain's indexable 200 OK pages, ranking them by internal PageRank (LinkScore), grouping them by directory vertical, and formatting their titles and meta descriptions into clean Markdown. ### 7. Does My Website Have Valid JSON-LD Schema for AI Search? - **URL:** https://gocrawl.tech/q/does-my-website-have-json-ld-schema - **Category:** llms.txt & Grounding - **Summary:** Inspect your page for structured JSON-LD schema.org entities (Organization, WebSite, SoftwareApplication, FAQPage) and generate missing structured data. - **Direct Answer:** JSON-LD (<script type="application/ld+json">) provides explicit machine-readable entity definitions (Organization, WebSite, Product, FAQPage) that Google Knowledge Graph and AI answer engines use to disambiguate your brand, pricing, and authorship. ### 8. Can AI Crawlers Read My React, Next.js, or SPA Website Without JavaScript? - **URL:** https://gocrawl.tech/q/can-ai-crawlers-read-my-javascript-site - **Category:** llms.txt & Grounding - **Summary:** Test what non-JS AI crawlers (GPTBot, ClaudeBot, PerplexityBot) actually see in your raw server HTML response versus client-side JavaScript. - **Direct Answer:** Unlike Googlebot—which runs a headless Chromium rendering queue—most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) fetch only the raw initial HTTP HTML response. If your React, Vue, or Next.js app renders an empty <div id="root"></div> shell without SSR or SSG, AI bots see zero content. ### 9. Will My Page Title or Meta Description Get Cut Off in Google Search? - **URL:** https://gocrawl.tech/q/will-my-title-get-cut-off-in-google - **Category:** On-Page & Keywords - **Summary:** Measure the exact pixel width and character length of your live <title> and <meta name="description"> tags and preview your Google SERP snippet. - **Direct Answer:** Google truncates page titles based on pixel width (roughly 580px–600px on desktop, or 30–60 characters) rather than raw character count alone, and truncates meta descriptions around 920px (70–160 characters). Wide uppercase letters like W and M consume significantly more pixels than narrow characters. ### 10. Are My Page Keywords Aligned With My Title, H1, and Meta Description? - **URL:** https://gocrawl.tech/q/are-my-keywords-aligned-with-title-and-h1 - **Category:** On-Page & Keywords - **Summary:** Analyze 1-word, 2-word, and 3-word N-gram keyword frequency on your page and verify semantic alignment across Title, H1, H2, and Meta Description. - **Direct Answer:** Semantic keyword alignment measures whether the dominant 1-word, 2-word, and 3-word phrases in your visible body copy actually appear in your highest-weight HTML signals: <title>, <h1>, <h2>, and <meta name="description">. ### 11. Why Is Google Ignoring or Rewriting My Page Title? - **URL:** https://gocrawl.tech/q/why-is-google-rewriting-my-page-title - **Category:** On-Page & Keywords - **Summary:** Check the 5 technical triggers that cause Google to rewrite your <title> tag in search results: pixel overflow, H1 mismatch, keyword stuffing, or boilerplate. - **Direct Answer:** Google rewrites over 60% of page titles that suffer from one of five defects: pixel width exceeding 600px, severe mismatch between the <title> and the visible <h1> heading, generic placeholder text ('Home' or 'Untitled'), repetitive keyword stuffing, or missing brand/topic alignment. ### 12. Is My Page Indexable by Google and AI Search Engines? - **URL:** https://gocrawl.tech/q/is-my-page-indexable - **Category:** On-Page & Keywords - **Summary:** Verify HTTP status code, meta robots directives, X-Robots-Tag headers, and self-referencing canonical tags to confirm whether your URL can be indexed. - **Direct Answer:** For a page to be indexable by Google and AI search engines, it must return HTTP 200 OK, contain no 'noindex' directive in <meta name="robots"> or the X-Robots-Tag HTTP header, and specify a self-referencing <link rel="canonical"> URL. ### 13. Does My URL Have a Redirect Chain or Redirect Loop? - **URL:** https://gocrawl.tech/q/does-my-url-have-a-redirect-chain - **Category:** Redirects & Crawlability - **Summary:** Trace every HTTP 301, 302, 307, and 308 redirect hop from source to destination, measure per-hop latency, and generate a flattened 1-hop server rule. - **Direct Answer:** A redirect chain occurs when URL A redirects to URL B, which then redirects to URL C (2 or more hops). Each additional hop adds network latency, wastes crawl budget, and risks Googlebot or AI crawlers abandoning the request before reaching the final 200 OK destination. ### 14. Does My http:// and www Redirect in a Single Hop to https://? - **URL:** https://gocrawl.tech/q/check-http-to-https-www-redirects - **Category:** Redirects & Crawlability - **Summary:** Test all 4 canonical domain variants (http://, http://www., https://, https://www.) to ensure 3 variants redirect cleanly in 1 hop to your canonical origin. - **Direct Answer:** Every domain has four protocol/subdomain variants: http://domain.com, http://www.domain.com, https://domain.com, and https://www.domain.com. Exactly one should return 200 OK as your canonical origin, while the other three should 301 redirect to it in a single hop. ### 15. Does My Website Have Broken Links or Crawl Errors? - **URL:** https://gocrawl.tech/q/does-my-page-have-broken-links - **Category:** Redirects & Crawlability - **Summary:** Run a live multi-page crawl on your domain to detect 404 broken links, 5xx server errors, redirect chains, and orphan pages. - **Direct Answer:** Internal 404 broken links act as dead ends for both human visitors and search engine crawlers, leaking internal PageRank (LinkScore) and signaling poor content maintenance to ranking algorithms. ### 16. Is My sitemap.xml Valid and Free of Redirecting or Noindex URLs? - **URL:** https://gocrawl.tech/q/is-my-xml-sitemap-clean - **Category:** Redirects & Crawlability - **Summary:** Check your domain's indexability health and learn how to purge 3xx redirects, 404 errors, and noindex pages from your XML sitemap. - **Direct Answer:** A clean XML sitemap must contain only canonical, indexable URLs that return HTTP 200 OK. Including 301 redirecting URLs, 404 pages, or noindex URLs in sitemap.xml sends conflicting signals to Google Search Console and wastes crawl budget. ### 17. How Does My Website's SEO and AI Readiness Compare to My Competitor? - **URL:** https://gocrawl.tech/q/compare-website-seo-and-ai-readiness - **Category:** Competitor & Migration - **Summary:** Run a live head-to-head Technical SEO and AI Search Readiness comparison between your domain and any competitor domain. - **Direct Answer:** Comparing your technical SEO and AI Search readiness side-by-side against a direct competitor reveals whether they outrank or out-cite you due to faster response times, cleaner indexability, superior keyword alignment, or deployed /llms.txt grounding. ### 18. How Do I Map 301 Redirects When Redesigning or Migrating a Website? - **URL:** https://gocrawl.tech/q/how-to-map-301-redirects-for-site-migration - **Category:** Competitor & Migration - **Summary:** Audit your domain before a site migration and generate ready-to-deploy 301 redirect rules for Nginx, Caddyfile, Apache .htaccess, and Cloudflare. - **Direct Answer:** To prevent traffic loss during a website redesign or CMS migration, every legacy indexable URL must be mapped via a permanent 301 redirect to its closest semantic equivalent on the new site—matched by URL slug, H1 heading, and page title similarity. ### 19. What Is My Website's Technical SEO and AI Search Health Score? - **URL:** https://gocrawl.tech/q/website-seo-and-ai-search-score - **Category:** Competitor & Migration - **Summary:** Calculate your website's 0-100 executive health score and letter grade across Crawlability, Indexability, On-Page SEO, Link Graph, Speed, and AI Search Readiness. - **Direct Answer:** GoCrawl evaluates websites across six weighted architecture pillars: Crawlability & Status Health (25%), Indexability & Canonicals (20%), On-Page & Content Structure (20%), Internal Link Graph Authority (15%), Speed & Payload Efficiency (10%), and AI Search / GEO Readiness (10%). ### 20. How Do I Audit a Client or Prospect Website for an SEO Agency Pitch? - **URL:** https://gocrawl.tech/q/how-to-audit-prospect-website-for-seo-pitch - **Category:** Competitor & Migration - **Summary:** Generate an executive Technical SEO & AI Search teardown report for any prospect domain to win agency retainers with real crawl data. - **Direct Answer:** The fastest way to win an SEO or GEO agency retainer is to send the prospect a personalized, data-backed teardown showing their exact letter grade, AI bot blocking status, missing /llms.txt manifest, and top 3 high-impact technical fixes.