# robots.txt for SwagBrokers # Updated: 2026-05-06 # ── Global rules ───────────────────────────────────────────────────────────── User-agent: * # DISALLOW: Search (triggers paid Vertex AI API calls) Disallow: /SB-Search Disallow: /SB-Search.aspx # DISALLOW: Transactional pages Disallow: /SB-Cart Disallow: /SB-Cart.aspx Disallow: /SB-Checkout Disallow: /SB-Checkout.aspx Disallow: /SB-Payment Disallow: /SB-Payment.aspx Disallow: /SB-Pay Disallow: /SB-Pay.aspx Disallow: /SB-OrderConfirmation Disallow: /SB-OrderConfirmation.aspx Disallow: /SB-OrderDetails Disallow: /SB-OrderDetails.aspx Disallow: /SB-Orders Disallow: /SB-Orders.aspx # DISALLOW: Quote system Disallow: /SB-Quotes Disallow: /SB-Quotes.aspx Disallow: /SB-QuoteRequest Disallow: /SB-QuoteRequest.aspx Disallow: /SB-QuoteView Disallow: /SB-QuoteView.aspx # DISALLOW: Auth / account pages Disallow: /SB-Login Disallow: /SB-Login.aspx Disallow: /SB-Register Disallow: /SB-Register.aspx Disallow: /SB-Profile Disallow: /SB-Profile.aspx Disallow: /SB-ForgotPassword Disallow: /SB-ForgotPassword.aspx Disallow: /SB-ResetPassword Disallow: /SB-ResetPassword.aspx Disallow: /SB-VerifyEmail Disallow: /SB-VerifyEmail.aspx # DISALLOW: Post-sale service (thin/auth-gated content) Disallow: /SB-Claims Disallow: /SB-Claims.aspx # DISALLOW: Admin / debug Disallow: /SB-Dashboard Disallow: /SB-Dashboard.aspx Disallow: /SB-Config Disallow: /SB-Config.aspx Disallow: /SB-DebugLogs Disallow: /SB-DebugLogs.aspx Disallow: /visitor-diagnostics.aspx Disallow: /SB-Home-Blue Disallow: /SB-Home-Blue.aspx User-agent: * Allow: /SB-Image.ashx User-agent: AdsBot-Google Allow: /SB-Image.ashx # DISALLOW: .ashx API handlers (not content) Disallow: /*.ashx$ # DISALLOW: Server-side source and config Disallow: /bin/ Disallow: /App_Code/ Disallow: /App_Data/ Disallow: /App_Start/ Disallow: /*.vb$ Disallow: /*.config$ Disallow: /*.dll$ # DISALLOW: Internal query parameters that create duplicate content Disallow: /*?*rewritten= Disallow: /*?*session= # ── AdsBot-Google (Google Ads & GMC landing page quality checks) ────────────── # AdsBot ignores User-agent: * — explicit rules required. # Allow product, home, search, and category pages; block transactional/admin. User-agent: AdsBot-Google Disallow: /SB-Search Disallow: /SB-Cart Disallow: /SB-Checkout Disallow: /SB-Payment Disallow: /SB-Pay Disallow: /SB-OrderConfirmation Disallow: /SB-OrderDetails Disallow: /SB-Orders Disallow: /SB-Quotes Disallow: /SB-QuoteRequest Disallow: /SB-QuoteView Disallow: /SB-Login Disallow: /SB-Register Disallow: /SB-Profile Disallow: /SB-ForgotPassword Disallow: /SB-ResetPassword Disallow: /SB-VerifyEmail Disallow: /SB-Dashboard Disallow: /SB-Config Disallow: /SB-DebugLogs Disallow: /visitor-diagnostics.aspx Disallow: /SB-Home-Blue Disallow: /*.ashx$ Disallow: /bin/ Disallow: /App_Code/ Disallow: /App_Data/ Disallow: /App_Start/ # ── Aggressive SEO crawlers ─────────────────────────────────────────────────── User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 10 User-agent: MJ12bot Disallow: / User-agent: DotBot Crawl-delay: 10 User-agent: Bingbot Crawl-delay: 1 # ── Sitemap ─────────────────────────────────────────────────────────────────── Sitemap: https://www.swagbrokers.com/sitemap.xml