robots.txt Viewer & Explainer

Enter a website URL (e.g., https://ddginc-usa.com or just ddginc-usa.com):

📘 About robots.txt

The robots.txt file provides instructions to web crawlers about which parts of a site should not be accessed or crawled. It is publicly accessible at the root of a domain (e.g., https://ddginc-usa.com/robots.txt).

Common directives:

• User-agent: * — applies rules to all crawlers
• Disallow: / — blocks all access
• Allow: /path/ — specifically allows access
• Sitemap: https://ddginc-usa.com/sitemap.xml — points bots to your sitemap
• Crawl-delay: 10 — requests bots to wait N seconds between requests

🤖 Common and Known User Agent Strings

🔍 Search Engine Crawlers

• Googlebot — Googlebot
• Googlebot-Image — Googlebot-Image
• Googlebot-News — Googlebot-News
• Googlebot-Video — Googlebot-Video
• Bingbot — bingbot
• BingPreview — BingPreview
• Slurp (Yahoo) — Slurp
• DuckDuckGo — DuckDuckBot
• Baidu — Baiduspider
• Yandex — Yandex
• Sogou — Sogou web spider
• Exabot — Exabot
• Mojeek — MojeekBot
• Qwant — Qwantify

📱 Social Media Bots

• Facebook — facebookexternalhit
• Twitter — Twitterbot
• LinkedIn — LinkedInBot
• Pinterest — Pinterestbot

📈 SEO & Monitoring Bots

• Ahrefs — AhrefsBot
• Semrush — SemrushBot
• Majestic — MJ12bot
• DotBot — DotBot
• BLEXBot — BLEXBot
• Archive.org — ia_archiver
• Screaming Frog — Screaming Frog SEO Spider

📱 Mobile & Validation

• Googlebot-Mobile — Googlebot-Mobile
• Google PageSpeed — Google Page Speed Insights
• W3C Validator — W3C_Validator