About
☰
All Tools
Categories
Blog About

Robots.txt Generator – Create Crawl Rules & Sitemap Listings

Build robots.txt files with user-agent rules, crawl delays, and sitemap listings for technical SEO and staging environments.

🟢 Runs locally · no uploads

Robots.txt Generator

Generate robots.txt files for SEO

Related tools

Show more

Meta Tag Preview - SEO & Social Media Optimizer

Preview and optimize your meta tags for search engines and social media platforms. See real-time previews of how your page appears on Google, Facebook, and Twitter.

HTTP Status Code Reference

Stop searching for HTTP status code meanings! Complete reference guide with detailed descriptions, usage examples, RFC references, and practical guidance for web developers and API designers. Covers all codes from 100 to 599 with real-world examples. Free, works offline, searchable.

URL Validator

Stop sharing broken or dangerous links! Validate URLs instantly with comprehensive security analysis, domain verification, scheme checking, and SEO recommendations. Detects phishing attempts, malformed URLs, and accessibility issues. Perfect for web development, content management, and link safety. Free, works offline, unlimited validations.

.htaccess Generator

Build Apache .htaccess files with redirects, caching, HTTPS enforcement, and security headers. Export clean, production-ready directives.

SEO Audit Tool

Run quick technical and content audits for any URL. Identify indexation issues, missing structured data, thin content, and performance risks in seconds.
Show more

CSS Minifier

Stop using bloated CSS files! Minify and compress CSS instantly by removing whitespace, comments, and optimizing properties for 30-40% smaller file sizes. Improves page load speed, reduces bandwidth, and optimizes for production. Supports modern CSS including variables, Grid, Flexbox, and media queries. Free, works offline, unlimited minification.

HTML Minifier

Stop deploying bloated HTML files! Compress and optimize HTML code instantly by removing unnecessary whitespace, comments, empty elements, and redundant attributes for production deployment. Reduces file size by 20-40% for faster page loads. Free, works offline, unlimited minification.

JavaScript Minifier

Stop deploying uncompressed JavaScript! Minify and compress JavaScript code instantly by removing comments, whitespace, and optimizing syntax for production use. Reduces file size by 30-60% for faster page loads. Supports ES6+, async/await, and modern JavaScript. Free, works offline, unlimited minification.

CSS Selector Generator

Stop manually writing complex CSS selectors! Generate precise selectors for targeting HTML elements instantly with support for classes, IDs, attributes, pseudo-classes, and complex combinators. Includes specificity calculation and XPath conversion. Perfect for web scraping and styling. Free, works offline, unlimited generation.

CSS Grid Generator

Generate CSS Grid layouts with visual editor, responsive breakpoints, named areas, and custom positioning. Create complex grid systems with HTML and CSS code generation.
› About this tool · FAQ

Interactive robots.txt builder to manage search engine crawl directives, block sensitive paths, and publish sitemap locations in seconds.

Do I need a robots.txt file?

No, but it is recommended. A robots.txt file tells crawlers which areas are off limits, points them to your sitemaps, and prevents wasting crawl budget on duplicate or utility pages.

Will disallowing a path remove it from Google search?

Disallow prevents future crawling, but it does not remove pages already indexed. Use the Google Search Console removal tool or add a noindex tag for permanent removal.

Does Google respect crawl-delay?

Google treats crawl-delay as a hint rather than a rule. Bing and other crawlers are more likely to follow it. Adjusting crawl rate in Search Console is recommended for Google-specific throttling.

How many sitemap entries can I list?

You can list multiple sitemap files or index files. Each sitemap can include up to 50,000 URLs and 50MB of data, so splitting by section (e.g., /blog, /products) is a common approach.

What is the Clean-param directive?

Clean-param is Yandex-specific. It tells crawlers to ignore certain URL parameters to reduce duplicate crawling. Use it when you have tracking parameters or session IDs that should be ignored.