Skip to main content
WEB MAESTRO
العربيةStart a project
Robots.txt generator

Robots.txt generator with a path tester

Choose your site type to get a valid file, and try any URL to see whether it would be blocked from Google before you upload the file. The file does not block CSS, JS or images, because Google needs them to see your page as a visitor does.

Live path testingStore and WordPress templatesDownload the ready file
Site type
Block AI crawlers (optional)
Stops these companies from using your content for training. It does not affect your normal Google Search visibility. Compliance with the file is voluntary on the crawler’s side.
User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Test a URL

Blocked by rule: Disallow: /admin/

Put the file in your site root named robots.txt so it is served at /robots.txt. Blocking here guides crawlers; it does not protect private content. To keep a page out of results use a noindex tag, because a page blocked by the file can stay indexed by its URL if other pages link to it.

How the tool works

Choose your site type (brochure, store, WordPress), add paths to block and your sitemap URL, and get a robots.txt ready to copy or download. Try any path in the test box to see whether it would be blocked before you upload the file. The file is guidance for crawlers, not protection for private content.

Frequently asked questions

Where do I put the file?

In the root of your site: example.com/robots.txt.

Does blocking a page keep it out of Google?

Not always. A blocked page can still appear without a description if other pages link to it. To prevent indexing, use a noindex tag.

Should I block CSS and JS files?

No. Google needs them to see your page as a visitor does, and blocking them hurts indexing.

Related tools

Meta Tags GeneratorImage CompressorGoogle Snippet PreviewAll tools →

Need this built around your business?

Web Maestro is a registered company in Amman that builds websites, stores and systems and improves their search visibility. Tell us about your project and we will suggest what fits.

Chat on WhatsAppContact form

How would you like to reach us?

Pick whatever is easiest. A senior team member will get back to you.

Call us now+962 789 123 421WhatsAppMessage us directly