Control search engine access to your site.
By default, robots assume they can crawl everything. Use Disallow to block sensitive folders like admin panels or generated search results.
Always include specific Sitemap directives to help bots find your XML sitemap quickly.