Robots.txt

A robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site. It is placed in the root directory of your website (e.g., example.com/robots.txt).

The asterisk (*) is a wildcard that matches all web crawlers. Rules under "User-agent: *" apply to every bot unless a more specific user-agent section overrides them.

Not necessarily. While Disallow prevents crawling, a page can still appear in search results if other sites link to it. To truly prevent indexing, use a "noindex" meta tag or X-Robots-Tag HTTP header.

Crawl-delay tells compliant bots to wait a specified number of seconds between requests. This can reduce server load from aggressive crawlers. Note that Googlebot ignores Crawl-delay; use Google Search Console instead.