Robots.txt

A robots.txt file is a plain text file that is uploaded to a website's folder. It will live at the root of the site. It can be found by adding /robots.txt to the end of the home page URL. For example. https://domain.com/robots.txt It contains directives that tell web crawlers what they are allowed to crawl and what they are not allowed (or disallowed) to crawl.

You can use several directives in a robots.txt file:
 * User-agent: The name of the web crawler that the rule applies to. Using an asterisk (*) means all.
 * Disallow: Anything listed here will be disallow or block the User-agent that was specified
 * Allow: Anything listed here will be allow the User-agent that was specified
 * Sitemap: You can use this to tell the location of the XML sitemap for the site

Examples of robots.txt directive and what they do:

To exclude all crawlers from the whole site User-agent: * Disallow: /

To allow all crawlers to access the whole site User-agent: * Disallow:

To allow Googlebot to access the whole site User-agent: Googlebot Disallow: