Robots.txt | Axero Documentation

Websites use the robots exclusion standard (robots.txt) to tell web crawlers and other web robots which areas of the website should not be processed or scanned. Search engines often use web robots to categorize websites.

Click Save.

Example

To exclude all web crawlers from indexing any part of your site, you can use the following content in your robots.txt file:

User-agent: *
Disallow: /

User-agent: * refers to all web crawlers.
Disallow: / tells these web crawlers not to index any page or file on the site.

To exclude specific directories and/or files in your site, add each page or file as a new line in robots.txt:

Disallow: /search/
Disallow: /login/
Disallow: /file.aspx

Setting Recognized Crawlers

You can specify which web crawlers and bots Axero recognizes using the Crawlers system property. This setting assists with SEO and site indexing by identifying key crawlers.

• Go to Control Panel > System > System Properties.
• Search for Crawlers.
• Use vertical bars (|) to separate each bot name in the list. For example: SEOChat::Bot|Gecko XML-Sitemaps|Googlebot|msnbot
• Click Save.