Customers visit your eCom store using a search engine like Google Search. They select your store's link from search results. In order to provide results, search engines collect information about your eCom store using web crawlers. Your robots.txt file controls web crawlers.

It is not required to edit your Robots.txt file. It is normal to allow indexing all pages and from all search engines. Edit the robots.txt file only if you have experience with search engine optimization. With it you can:

• Prevent search engines from indexing your site. For example, while building your eCom store you may have unfinished content that shouldn't be indexed.
• Disallow indexing of certain pages on your eCom store. For example, pages that you don't want to be indexed.
• Block specific search engines from indexing your site. For example, if China is outside your target market, you can block Chinese search engines from indexing your site.

You can see your current robots.txt file by going to yourdomain.com/robots.txt. By default, Lightspeed eCom provides your sitemap to search engines and blocks:

• The Back Office login screen - /admin
• Cart page - /cart
• Compare pages - /compare
• Checkout pages - /checkout

Modify your robots.txt file in Lightspeed eCom:

1. On the left menu of the Back Office click Settings and from Website Settings click Web Extras.
2. In the ROBOTS.TXT section, toggle on the switch labeled Status.
3. Enter your robots.txt content in the field labeled Robot.

The basic format of robots.txt is: 

User-agent: [Name of the search engine's web crawler] 
Disallow: [web address not to be crawled]

User-agent identifies a web crawler. For example:

• User-agent: Googlebot - Google Search
• User-agent: Bingbot - Microsoft Bing & Yahoo
• User-agent: Baiduspider - Baidu, a leading Chinese search engine
• User-agent: * - Targets all web crawlers.

Disallow specifies the page you want to block. Enter the file location instead of the full web address. For example:

• Disallow: /Womens-apparel - Prevents indexing of any content in yourdomain.com/Womens-apparel page and any sub pages.
• Disallow: /Womens-apparel/ - Prevents only the content in yourdomain.com/Womens-apparel. All sub pages will still be indexed.
• Disallow: / - Prevents all pages from being indexed.
• Disallow:  - Allows indexing for all pages.

For more information about the robots.txt file, go to the Google support page

Troubleshooting and support for robots.txt

1. Check your code carefully for the use of slash / characters.
2. Ensure there are no blank spaces in the robots.txt file.
3. Use the robots.txt Tester in Google Search console to test for warnings or errors.

For more information about:

• Setting up Google Search Console, visit our Help article here.
• Accessing Google's robots.txt tester, visit Google's Help article here.

Our Support team is committed to assisting you with your eCom store. Editing the robots.txt of your theme could cause issues with your search engine ranking. When this occurs, our Support team may not have the SEO knowledge required to directly solve the problem in your robots.txt file. In these cases they may recommend:

• Contacting a third party SEO expert to solve the issue. 
• Deleting the changes to the robots.txt field to restore its default settings.