How to Configure Robots.txt in Magento 2 for SEO?

Learn how to configure robots txt in Magento 2 store to help it perform better in SEO as these robots.txt files are vital to placing in the right position in search engine.

The following steps show you how to configure Robots txt in Magento 2?

1. On the Admin sidebar, select Stores
2. Develop the Search Engine Robots section
3. When complete, choose Save Config

Build your Amazon-like multi-vendor marketplace website @ $249/- now!!!

           Get your free live demo now: Click here

Hello friends,

As you know, the Magento configuration contains settings to create and control instructions for Web Crawlers and bots that index your site. The guidelines are saved in a file called “robots.txt” that resides at the root of your Magento installation.

The instructions are commands that are documented and followed by most search engines. By default, the robots.txt file that is produced by Magento consists of commands for web crawlers to evade indexing assured parts of the site that contain files that are used internally by the system. You can use the default settings or describe your own custom instructions for all or for specific search engines.

Here we will discuss how to configure robot.txt and establish some Custom instructions.

Configure Robots txt in Magento 2 for SEO

On the Admin sidebar, choose Stores

In settings, select configuration. Then, select Design in the panel on the left under General.

Robots TXT Enlarges the Search Engine Optimization Section

The following are:

Select Default Robots

• INDEX, FOLLOW initiates web crawlers to catalog the site and to check back later for modify.
• NO INDEX, FOLLOW initiates web crawlers to evade indexing the site, but to check back later or modify.
• INDEX, NO FOLLOW initiates web crawlers to index the site once, but to not check back later for modification.
• NO INDEX, NO FOLLOW initiates web crawlers to evade indexing the site and not verify back later for modification.

After that, tap Reset to Default to restore the default instructions.

When complete, Save to Configure

Here, we will establish some examples of Custom Instruction, the following are:

• Allows full Access
• Disallows access to all Folders
• Default instructions

Allows Full Access

• User-agent:*
• Disallow:

Disallows Access to All Folders

• User-agent:*
• Disallow: /

Default Instructions

Disallow: /lib/

Disallow: /*.php

Disallow: /pkginfo/

Disallow: /var/

Disallow: /catalog/

Disallow: /customer/

Disallow: /sendfriend/

Disallow: /review/

Disallow: /*SID=

The following steps help you to learn how to Configure Robots txt in Magento 2 store for better SEO.

Contact us to develop and configure your Magento 2 store to scale your online business.