How To Configure Robots.txt In Magento 2

Configuring robots.txt is an important part of optimizing your website’s SEO. It provides instructions for search engines to index your store. In this post, I will guide you How To Configure Robots.txt In Magento 2.

What is Robots.txt?

Robots.txt is a text file that resides in a website’s root directory and provides instructions for search engines to crawl about which pages they can crawl for indexing.

You need to create instructions to avoid indexing disallowed pages.

Magento 2 gives you the ability to set robots.txt instructions in Admin Panel with just the following very simple steps:

Steps To Configure Robots.txt In Magento 2

Step 1: Login to Admin Panel.

Step 2: Choose CONTENT -> Design -> Configuration.

Step 3: Choose Edit the Global of the Action column.

Step 4: Expand the Search Engine Robots section.

Step 5: Set Default Robots option to one of the following:

• INDEX, FOLLOW: Instructs web crawlers to index the site and to check back later for changes.
• NOINDEX, FOLLOW: Instructs web crawlers to avoid indexing the site, but to check back later for changes.
• INDEX, NOFOLLOW: Instructs web crawlers to index the site once, but to not check back later for changes.
• NOINDEX, NOFOLLOW: Instructs web crawlers to avoid indexing the site, and to not check back later for changes.

Step 6: In Edit Custom Instruction Of Robots.txt File option, you can enter the custom instructions following:

• Allow Full Access:

User-agent:*
Disallow:

• Disallow Access to All Folders:

User-agent:*
Disallow: /

• Default:

Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

• Disallow Duplicate Content:

Disallow: /tag/
Disallow: /review

• Disallow Access to User Account & Checkout Pages:

Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

• Disallow Access to CMS Directories:

Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/

• Disallow Access to Catalog Search Pages:

Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

• Disallow Access to URL Filter Searches:

Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*

To restore the default instructions, click Reset To Default button.

Step 7: Click to Save Configuration.

You can read more useful articles like How To Add Order ID In Invoice PDF Print-outs In Magento 2.

Follow us for the more helpful posts!

We hope this is a useful post for you.

Thank you for reading!