Improve Your Drupal 9 SEO with the RobotsTxt Module

If you have a Drupal 9 website, we are here to disclose the most effective and simple ways to enhance your SEO. All you have to do is optimize how search engines interact with your site. The best possible critical tool in this process is the robots.txt file (using it in Drupal 9 could make the whole process even easier).

What Are Robots.txt File?

One significantly overlooked on-page SEO component is the Robots.txt file. Not everyone understands the benefits it offers. The crawler bots are informed which pages require crawling and which do not via the Robots.txt file, which functions as a sort of access control system. It is a set of guidelines that the different web spiders examine before attempting to explore your website.

Drupal 8 and 9 offer a lot of powerful SEO modules that make optimization easy and also improve search engines. One such module is the RobotsTxt module. It allows effortless management of the Robot.txt file, in a multisite Drupal environment. You can create and modify Robot.txt files thanks to their user-friendly interface. Let us learn more about it and how you can implement it in Drupal 9.

How Does Robots.txt Boost Your SEO?

Robot.txt files restrict crawlers from crawling your web pages, sections, and files. But why would you not want all your pages to be crawled? Why is there any kind of requirement to have any restrictions? Because, in this case, the more is not always merrier.

• Not having a Robot.txt file will only lead to web spiders crawling all over your web pages, files, and sections, which uses up the crawl budget, affecting your SEO.
• The number of sites that web spiders (such as Google Bot, Yahoo, Bing, etc.) crawl during a specified period is your crawl budget. Your chances of being indexed more quickly may be lowered if you have too many pages to crawl. Furthermore, you may miss out on indexing the crucial pages!
• It's not necessary to crawl all of your pages. For example, you would not want Google to crawl your internal login pages or the websites of your development/staging environment.
• You might want to prevent the crawling of your media files, such as videos, images, or other documents.
• If you have a reasonable number of duplicate content pages, adding them to the Robots.txt file rather than using canonical links on each page is best.

Step-By-Step Guide on How To Install & Implement The Robots.txt Module In Drupal 9

The Robot.txt module is the best option for generating a Robot.txt file. It is great for each website when you are running multiple sites from one codebase.

Step 1: Install the robots.txt module for Drupal 9

Using Composer:
composer require 'drupal/robotstxt:^1.4'

Step 2: Enable the module

Go to Home > Administration > Extend (/admin/modules) and enable RobotsTxt module

Step 3: Remove the existing Robots.txt file

After installing the module, make sure to delete the robots.txt file in the root of your Drupal installation so that this module can showcase its own robots.txt file. Otherwise, the module will not be able to intercept requests for the URL /robots.txt.

Step 4: Configure

Navigate to Home -> Administration -> Configuration ->Search and metadata-> Robots.txt (/admin/config/search/robotstxt), where your modifications can be added to the "Contents of robots.txt" section. Save the configuration.

Step 5: Verify

Please visit https://yoursitename.com/robots.txt to confirm your modifications.

Robot.Txt API

The RobotsTxt API can be used to implement a common set of directives for your multisite context. The hook_robotstxt() is the only hook in the module. You can use the hook to define additional directives in your code.

We have given an example that will add a Disallow for /foo and /bar to the bottom of the robots.txt file without requiring them to be explicitly added to the UI's "Contents of robots.txt" section.

/**
* Add additional lines to the site's robots.txt file.
*
* @return array 
*   An array of strings to add to the robots.txt.
*/
function hook_robotstxt() {
 return [
  'Disallow: /foo',
  'Disallow: /bar',
];
}

Final Thoughts

Optimizing your Drupal 9.x website for search engines is necessary, and the Robots.txt module will make managing your robots.txt file seamless. By successfully managing how search engines crawl your website, you can optimize indexing, maintain a crawl budget, and improve your SEO strategy. Whether you need to limit certain pages, avoid duplicate content issues, or manage multiple sites, the Robots.txt module offers a simple yet effective solution. Implementing the Robots.txt module will ensure the website remains organized, efficient, and search engine-friendly.

Start optimizing today and take control of your site’s visibility!

Content provided by Laxmi Narayan Webworks Pvt Ltd. Thanks to Raman Kumar.

LN Web Works is a dynamic Drupal development company in India, delivering innovative web and mobile app solutions that drive business success. With over a decade of experience, we empower organizations across diverse industries with tailored, client-focused IT services.