Many people don’t know what the Robots.txt file does and how important it really is. In this article, WordPress SEO Compliant Robots.txt File We will talk about the importance of SEO and prepare a sample Robots.txt file.
The file is a file that provides the link between your site and search engines and directly affects the SEO performance of your site. With the robots.txt file, you can give search engine bots commands that tell you which data and content to index on your site. You can also add the sitemap address to your Robots.txt file so that bots can visit your Sitemap.xml address.
Where is Robots.txt File? How to create?
The robots.txt file is usually located in your FTP address, in the main directory where your site is located. You can use the cPanel File Manager or various FTP connection programs to edit / create the robots.txt file. The file is an ordinary TXT file and can be created with a simple text editor such as Notepad. Just create a new text document that you create with Notepad, then name it robots.txt and place it in your site’s home directory.
If your site’s home directory doesn’t have a file, we strongly recommend creating one immediately.
Robots.txt file is also very important for the security of your site. The readme file in the main directory of your site can reveal your version of WordPress site. Likewise, not indexing the WP-Admin and WP-Content fields is very important for the security of your site.
WP-Admin, Login and Registration pages are not included in the Robots.txt file because WordPress has the “Noindex” tag. However, in particular, to prevent the indexing of the plug-ins (Wp-content / plugins) you use, a vulnerability in a specific plug-in through the site that tries to infiltrate the crawling efforts of people will be frustrated.
Below is a sample Robots.txt file.
The * sign refers to rules that cover all search engine bots. If you just want to set rules for Google or Bing, you can type Googlebot or Bingbot in the “User-Agent” section.
User–Agent: *Disallow: /wp–admin/Allow: /wp–admin/admin–ajax.phpAllow: /wp–content/uploads/Disallow: /wp–content/plugins/Disallow: /readme.htmlSitemap:
We used to add pages, categories, and tags to the file that we didn’t want indexed. However, Google has recently published information; instead of restricting these pages to Robots.txt, Noindex, and so on. began to recommend the use of labels. If you use SEO plugins for WordPress, such as Yoast SEO or All in One SEO Pack, you can easily select noindex from the plugin settings.
For more information about WordPress, visit our WordPress page.