While the Robots.txt file is not a part of the Showcase IDX plugin itself, it's important to understand the impact it can have on your site. This article is to aim you in the right direction so that you can understand it and adjust it as desired.
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. More details on the purpose of a Robots.txt file in Google's documentation here.
As a user with a Wordpress site, by default, your site has its own robots.txt file that search engines will pick up. More details on this kind of setting here. You have a few options if you need to add additional customizations to your robots.txt file.
- You can create a custom robots.txt file and upload it to your root directory via SFTP.
- You can use a plugin specifically designed to edit your robots.txt file, like Robots.txt Editor.
- Plugins designed to improve SEO (like Yoast or Rank Math SEO) also include sections for editing your robots.txt file.
WordPress.com only allows certain plans to edit this file, such as Business or eCommerce.