google console is complaining about Server error (5xx)

Hello,

Received a "New Coverage issue detected for site https://nandalow.com/"  warning email from google search console. Apparently they are scanning my https://nandalow.com/data/ directory and getting some 5xx type errors….. 

For example:

https://compo.sr/data/notifications.php



google is complaining about the data/notifications.php….  Would you recommend we add to the disallow list on the robots.txt file?
 

The robots.txt does not specifically block the /data/ folder (I dont think):

https://compo.sr/robots.txt

the .htaccess file in the /data/ folder is empty

Thanks for your thoughts
 

 

Hi,

Our approach is to put out X-Robots-Tag headers, or <meta> declarations, telling search engines if they should not be indexing something. That way people don't need to worry about robots.txt for anything inbuilt to Composr.

In this particular case if a required parameter is missing for notifications.php it doesn't get to the point of putting out that header.
Fixed in:
https://github.com/ocproducts/composr/commit/8f939dd314c684a9f5c5169e08334a94ce492524