|
|
This will need some consideration. Do we really want to exclude web crawlers? Or do we want to follow the philosophy that a web crawler that triggers spam detection is a poorly-made web crawler and "gets what it deserves"? |
|
|
|
It's difficult because any request can easily have a 'spoofed' (if you can even call it that) user-agent. |
|
|
|
I've moved to a feature request. I doubt this is happening on default settings. If it is, please mention such. I actually would like to know the settings it is happening for anyway.
TBH the importance of search engines are declining. Personally I rarely Google stuff nowadays because I either use ChatGPT, or some kind of walled garden (e.g. Reddit). That's sad, but it does lower the criticality of this kind of thing, especially if the crawlers are faulty in some way. |
|
|
|
Possibly the health check should use different wording then... instead of saying "accidentally banned a web crawler", implying it should be unbanned, maybe it should say "Banned a potential web crawler... (IP). If you believe this to be a false-positive, unban the IP address." |
|
