NullPointer@programming.devtoPrivacy@lemmy.ml•How to block AI Crawler Bots using robots.txt file
20·
1 month agorobots.txt will not block a bad bot, but you can use it to lure the bad bots into a “bot-trap” so you can ban them in an automated fashion.
1-800-COLLECT