fossilesque@mander.xyzM to Science Memes@mander.xyzEnglish · 5 months agoBlack Mirror AImander.xyzimagemessage-square210linkfedilinkarrow-up11.61Karrow-down115
arrow-up11.59Karrow-down1imageBlack Mirror AImander.xyzfossilesque@mander.xyzM to Science Memes@mander.xyzEnglish · 5 months agomessage-square210linkfedilink
minus-squarejaschen@lemm.eelinkfedilinkEnglisharrow-up4·5 months agoIt’s naive to assume that google crawlers respect robot.txt.
minus-squarerosco385@lemm.eelinkfedilinkEnglisharrow-up3·5 months agoIt’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂
It’s naive to assume that google crawlers respect robot.txt.
It’d be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don’t stay away. 😂