@dramaticcat@sh.itjust.works to Lemmy Shitpost@lemmy.world • 2 years agoChad scrapersh.itjust.worksimagemessage-square97arrow-up11.03K
arrow-up11.03KimageChad scrapersh.itjust.works@dramaticcat@sh.itjust.works to Lemmy Shitpost@lemmy.world • 2 years agomessage-square97
minus-square@ViscloReader@lemmy.worldlinkfedilink71•2 years agoJust a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
minus-square@DAMunzy@lemmy.dbzer0.comlinkfedilink17•2 years agoBut, but, norobots.txt! Chad doesn’t care!
minus-square@PoolloverNathan@programming.devlinkfedilinkEnglish6•2 years agouses curl manually instead
Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
But, but, norobots.txt!
Chad doesn’t care!
uses curl manually instead