We are the robots

July 20, 2019 in robots.txt

Robots.txt is an interesting file. For years it has been exploited by hackers, pentesting tools writers, crawlers, web scrapers, SEOs, etc. I was thinking the other day what sort of […]