Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If by offender you mean the bot, then I'm confused. The bot asked for robots.txt in a plaintext format. The bot was delivered some binary garbage that it couldn't parse. The bot continued to ask for an updated robots.txt, and continued to be told that the binary garbage was the intended content. What more was it supposed to do, exactly? The offending party here is the broken hosting platform.


I don't think banning the offender is in order to punish them for doing something wrong so much as to manually ensure it's not crawling your site




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: