I can only think of using 80legs as a crawler, since its distributed enough to m... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		akshayubhat on Dec 17, 2010 \| parent \| context \| favorite \| on: Delicious's Data Policy is Like Setting a Museum o... I can only think of using 80legs as a crawler, since its distributed enough to make sure that you don't run into any IP address based rate limitation. But it's just a guess.

slig2 on Dec 17, 2010 | [–]

You probably can't do it, because 80 respects robots.txt AFAIK.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact