Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I've run into a few sites that "prevent" scraping. I just jump on http://developer.yahoo.com/yql/ and scrape away. The surface level defenses (ip blocking) are usually sufficient, but if there is a real developer behind the scraper, they will get to your content.

Too much prevention could be counter-productive, as you may inadvertently deny the friendly spiders.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: