Search Results for "crawling"

Most powerful, popular and production crawling/scraping package for Node, happy hacking.

...The goal is to expose a few simple methods that mimic user actions (like goto, type and click), with an API that feels synchronous for each block of scripting, rather than deeply nested callbacks. It was originally designed for automating tasks across sites that don't have APIs, but is most often used for UI testing and crawling. Segment started with an open source project. Since then, we’ve open sourced hundreds of our repos. We want to continue supporting the community by publishing our code and other developers’ awesome open source projects. We use these open source projects as the foundations of our infrastructure to handle billions of API calls per day and to allow us to rapidly build and test code on the client.