Search Results for "scraper"

The aim of this library is to be a comprehensive source for extracting all HTML-embedded metadata. Currently, it supports Schema.org microdata using a third-party library, a native BEPress, Dublin Core, Highwire Press, JSON-LD, Open Graph, Twitter, EPrints, PRISM, and COinS implementation, and some general metadata that doesn't belong to a particular standard (for instance, the content of the title tag, or meta description tags). Planned is support for RDFa, AGLS, and other yet unheard-of...

scrape-it is a Node.js library for converting web pages and HTML documents into structured JavaScript objects. Developers describe the desired output through a declarative schema based on CSS selectors. Fields can extract text, raw HTML, element attributes, repeated items, and nested lists. Conversion functions can transform captured values into dates, numbers, or other application-specific types. The library supports promises, async and await workflows, and callback-based usage. It can...

...It aims to prioritize user privacy and autonomy by minimizing tracking, blocking unnecessary requests, and restricting potentially harmful web technologies such as JavaScript execution. The platform operates as both a browser and a network service, capable of acting as a proxy, scraper, and content filtering system for other applications. Stealth introduces peer-to-peer networking features that allow users to share cached content and reduce bandwidth usage, which can be especially beneficial in constrained environments. It also includes advanced content filtering and optimization mechanisms that strip unwanted or malicious elements from web pages before rendering.

Supports strings, arrays, arrays of objects, and nested object structures. The schema is not tied to the structure of the page you're scraping, allowing you to pull the data in the structure of your choosing. The API is entirely composable, giving you great flexibility in how you scrape each page. Paginate through websites, scraping each page. X-ray also supports a request delay and a pagination limit. Scraped pages can be streamed to a file, so if there's an error on one page, you won't...