Search Results for "scraping"
Sort By:
Showing 15 open source projects for "scraping"
View related business solutions-
Stop Storing Third-Party Tokens in Your DatabaseRolling your own OAuth token storage can be a security liability. Token Vault securely stores access and refresh tokens from federated providers and handles exchange and renewal automatically. Connected accounts, refresh exchange, and privileged worker flows included.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
1
HeadlessX
The undetected self-hosted browser automation platform
HeadlessX is an open-source, self-hosted browser automation platform designed to run headless browsers for tasks such as web scraping, automation, and testing. The system provides a centralized service that allows developers to programmatically control browser sessions and extract data from websites through a structured API. It is built using modern technologies including Node.js, Next.js, TypeScript, and Playwright, and uses a specialized browser engine called Camoufox based on Firefox. ... -
2
Ulixee Hero
The web browser built for scraping
It's the first modern headless browsers designed specifically for scraping instead of just automated testing. Hero provides access to the W3C DOM specification without the need for Puppeteer's complicated evaluate callbacks and multi-context switching. We've recreated a fully compliant DOM directly in NodeJS allowing you bypass the headaches of previous scraper tools. The powerful Chrome engine sits under the hood, allowing for lightning fast rendering. -
3
Browserless
Deploy headless browsers in Docker
...It lets developers connect existing Puppeteer and Playwright code to remote browser sessions over WebSocket, which helps move heavy browser work away from local machines or application servers. The project also provides REST APIs for common automation tasks such as screenshots, PDF generation, scraping, crawling, and content export. Browserless is useful for teams that need scalable browser execution for testing, data collection, rendering, or AI-agent browsing workflows. Its deployment model supports self-hosting, private infrastructure, queues, concurrency controls, and enterprise-oriented configuration. The project’s main value is turning browser automation into a managed service layer that can be reused across applications and workflows. -
4
crawlee
A web scraping and browser automation library for Node.js
Crawlee is a web scraping and browser automation library. It helps you build reliable crawlers. Fast. Crawlee won't fix broken selectors for you (yet), but it helps you build and maintain your crawlers faster. When a website adds JavaScript rendering, you don't have to rewrite everything, only switch to one of the browser crawlers. When you later find a great API to speed up your crawls, flip the switch back. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
Rod
A Devtools driver for web automation and scraping
Rod is a high-level driver for DevTools Protocol. It's widely used for web automation and scraping. Rod can automate most things in the browser that can be done manually. Chained context design, intuitive to timeout or cancel the long-running task. Auto-wait elements to be ready. Debugging friendly, auto input tracing, remote monitoring headless browser. Thread-safe for all operations. Automatically find or download browser. -
6
chromedp
A faster, simpler way to drive browsers supporting the Chrome DevTools
...Because it communicates directly with Chrome’s debugging interface, chromedp offers high performance and reliable automation compared with tools that rely on intermediary drivers. It is frequently used for web scraping, automated testing, performance monitoring, and browser-based data extraction workflows. -
7
Lightpanda Browser
Lightpanda: the headless browser designed for AI and automation
Lightpanda is an open-source headless browser designed specifically for automation, artificial intelligence workflows, and large-scale web interaction tasks. Unlike traditional browsers that include full graphical rendering engines meant for human users, Lightpanda is built from scratch to operate entirely in headless mode, focusing only on the components required for programmatic web interaction. This design allows it to execute JavaScript and interact with web pages while avoiding the... -
8
Ferrum
Headless Chrome Ruby API
...Because of this low-level access, Ferrum offers greater flexibility and performance compared to traditional WebDriver-based automation tools. It is commonly used for tasks such as web scraping, automated testing, crawling, and screenshot or PDF generation. -
9
Cuprite
Headless Chrome/Chromium driver for Capybara
...By communicating directly with Chromium-based browsers through the DevTools protocol, Cuprite enables faster and more reliable browser automation for testing and scraping tasks. The driver integrates seamlessly with Capybara, allowing developers to write feature tests that simulate real user interactions with web applications. Because it uses headless Chrome by default, Cuprite is well suited for automated test environments and continuous integration pipelines. Developers can also run tests with a visible browser window for debugging purposes during development. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
SecretAgent
The web scraper that's nearly impossible to block
SecretAgent is a headless browser that’s nearly impossible to detect. It achieves this by emulating real users. And it has powerful auto-replay functionality that lets you create and debug scripts in record setting time. -
11
Browser Pool
A Node.js library to easily manage and rotate a pool of web browsers
...We created Browser Pool because we regularly needed to execute tasks concurrently in many headless browsers and their pages, but we did not want to worry about launching browsers, closing browsers, restarting them after crashes and so on. We also wanted to easily and reliably manage the whole browser/page lifecycle. You can use Browser Pool for scraping the internet at scale, testing your website in multiple browsers at the same time or launching web automation robots. -
12
Erik
Erik is an headless browser based on WebKit
Erik is a headless browser based on WebKit, written in Swift, allowing developers to run functional tests and manipulate web pages using JavaScript. -
13
jBrowserDriver
A programmable, embeddable web browser driver
jBrowserDriver is a programmable, embeddable web browser driver compatible with the Selenium WebDriver specification, implemented in pure Java and based on WebKit. -
14
WKZombie
WKZombie is a Swift framework for iOS/OSX to navigate within websites
WKZombie is a Swift framework for iOS/OSX to navigate within websites and collect data without the need of a User Interface or API, also known as a Headless browser. It can be used to run automated tests/snapshots and manipulate websites using Javascript. WKZombie is an iOS/OSX web-browser without a graphical user interface. It was developed as an experiment in order to familiarize myself with using functional concepts written in Swift 4. It incorporates WebKit (WKWebView) for rendering and... -
15
Surf
Stateful programmatic web browsing in Go
Surf is a Go library that implements a virtual web browser, allowing developers to programmatically interact with web pages as a real browser would.
- Previous
- You're on page 1
- Next
