Search Results for "pages"
Sort By:
Showing 16 open source projects for "pages"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
1
Ferret
Declarative web scraping
A web scraping system aiming to simplify data extraction from the web. ferret has a declarative query language that makes it easy to focus on the data that you need to get. ferret has the ability to scrape JS rendered pages, handle all page events, and emulate user interactions. the ferret was designed as a library from the ground up. it can be easily embedded into any Go application. ferret helps you to focus on the data you need using an easy-to-learn declarative language. ferret uses Chrome/Chromium via Chrome Devtools Protocol to handle dynamically rendered web pages. ferret is extremely extensible, and creating custom functions and types is super easy. ferret allows users to focus on the data. ... -
2
goclone
Fast CLI tool for cloning entire websites for local browsing offline
goclone is a command-line utility designed to download and mirror complete websites to a local directory for offline access. It retrieves HTML pages, stylesheets, JavaScript files, images, and other assets from a target site and stores them on the user’s computer. It preserves the original site’s structure by maintaining relative links between pages, allowing the mirrored copy to function similarly to the live version when opened locally. Once a site has been cloned, users can browse the pages offline and navigate between them as if they were viewing the site online. goclone is written in Go and leverages concurrency through Go routines to perform downloads efficiently. goclone can also optionally start a local web server to serve the mirrored files for a more realistic browsing experience. ... -
3
ChangeMonitor
ChangeMonitor is an application for monitoring changes on websites
ChangeMonitor is an application for monitoring changes on websites. The recommended way to run ChangeMonitor is using Docker. -
4
crawley
The unix-way web crawler
Crawls web pages and prints any link it can find. Fast HTML SAX-parser (powered by golang.org/x/net/html) Small (below 1500 SLOC), idiomatic, 100% test-covered codebase. Grabs most of useful resources URLs (pics, videos, audios, forms, etc...) Found URLs are streamed to stdout and guaranteed to be unique (with fragments omitted) Scan depth (limited by starting host and path, by default - 0) can be configured. -
99.99% Uptime for MySQL and PostgreSQL DatabasesCloud SQL Enterprise Plus delivers near-zero downtime with 35 days of point-in-time recovery. Supports MySQL, PostgreSQL, and SQL Server.
-
5
Pinchtab
High-performance browser automation bridge and orchestrator
Pinchtab is a lightweight browser automation backend built specifically for AI agents that need efficient, programmatic web control. Implemented as a small standalone HTTP server, it allows any agent or script to interact with web pages using simple API calls instead of heavyweight browser frameworks. The tool emphasizes accessibility-first snapshots that dramatically reduce token usage compared to screenshot-based approaches, making it cost-effective for large-scale automation. It launches and manages its own Chrome instance while remaining framework-agnostic, so it can be used with any language or agent system. ... -
6
chromedp
A faster, simpler way to drive browsers supporting the Chrome DevTools
...Using chromedp, developers can automate tasks such as navigating websites, filling out forms, clicking elements, capturing screenshots, and extracting structured data from pages. The library typically operates in headless mode but can also run with a visible browser window when needed for debugging or development. Because it communicates directly with Chrome’s debugging interface, chromedp offers high performance and reliable automation compared with tools that rely on intermediary drivers. It is frequently used for web scraping, automated testing, performance monitoring, and browser-based data extraction workflows. -
7
Geziyor
Blazing fast Go framework for web crawling and data scraping tasks
Geziyor is a high-performance web crawling and web scraping framework built for the Go programming language. It is designed to help developers crawl websites and extract structured information from web pages efficiently. It focuses on speed and scalability, allowing large numbers of requests to be processed concurrently. Geziyor supports use cases such as data mining, monitoring web content, and automated testing workflows. It provides a flexible architecture where developers define parsing functions that process responses and extract the desired data. ... -
8
Rod
A Devtools driver for web automation and scraping
Rod is a high-level driver for DevTools Protocol. It's widely used for web automation and scraping. Rod can automate most things in the browser that can be done manually. Chained context design, intuitive to timeout or cancel the long-running task. Auto-wait elements to be ready. Debugging friendly, auto input tracing, remote monitoring headless browser. Thread-safe for all operations. Automatically find or download browser. High-level helpers like WaitStable, WaitRequestIdle,... -
9
go-dork
Fast Go-based CLI scanner for running automated search engine dorks
...It supports several major search engines and enables users to switch between them depending on the target or query requirements. go-dork can retrieve results from multiple pages of search results and process them sequentially for broader coverage during scans. go-dork also supports custom HTTP headers and proxy configuration, which can help users work around restrictions such as captchas or filtering mechanisms. Because it is a command-line tool, it can be integrated into automation pipelines or chained with other security tools to streamline reconnaissance workflows. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
crawlergo
Headless Chrome crawler for collecting URLs for vulnerability scans
crawlergo is a browser-based web crawler designed to collect URLs and request data that can be used by web vulnerability scanning tools. It uses a Chrome headless environment to render web pages and observe behavior during the DOM rendering stage in order to capture as many accessible endpoints as possible. By monitoring the page lifecycle and interacting with web elements, the crawler automatically triggers JavaScript events and navigational actions that would normally occur during real user interaction. It also automatically fills and submits forms, helping discover hidden routes or parameters that might otherwise be missed by traditional crawlers. crawlergo includes a built-in URL de-duplication system that removes repeated or pseudo-static links while maintaining fast crawling speeds for large websites. crawlergo also analyzes page content to extract links and resources from multiple sources, including JavaScript files, comments, and configuration files. -
11
Till
DataHen Till is a companion tool to your existing web scraper
...However, as you try to scale it up, it gets exponentially difficult. Scraping 10,000 records can easily be done with simple web scraper scripts in any programming language, but as you try to scrape millions of pages, you would need to architect and build features on your web scraping script that allows you to scale, maintain and unblock your scrapers. Scraping to millions or even billions of records requires much more pre-planning. It's not simply running your existing web scraper script in a bigger CPU/Ram machine. More thoughts are needed. -
12
websocketd
Turn any program that uses STDIN/STDOUT into a WebSocket server
...Serves your static HTML, JavaScript, etc. Route different URLs to different programs. Dynamic generate content over HTTP too. Out-of-the-box support for serving content using HTTPS and WSS. Restrict which pages can make WebSocket connections. Interact with your WebSocket programs before you've built your frontend. -
13
proxypool
Proxy crawler that aggregates, tests, and serves usable proxy nodes
proxypool is an open source proxy aggregation tool that automatically collects proxy node information from publicly available sources on the internet. It crawls different sources such as Telegram channels, subscription links, and publicly accessible web pages to gather proxy configurations. After collecting these nodes, proxypool processes them by removing duplicates and verifying whether each node is functional. proxypool then provides a usable list of proxy nodes that have passed availability checks. proxypool supports several popular proxy protocols, allowing it to work with multiple types of proxy infrastructures. ... -
14
Rendora
dynamic server-side rendering using headless Chrome
Rendora is a dynamic renderer to provide zero-configuration server-side rendering mainly to web crawlers in order to effortlessly improve SEO for websites developed in modern Javascript frameworks such as React.js, Vue.js, Angular.js, etc. Rendora works totally independently of your frontend and backend stacks. Rendora can be seen as a reverse HTTP proxy server sitting between your backend server (e.g. Node.js/Express.js, Python/Django, etc...) and potentially your frontend proxy server... -
15
Surf
Stateful programmatic web browsing in Go
Surf is a Go library that implements a virtual web browser, allowing developers to programmatically interact with web pages as a real browser would. -
16
dataflowkit
Golang framework for scraping data from web pages
Golang Web Scraper library for extracting data from web pages. Save results as CSV, JSON, XML
- Previous
- You're on page 1
- Next
