Search Results for "crawl"
Sort By:
Showing 10 open source projects for "crawl"
View related business solutions-
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
The Web MCP
A powerful Model Context Protocol (MCP) server
Bright Data’s Web MCP server gives AI assistants robust, real-time web capabilities through an MCP interface designed to avoid blocks, rate limits, and CAPTCHAs. It presents search, crawl, navigate, and extraction tools that agents can call directly, replacing brittle scraping prompts with typed operations. The README markets it as a “gateway” to the live web so assistants don’t fall back to stale training data. Bright Data also advertises a getting-started tier with a free monthly allotment, plus options for remote or self-hosted operation depending on governance needs. ... -
2
katana
Fast CLI web crawler for discovering endpoints in modern web apps
Katana is an open source command-line web crawling and spidering framework developed by ProjectDiscovery. It is designed to efficiently crawl websites and web applications in order to discover endpoints, resources, and other useful information that may not be easily visible through manual browsing. Katana focuses on speed and automation, making it suitable for use in security reconnaissance workflows and automated pipelines. Katana supports both standard HTTP crawling and headless browser crawling, allowing it to navigate modern web applications that rely heavily on JavaScript. ... -
3
Puppeteer
Headless Chrome Node.js API
...However, it can also be set to run full or non-headless Chrome or Chromium, simply set the headless option when launching a browser. Many of the things you can do manually in the browser, you can also do with Puppeteer such as generate page screenshots and PDFs, crawl a Single-Page Application, test Chrome extensions and more. -
4
Deep Research
Use any LLMs (Large Language Models) for Deep Research
Deep Research is a local-first research agent that orchestrates multiple LLMs to generate in-depth reports in minutes. It combines “thinking” and “task” model roles with live internet access to plan, search, read, and synthesize findings into structured outputs. The project emphasizes privacy: processing and storage happen locally, avoiding server-side retention of your queries and notes. A simple web UI lets you enter topics and configure models, while the backend streams progress as... -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
5
readwarf
Readwarf: a web reader & workstation
Readwarf: a web reader & workstation -
6
X-RAY
The next web scraper, see through the <html> noise
...Scraped pages can be streamed to a file, so if there's an error on one page, you won't lose what you've already scraped. Start on one page and move to the next easily. The flow is predictable, following a breadth-first crawl through each of the pages. X-ray has support for concurrency, throttles, delays, timeouts and limits to help you scrape any page responsibly. Swap in different scrapers depending on your needs. Currently supports HTTP and PhantomJS driver drivers. In the future, I'd like to see a Tor driver for requesting pages through the Tor network. -
7
pxer
Pixiv crawler userscript for downloading artwork and galleries easily
...It is implemented primarily in client-side JavaScript and runs directly in the browser through a userscript environment, allowing it to integrate seamlessly with Pixiv pages. Pxer provides functionality to crawl and gather images, artwork metadata, and other related content from supported Pixiv pages. It is designed to be accessible even for users who are not developers, emphasizing ease of use and quick setup through browser extensions such as Tampermonkey. Once installed, Pxer adds controls to compatible Pixiv pages so users can trigger batch retrieval of illustrations and related assets. ... -
8
Headless Chrome Crawler
Distributed crawler powered by Headless Chrome
...However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js. Powered by Headless Chrome, the crawler provides simple APIs to crawl dynamic websites. Support both depth-first search and breadth-first search algorithm. Save screenshots for the crawling evidence, emulate devices and user agents, priority queue for crawling efficiency, obey robots.txt, and more. The static crawlers are based on simple requests to HTML files. They are generally fast, but fail scraping the contents when the HTML dynamically changes on browsers. ... -
9
lightcrawler
Website crawler that audits site pages automatically with Lighthouse
Lightcrawler is a command-line tool designed to crawl a website and run automated audits on the discovered pages using Google Lighthouse. It works by starting from a given URL and recursively exploring linked pages to collect a set of pages that should be analyzed. Each discovered page is then evaluated using Lighthouse, which performs checks related to performance, accessibility, and web development best practices. -
Host LLMs in Production With On-Demand GPUsDeploy your model, get an endpoint, pay only for compute time. No GPU provisioning or infrastructure management required.
-
10
SSEP - Site Search Engine PHP-Ajax
A Free site search engine script build with PHP and Ajax.
A Site Search engine script that uses MySQL to store your website's indexed pages, to add Search Functionality to Your Web Site. It is build with PHP and JavaScript, the search results are loaded via Ajax. The search system combine MySQL full text with SQL regexp, and words weight according to their location in the HTML elements, to determine the relevance of the search results. It can be included in any web site.
- Previous
- You're on page 1
- Next
