Search Results for "extensible web spider"
Sort By:
Showing 13 open source projects for "extensible web spider"
View related business solutions-
$300 Free Credits to Build on Google CloudStart your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Caddy
Powerful, enterprise-ready, open source web server w/ automatic HTTPS
...Caddy is the only web server that uses HTTPS automatically and by default. It automatically renews TLS certificates, staples OCSP responses and more. Though used mostly as an HTTPS server, Caddy can be used to run Go applications, offering automated documentation, graceful on-line config changes via API and more to these apps. Caddy is very extensible, with a powerful plugin system unlike any other web server. -
2
Ferret
Declarative web scraping
A web scraping system aiming to simplify data extraction from the web. ferret has a declarative query language that makes it easy to focus on the data that you need to get. ferret has the ability to scrape JS rendered pages, handle all page events, and emulate user interactions. the ferret was designed as a library from the ground up. it can be easily embedded into any Go application. ferret helps you to focus on the data you need using an easy-to-learn declarative language. ferret uses Chrome/Chromium via Chrome Devtools Protocol to handle dynamically rendered web pages. ferret is extremely extensible, and creating custom functions and types is super easy. ferret allows users to focus on the data. ... -
3
XRAY
XRay for recon, mapping and OSINT gathering from public networks
XRAY is a modular security toolset that helps developers and security professionals analyze, fuzz, and test web applications, protocols, and network services for vulnerabilities. It provides a framework for writing and executing inspection modules that can parse structured data (JSON, XML, HTML), traverse graphs of endpoints, and perform intelligent probing guided by discovered surface area. XRay is typically used as a reconnaissance and vulnerability discovery engine in red-team or app-security workflows: it leverages extensible plugins to adapt to different protocols, inject payloads, and detect common bug classes such as injection flaws, misconfigurations, and unsafe endpoints. ... -
4
Grafana
Leading open-source visualization and observability platform
Grafana OSS is the leading open-source platform for visualization and observability. It enables teams to query, visualize, alert on, and explore telemetry data from multiple sources in a single interface. With support for 100+ data source plugins—including Prometheus, Loki, Elasticsearch, InfluxDB, SQL/NoSQL databases, and OpenTelemetry—Grafana helps teams correlate metrics, logs, and traces across applications and infrastructure. Users can build interactive dashboards with rich... -
AI-powered service management for IT and enterprise teamsGive your IT, operations, and business teams the ability to deliver exceptional services—without the complexity. Maximize operational efficiency with refreshingly simple, AI-powered Freshservice.
-
5
Echo
High performance, minimalist Go web framework
High performance, extensible, minimalist Go web framework. Highly optimized HTTP router with zero dynamic memory allocation which smartly prioritizes routes. Build robust and scalable RESTful API, easily organized into groups. Automatically install TLS certificates from Let's Encrypt. HTTP/2 support improves speed and provides better user experience. Many built-in middleware to use, or define your own. -
6
Wave Terminal
An open-source, cross-platform terminal for seamless workflows
Wave is an open-source AI-native terminal built for seamless workflows. Wave isn't just another terminal emulator; it's a rethink on how terminals are built. Wave combines the command line with the power of the open web to help veteran CLI users and new developers alike. Keep development moving forward without the copy, pasting, saving, and exporting headache. Built on an open web framework that is fully extensible. Quickly edit code on a local or remote machine with the same editor that powers VSCode. Great alternative to vim for new terminal users or quick updates. ... -
7
PhoneInfoga
Information gathering framework for phone numbers
PhoneInfoga is an open-source intelligence framework focused on gathering and analyzing information related to international phone numbers. The tool aggregates data from multiple scanners and external services to provide contextual intelligence such as country, carrier, line type, and potential VoIP provider details. It is designed primarily for investigators, analysts, and security researchers who need structured phone-number reconnaissance rather than real-time tracking. PhoneInfoga... -
8
Crawlab
Distributed web crawler admin platform for spiders management
Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. Please use docker-compose to one-click to start up. By doing so, you don't even have to configure MongoDB database. The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. -
9
Octant
Highly extensible platform for developers
Octant is an open source developer-centric web interface for Kubernetes that lets you inspect a Kubernetes cluster and its applications. Provide a visual interface to managing Kubernetes that complements and extends existing tools like kubectl and customize. Add information to your cluster views through Octant’s plug-in system. Support a variety of debugging features such as filtering labels and streaming container logs to be part of the Kubernetes development toolkit. Learn about why Octant... -
Stop Cyber Threats with VM-Series Next-Gen Firewall on AzureGain integrated visibility across all traffic in a single pass. Deploy Palo Alto Networks VM-Series to determine application identity and content while automating security policy updates via rich APIs.
-
10
GoSpider
Gospider - Fast web spider written in Go
GoSpider - Fast web spider written in Go. Fast web crawling. Brute force and parse sitemap.xml. Parse robots.txt. Generate and verify link from JavaScript files. Link Finder. Find AWS-S3 from response source. Find subdomains from the response source. Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault. Format output easy to Grep. Support Burp input. -
11
gocrawl
Polite concurrent web crawler library for Go with flexible hooks
gocrawl is a lightweight web crawling library written in the Go programming language that enables developers to build custom web crawlers and data extraction tools. gocrawl focuses on providing a minimal yet powerful crawling engine that can be easily extended and adapted for different web scraping or indexing tasks. It is designed to be polite when accessing websites by respecting crawling rules such as robots.txt policies and applying crawl delays for each host. It executes requests... -
12
Ponzu
Headless CMS with automatic JSON API
Ponzu is a powerful and efficient open-source HTTP server framework and CMS. It provides automatic, free, and secure HTTP/2 over TLS (certificates obtained via Let's Encrypt), a useful CMS and scaffolding to generate content editors, and a fast HTTP API on which to build modern applications. Ponzu is released under the BSD-3-Clause license. With the rise in popularity of web/mobile apps connected to JSON HTTP APIs, better tools to support the development of content servers and management... -
13
GOPA
GOPA, a spider written in Golang, for Elasticsearch
GOPA, a spider written in Golang, for Elasticsearch. Lightweight, low footprint, memory requirement should, be 100MB. Easy to deploy, no runtime or dependency required. Easy to use, no programming or script ability needed, out-of-box features. First of all, get it, two opinions: download the pre-built package or compile it yourself. Besides Elasticsearch, Gopa doesn't require any other dependencies, just simply run ./gopa to start the program. It's safety to press ctrl+c to stop the current...
- Previous
- You're on page 1
- Next
