Search Results for "python web crawler" - Page 16
Sort By:
Showing 3076 open source projects for "python web crawler"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
1
Tongyi DeepResearch
Tongyi Deep Research, the Leading Open-source Deep Research Agent
DeepResearch (Tongyi DeepResearch) is an open-source “deep research agent” developed by Alibaba’s Tongyi Lab designed for long-horizon, information-seeking tasks. It’s built to act like a research agent: synthesizing, reasoning, retrieving information via the web and documents, and backing its outputs with evidence. The model is about 30.5 billion parameters in size, though at any given token only ~3.3B parameters are active. It uses a mix of synthetic data generation, fine-tuning and... -
2
Anything to NotebookLM
Multi-source content processor for NotebookLM
Qiaomu Anything to NotebookLM is a Claude Code skill that turns many types of source material into structured NotebookLM-ready outputs. It is built for users who want to convert articles, web pages, videos, PDFs, office files, podcasts, images, and search results into more usable study or presentation formats. The project uses natural-language commands, so the user can ask for a podcast, slide deck, mind map, report, quiz, flashcards, or infographic without manually building the workflow. It... -
3
ClawTeam
ClawTeam: Agent Swarm Intelligence (One Command → Full Automation)
ClawTeam is an advanced multi-agent orchestration framework that enables AI agents to form collaborative swarms capable of solving complex tasks autonomously. Instead of relying on a single agent, the system allows a leader agent to spawn and coordinate multiple specialized sub-agents, each responsible for different aspects of a problem. These agents communicate, share insights, and dynamically adapt their strategies based on real-time feedback, creating a form of collective intelligence.... -
4
Datapizza AI
Build reliable Gen AI solutions without overhead
Datapizza AI is a lightweight framework for building modular, multi-agent AI systems that collaborate to solve complex tasks through orchestration and tool usage. The project focuses on simplicity and transparency, enabling developers to construct agent-based workflows without the heavy abstractions and dependencies often found in larger AI frameworks. It provides a flexible architecture where individual agents can be assigned specialized roles, such as web search, reasoning, or... -
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
5
Mailu
Insular email distribution - mail server as Docker images
Mailu is a simple yet full-featured mail server as a set of Docker images. It is free software (both as in free beer and as in free speech), open to suggestions and external contributions. The project aims at providing people with an easily setup, easily maintained and full-featured mail server while not shipping proprietary software nor unrelated features often found in popular groupware. Security, enforced TLS, DANE, MTA-STS, Letsencrypt!, outgoing DKIM, anti-virus scanner, Snuffleupagus,... -
6
buku
Personal mini-web in text
buku is a powerful bookmark manager and a personal textual mini-web. For those who prefer the GUI, bukuserver exposes a browsable front-end on a local web host server. When I started writing it, I couldn't find a flexible command-line solution with a private, portable, merge-able database along with seamless GUI integration. Hence, buku. buku can import bookmarks from the browser(s) or fetch the title, tags and description of a URL from the web. Use your favorite editor to add, compose and... -
7
MasterHttpRelayVPN
Domain-fronted HTTP/SOCKS5 proxy tunneling traffic through Google Apps
MasterHttpRelayVPN is a networking tool designed to route traffic through HTTP-based relay systems, enabling VPN-like functionality over standard web protocols. It focuses on bypassing network restrictions by tunneling data through HTTP requests, making it effective in constrained or filtered environments. The system supports secure communication and can be configured to relay traffic across multiple endpoints. It is particularly useful for users who need to access restricted networks or... -
8
This tool is used to open websites/links by pressing one or more keys on the keyboard, acting as a true shortcut for web pages. When the shortcut keys are pressed, you will be directed to the site previously entered through the main browser set in the system (if the tool does not work after setting the shortcuts, try restarting it).
-
9
CTFd
CTFs as you need them
CTFd is a Capture The Flag framework focusing on ease of use and customizability. It comes with everything you need to run a CTF and it's easy to customize with plugins and themes. Create your own challenges, categories, hints, and flags from the Admin Interface. Dynamic Scoring Challenges. Unlockable challenge support. Challenge plugin architecture to create your own custom challenges. Static & Regex-based flags. Custom flag plugins. Unlockable hints. File uploads to the server or an Amazon... -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
10
Evennia
Python MUD/MUX/MUSH/MU* development system
Evennia is a mature, open-source framework written in Python — specifically designed to build text-based, online multiplayer games such as MUDs, MUCKs, MUSHes, MUXes, and other “MU-style” virtual worlds. Rather than prescribing a rigid game structure, Evennia gives you a bare-bones but powerful foundation: default systems handle networking, database/storage, server management, user accounts, characters, rooms, items, chat channels, and basic commands — but you define the gameplay rules,... -
11
AutoPkg
Automating packaging and software distribution on macOS
AutoPkg is a system that automatically prepares software for distribution to managed clients. Recipes allow you to specify a series of simple actions which combined together can perform complex tasks, similar to Automator workflows or Unix pipes. -
12
Habit Tracker
Habit Tracker for the AI Coding Workshop
Habit Tracker is a personal habit-tracking web application designed to help users build and maintain daily habits through intuitive UI and analytics that visualize progress over time. It runs locally with a FastAPI backend (Python) and a React frontend, storing all data in a lightweight SQLite database so there’s no need for user accounts or cloud storage, which keeps habit data fully private and self-contained. -
13
SAFRS
SqlAlchemy Flask-Restful Swagger Json:API OpenAPI
SAFRS exposes SQLAlchemy database models as a JSON:API web service and generates the corresponding swagger/OpenAPI. Database objects such as the User class from the demo.py example can be extended to include relationships with other objects. The demo_relationship.py contains the following extension of the User class where a relationship with the Book class is implemented. -
14
ddia
Designing Data-Intensive Application
ddia is a Chinese translation and online reading project for “Designing Data-Intensive Applications.” It covers both the first edition and ongoing second-edition translation work. The repository organizes the book content into chapters about data systems, storage, encoding, replication, partitioning, transactions, distributed systems, consistency, and derived data. It is built for readers who want deep systems design material in Chinese with a structured web reading experience. The project... -
15
WeKnora
LLM framework for document understanding and semantic retrieval
WeKnora is an open source framework developed for deep document understanding and semantic information retrieval using large language models. It focuses on analyzing complex and heterogeneous documents by combining multiple processing stages such as multimodal document parsing, vector indexing, and intelligent retrieval. It follows the Retrieval-Augmented Generation (RAG) paradigm, where relevant document segments are retrieved and used by language models to generate accurate, context-aware... -
16
GLM-4.7
Advanced language and coding AI model
GLM-4.7 is an advanced agent-oriented large language model designed as a high-performance coding and reasoning partner. It delivers significant gains over GLM-4.6 in multilingual agentic coding, terminal-based workflows, and real-world developer benchmarks such as SWE-bench and Terminal Bench 2.0. The model introduces stronger “thinking before acting” behavior, improving stability and accuracy in complex agent frameworks like Claude Code, Cline, and Roo Code. GLM-4.7 also advances “vibe... -
17
Omnara
Talk to Your AI Agents from Anywhere
Omnara is an open-source agent control platform that empowers developers to turn autonomous AI tools (e.g., Claude Code, Cursor, GitHub Copilot) into collaborative teammates by offering real-time dashboards, push notifications, and remote guidance across terminals, web, and mobile. Omnara transforms your AI agents (Claude Code, Codex CLI, n8n, and more) from silent workers into communicative teammates. Get real-time visibility into what your agents are doing, and respond to their questions... -
18
CogVideo
Text and image to video generation: CogVideoX and CogVideo
CogVideo is an open-source family of advanced video generation models that can create videos from text, images, or existing video inputs. Built on large-scale Transformer and diffusion architectures, it enables multimodal generation across text-to-video, image-to-video, and video continuation tasks. The latest CogVideoX models offer higher resolution outputs, longer video durations, and improved controllability through prompt engineering. The project includes tools for inference,... -
19
MagicTime
Time-lapse Video Generation Models as Metamorphic Simulators
This repository is the official implementation of MagicTime, a metamorphic video generation pipeline based on the given prompts. The main idea is to enhance the capacity of video generation models to accurately depict the real world through our proposed methods and dataset. Compared to general videos, metamorphic videos contain physical knowledge, long persistence, and strong variation, making them difficult to generate. -
20
AI-Media2Doc
AI tool converting video/audio into structured documents instantly
AI-Media2Doc is a web-based application that uses large language models to convert video and audio content into structured, readable documents in a single workflow. It is designed to transform multimedia inputs into formats such as knowledge notes, summaries, mind maps, and social-style articles, making content easier to review and reuse. AI-Media2Doc emphasizes privacy by processing media locally in the browser using WebAssembly-based ffmpeg, ensuring that original video files are not... -
21
PentestAgent
AI agent framework for black-box security testing
PentestAgent is an open-source autonomous security testing platform designed to help organizations identify vulnerabilities and assess security posture by simulating real-world attack scenarios without manual intervention. It brings a modular and automated approach to penetration testing by orchestrating a suite of tools and scripts that can emulate common exploitation techniques, reconnaissance workflows, and post-exploitation activities across targets. Users configure rules, policies, and... -
22
SimpleTuner
A general fine-tuning kit geared toward image/video/audio diffusion
SimpleTuner is an open-source toolkit designed to simplify the fine-tuning of modern diffusion models for generating images, video, and audio. The project focuses on providing a clear and understandable training environment for researchers, developers, and artists who want to customize generative AI models without navigating complex machine learning pipelines. It supports fine-tuning workflows for models such as Stable Diffusion variants and other diffusion architectures, enabling users to... -
23
ClusterFuzz
Scalable fuzzing infrastructure
ClusterFuzz is a scalable fuzzing infrastructure that finds security and stability issues in software. Google uses ClusterFuzz to fuzz all Google products and as the fuzzing backend for OSS-Fuzz. ClusterFuzz provides many features which help seamlessly integrate fuzzing into a software project's development process. Can run on any size cluster (e.g. OSS-Fuzz instance runs on 100,000 VMs). Fully automatic bug filing, triage and closing for various issue trackers (e.g. Monorail, Jira).... -
24
Feast
Feature Store for Machine Learning
Feast (Feature Store) is an open source feature store for machine learning. Feast is the fastest path to manage existing infrastructure to productionize analytic data for model training and online inference. Make features consistently available for training and serving by managing an offline store (to process historical data for scale-out batch scoring or model training), a low-latency online store (to power real-time prediction), and a battle-tested feature server (to serve pre-computed... -
25
X4G
Fast and modern gateway for VLESS tunneling on WebSocket and XHTTP
X4G is a self-hosted gateway for operating VLESS tunnels over WebSocket and two XHTTP transport modes. It also includes an internal HTTP proxy for clients that need a conventional proxy endpoint. A web dashboard displays service statistics, hourly traffic charts, live connections, activity records, and error logs. Administrators can create unlimited connection links with individual traffic quotas, bandwidth limits, expiration dates, ports, and simultaneous IP restrictions. Subscription...
