Search Results for "python web crawler"
Sort By:
Showing 58 open source projects for "python web crawler"
View related business solutions-
$300 Free Credits to Build on Google CloudStart your next project with $300 in free Google Cloud credit. Spin up VMs, run containers, query petabytes in BigQuery, or build agents with Gemini Enterprise Agent Platform. Once your credits are used, keep building with 20+ always-free tier products including Compute Engine, Cloud Storage, GKE, and Cloud Run functions. No commitment required—just sign up and start building.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
Hermes Web UI
The best way to use Hermes Agent from the web or from your phone
...It offers a clean, multi-panel layout that includes chat interaction, session management, and workspace file browsing. The interface allows users to manage agent sessions, configure models, and interact with persistent memory systems directly from a web environment. It is built using simple technologies like Python and vanilla JavaScript, avoiding complex frontend frameworks. The UI supports real-time interaction, context tracking, and visualization of token usage. It connects to a self-hosted agent that continuously learns and evolves over time. The project emphasizes usability, accessibility, and seamless integration with existing workflows. -
2
Portia SDK Python
Portia Labs Python SDK for building agentic workflows
portia‑sdk‑python is an open-source Python SDK by Portia Labs for creating reliable, stateful, authenticated multi-agent AI workflows. It supports tool-backed agents capable of real-world interactions—like web browsing, API access, and human-in-the-loop clarifications—while maintaining transparency and auditability through structured plans and execution hooks. -
3
Agent Development Kit (ADK)
Open-source, code-first Python toolkit for building, evaluating, etc.
ADK Python helps developers verify hardware-backed keys, work with JSON Web Tokens (JWT), and integrate with Android’s Key Attestation infrastructure. -
4
Dendrite
Tools to build web AI agents that can authenticate
Dendrite Python SDK is a toolkit for building web AI agents that can authenticate, interact with, and extract data from any website, facilitating web automation tasks. -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
5
OpenSquilla
Token-Efficient AI Agent with same budget, higher intelligence density
OpenSquilla is a token-efficient microkernel AI agent runtime designed for CLI, web UI, and chat-based workflows. It routes each turn through a shared loop that can select lower-cost models when appropriate while preserving tool dispatch, retries, memory, and decision logging. The project supports multiple LLM providers through a pluggable provider layer, making it adaptable to different model ecosystems. It includes persistent memory, built-in web search, on-device embeddings, and... -
6
DeerFlow
Deep Research framework, combining language models with tools
DeerFlow is an open-source, community-driven “deep research” framework / multi-agent orchestration platform developed by ByteDance. It aims to combine the reasoning power of large language models (LLMs) with automated tool-use — such as web search, web crawling, Python execution, and data processing — to enable complex, end-to-end research workflows. Instead of a monolithic AI assistant, DeerFlow defines multiple specialized agents (e.g. “planner,” “searcher,” “coder,” “report generator”) that collaborate in a structured workflow, allowing tasks like literature reviews, data gathering, data analysis, code execution, and final report generation to be largely automated. ... -
7
Windows Copilot API
Reverse engineered Windows Copilot into an OpenAI-compatible API
Windows Copilot API is an unofficial Python project that exposes the consumer Microsoft Copilot web experience through a programmable interface. It can be used directly as a Python library or launched as a local server compatible with the OpenAI API format. The client supports streamed responses, multi-turn conversations, and conversation identifiers. Browser-based sign-in stores a reusable session and refreshes it automatically for later requests. -
8
Notte
Opensource browser using agents
Notte is an open-source browser framework that enables the development and deployment of web-based AI agents. It introduces a perception layer that transforms web pages into structured, navigable maps described in natural language, allowing agents to interact with the internet more effectively. Notte is designed for building scalable and efficient browser-based AI applications. -
9
OpenWorker
AI that gets your everyday tasks done
OpenWorker is an open-source AI coworker that runs on the desktop and completes practical work instead of only producing chat responses. It can create documents, spreadsheets, reports, web pages, Slack replies, calendar changes, and inbox organization. The agent breaks requested outcomes into steps and works across local files, the terminal, desktop resources, and connected applications. Users can bring API keys for multiple commercial or open-weight model providers, or run locally through... -
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
10
Composio
Composio equip's your AI agents & LLMs
Empower your AI agents with Composio - a platform for managing and integrating tools with LLMs & AI agents using Function Calling. Equip your agent with high-quality tools & integrations without worrying about authentication, accuracy, and reliability in a single line of code. -
11
Sinas
Open-source platform for building AI agents and serverless automation
Sinas is an open-source platform for building AI agents and serverless automation with fine-grained access control. It provides a self-hosted backend where developers can configure agents, connect LLM providers, write Python functions, and trigger workflows through webhooks or schedules. The platform supports isolated container execution for functions, which helps separate automation logic from the rest of the system. It also includes reusable skills, state stores, document collections, database connections, and embeddable UI components. Sinas can be managed through a web console or declarative YAML configuration, making it suitable for both interactive administration and GitOps-style workflows. ... -
12
OpenHands
Open-source autonomous AI software engineer
Welcome to OpenHands (formerly OpenDevin), an open-source autonomous AI software engineer who is capable of executing complex engineering tasks and collaborating actively with users on software development projects. Use AI to tackle the toil in your backlog, so you can focus on what matters: hard problems, creative challenges, and over-engineering your dotfiles We believe agentic technology is too important to be controlled by a few corporations. So we're building all our agents in the... -
13
Browser Use
Make websites accessible for AI agents
...Overall, Browser Use transforms web interaction into an intelligent, programmable workflow driven by AI agents. -
14
/last30days
Claude Code skill that researches any topic across Reddit + X
/last30days is a specialized Claude Code skill designed to research current trends and practices across Reddit, X, and the wider web from the last 30 days, synthesize that data, and produce copy-paste-ready prompts or summaries that reflect what the community is actually talking about now. Rather than returning generic model responses, it intelligently analyzes social media and community discussions to identify what’s genuinely trending or working in practice across topics ranging from... -
15
MolmoWeb
Open multimodal web agent built by Ai2
MolmoWeb is an open-source multimodal web agent designed to autonomously navigate and interact with web browsers using vision-language models, representing a significant step toward fully agentic AI systems that can operate in real-world digital environments. The system takes natural language instructions and translates them into sequences of browser actions such as clicking, typing, scrolling, and navigating, effectively performing tasks on behalf of the user. Unlike traditional automation... -
16
GPT All Star
AI-powered code generation tool for scratch development of web apps
AI-powered code generation tool for scratch development of web applications with a team collaboration of autonomous AI agents. This is a research project, and its primary value is to explore the possibility of autonomous AI agents. -
17
PentestAgent
AI agent framework for black-box security testing
PentestAgent is an open-source autonomous security testing platform designed to help organizations identify vulnerabilities and assess security posture by simulating real-world attack scenarios without manual intervention. It brings a modular and automated approach to penetration testing by orchestrating a suite of tools and scripts that can emulate common exploitation techniques, reconnaissance workflows, and post-exploitation activities across targets. Users configure rules, policies, and... -
18
A.I.G
Full-stack AI Red Teaming platform
AI-Infra-Guard is a powerful open-source security platform from Tencent’s Zhuque Lab designed to assess the safety and resilience of AI infrastructures, codebases, and components through automated scanning and evaluation tools. It brings together AI infrastructure vulnerability scanning, MCP server risk analysis, and jailbreak evaluation into a unified workflow so that enterprises and individuals can identify critical security issues without relying on external services. Users can deploy it... -
19
OpenAdapt
Open Source Generative Process Automation
OpenAdapt is the open source software adapter between Large Multimodal Models (LMMs) and traditional desktop and web Graphical User Interfaces (GUIs). OpenAdapt learns to automate your desktop and web workflows by observing your demonstrations. Spend less time on repetitive tasks and more on work that truly matters. Boost team productivity in HR operations. Automate candidate sourcing using LinkedIn Recruiter, LinkedIn Talent Solutions, GetProspect, Reply.io, outreach.io, Gmail/Outlook, and... -
20
Agent Reach
Give your AI agent eyes to see the entire internet
Agent Reach is a command-line tool that gives AI agents access to public internet content across multiple platforms. It is designed for agent workflows that need to read and search sources like Twitter, Reddit, YouTube, GitHub, Bilibili, and XiaoHongShu without relying on paid platform APIs. The project focuses on giving AI tools broader visibility into social, video, code, and community data through one interface. It can be useful for research agents, automation workflows, competitive... -
21
notebooklm-py
Unofficial Python API and agentic skill for Google NotebookLM
notebooklm-py is an unofficial Python API and agent-ready integration layer for Google NotebookLM that exposes NotebookLM functionality through code, the command line, and AI agent workflows. Its goal is to provide programmatic access not just to standard notebook operations, but also to many capabilities that are either limited or unavailable in the web interface, making it especially useful for automation and custom pipelines. -
22
Magentic UI
A research prototype of a human-centered web agent
Magentic-UI is a research prototype developed by Microsoft that serves as a human-centered interface powered by a multi-agent system. It enables users to automate complex web tasks, such as browsing, form filling, and data analysis, while maintaining control over the process. The system emphasizes transparency and user involvement, making it suitable for tasks requiring both automation and human oversight. -
23
GPT Researcher
LLM based autonomous agent that does online comprehensive research
Say Hello to GPT Researcher, your AI agent for rapid insights and comprehensive research. GPT Researcher is the leading autonomous agent that takes care of everything from accurate source gathering to organization of research results. -
24
VulnClaw
Based on AI Agent + MCP toolchain + penetration Skill orchestration
VulnClaw is an AI-powered penetration testing agent that turns natural language security goals into structured testing workflows. It combines LLM agents, MCP toolchains, penetration testing skills, and command-line automation to support authorized security assessments. The project can guide information gathering, vulnerability discovery, validation, and report generation while keeping the workflow organized through sessions and tools. Its newer architecture uses a goal-driven solving engine... -
25
PixelRAG
The beginning of scalable pixel-native search
PixelRAG is a visual retrieval-augmented generation system that searches documents by how they look, not only by the text they contain. It renders web pages, PDFs, and images into screenshot tiles, then performs retrieval over those visual representations. This approach preserves layout, tables, charts, diagrams, infographics, and other visual structure that traditional HTML or text parsing can miss. The project includes tools for rendering, chunking, embedding, indexing, and serving visual...
