Search Results for "python web crawler"
Sort By:
Showing 58 open source projects for "python web crawler"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Hermes Web UI
The best way to use Hermes Agent from the web or from your phone
...It offers a clean, multi-panel layout that includes chat interaction, session management, and workspace file browsing. The interface allows users to manage agent sessions, configure models, and interact with persistent memory systems directly from a web environment. It is built using simple technologies like Python and vanilla JavaScript, avoiding complex frontend frameworks. The UI supports real-time interaction, context tracking, and visualization of token usage. It connects to a self-hosted agent that continuously learns and evolves over time. The project emphasizes usability, accessibility, and seamless integration with existing workflows. -
2
Portia SDK Python
Portia Labs Python SDK for building agentic workflows
portia‑sdk‑python is an open-source Python SDK by Portia Labs for creating reliable, stateful, authenticated multi-agent AI workflows. It supports tool-backed agents capable of real-world interactions—like web browsing, API access, and human-in-the-loop clarifications—while maintaining transparency and auditability through structured plans and execution hooks. -
3
Dendrite
Tools to build web AI agents that can authenticate
Dendrite Python SDK is a toolkit for building web AI agents that can authenticate, interact with, and extract data from any website, facilitating web automation tasks. -
4
Agent Development Kit (ADK)
Open-source, code-first Python toolkit for building, evaluating, etc.
ADK Python helps developers verify hardware-backed keys, work with JSON Web Tokens (JWT), and integrate with Android’s Key Attestation infrastructure. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
5
Notte
Opensource browser using agents
Notte is an open-source browser framework that enables the development and deployment of web-based AI agents. It introduces a perception layer that transforms web pages into structured, navigable maps described in natural language, allowing agents to interact with the internet more effectively. Notte is designed for building scalable and efficient browser-based AI applications. -
6
OpenSquilla
Token-Efficient AI Agent with same budget, higher intelligence density
OpenSquilla is a token-efficient microkernel AI agent runtime designed for CLI, web UI, and chat-based workflows. It routes each turn through a shared loop that can select lower-cost models when appropriate while preserving tool dispatch, retries, memory, and decision logging. The project supports multiple LLM providers through a pluggable provider layer, making it adaptable to different model ecosystems. It includes persistent memory, built-in web search, on-device embeddings, and... -
7
DeerFlow
Deep Research framework, combining language models with tools
DeerFlow is an open-source, community-driven “deep research” framework / multi-agent orchestration platform developed by ByteDance. It aims to combine the reasoning power of large language models (LLMs) with automated tool-use — such as web search, web crawling, Python execution, and data processing — to enable complex, end-to-end research workflows. Instead of a monolithic AI assistant, DeerFlow defines multiple specialized agents (e.g. “planner,” “searcher,” “coder,” “report generator”) that collaborate in a structured workflow, allowing tasks like literature reviews, data gathering, data analysis, code execution, and final report generation to be largely automated. ... -
8
notebooklm-py
Unofficial Python API and agentic skill for Google NotebookLM
notebooklm-py is an unofficial Python API and agent-ready integration layer for Google NotebookLM that exposes NotebookLM functionality through code, the command line, and AI agent workflows. Its goal is to provide programmatic access not just to standard notebook operations, but also to many capabilities that are either limited or unavailable in the web interface, making it especially useful for automation and custom pipelines. -
9
Sinas
Open-source platform for building AI agents and serverless automation
Sinas is an open-source platform for building AI agents and serverless automation with fine-grained access control. It provides a self-hosted backend where developers can configure agents, connect LLM providers, write Python functions, and trigger workflows through webhooks or schedules. The platform supports isolated container execution for functions, which helps separate automation logic from the rest of the system. It also includes reusable skills, state stores, document collections, database connections, and embeddable UI components. Sinas can be managed through a web console or declarative YAML configuration, making it suitable for both interactive administration and GitOps-style workflows. ... -
99.99% Uptime for MySQL and PostgreSQL DatabasesCloud SQL Enterprise Plus delivers near-zero downtime with 35 days of point-in-time recovery. Supports MySQL, PostgreSQL, and SQL Server.
-
10
Agent Reach
Give your AI agent eyes to see the entire internet
Agent Reach is a command-line tool that gives AI agents access to public internet content across multiple platforms. It is designed for agent workflows that need to read and search sources like Twitter, Reddit, YouTube, GitHub, Bilibili, and XiaoHongShu without relying on paid platform APIs. The project focuses on giving AI tools broader visibility into social, video, code, and community data through one interface. It can be useful for research agents, automation workflows, competitive... -
11
OpenHands
Open-source autonomous AI software engineer
Welcome to OpenHands (formerly OpenDevin), an open-source autonomous AI software engineer who is capable of executing complex engineering tasks and collaborating actively with users on software development projects. Use AI to tackle the toil in your backlog, so you can focus on what matters: hard problems, creative challenges, and over-engineering your dotfiles We believe agentic technology is too important to be controlled by a few corporations. So we're building all our agents in the... -
12
Composio
Composio equip's your AI agents & LLMs
Empower your AI agents with Composio - a platform for managing and integrating tools with LLMs & AI agents using Function Calling. Equip your agent with high-quality tools & integrations without worrying about authentication, accuracy, and reliability in a single line of code. -
13
PixelRAG
The beginning of scalable pixel-native search
PixelRAG is a visual retrieval-augmented generation system that searches documents by how they look, not only by the text they contain. It renders web pages, PDFs, and images into screenshot tiles, then performs retrieval over those visual representations. This approach preserves layout, tables, charts, diagrams, infographics, and other visual structure that traditional HTML or text parsing can miss. The project includes tools for rendering, chunking, embedding, indexing, and serving visual... -
14
Browser Use
Make websites accessible for AI agents
...Overall, Browser Use transforms web interaction into an intelligent, programmable workflow driven by AI agents. -
15
GPT Researcher
LLM based autonomous agent that does online comprehensive research
Say Hello to GPT Researcher, your AI agent for rapid insights and comprehensive research. GPT Researcher is the leading autonomous agent that takes care of everything from accurate source gathering to organization of research results. -
16
OpenWorker
AI that gets your everyday tasks done
OpenWorker is an open-source AI coworker that runs on the desktop and completes practical work instead of only producing chat responses. It can create documents, spreadsheets, reports, web pages, Slack replies, calendar changes, and inbox organization. The agent breaks requested outcomes into steps and works across local files, the terminal, desktop resources, and connected applications. Users can bring API keys for multiple commercial or open-weight model providers, or run locally through... -
17
GPT All Star
AI-powered code generation tool for scratch development of web apps
AI-powered code generation tool for scratch development of web applications with a team collaboration of autonomous AI agents. This is a research project, and its primary value is to explore the possibility of autonomous AI agents. -
18
Magentic UI
A research prototype of a human-centered web agent
Magentic-UI is a research prototype developed by Microsoft that serves as a human-centered interface powered by a multi-agent system. It enables users to automate complex web tasks, such as browsing, form filling, and data analysis, while maintaining control over the process. The system emphasizes transparency and user involvement, making it suitable for tasks requiring both automation and human oversight. -
19
AskUI Vision Agent
Enable AI to control your desktop, mobile and HMI devices
AskUI’s Vision Agent is an automation framework that allows you—and AI agents—to control real desktops, mobile devices, and HMI systems by perceiving the UI and performing actions like clicking, typing, scrolling, and drag-and-drop. It is designed for multi-platform compatibility and supports multiple AI models so you can tailor perception and decision-making to your workload. The repository presents a feature overview, sample media, and frequent release notes, which show ongoing... -
20
OpenAdapt
Open Source Generative Process Automation
OpenAdapt is the open source software adapter between Large Multimodal Models (LMMs) and traditional desktop and web Graphical User Interfaces (GUIs). OpenAdapt learns to automate your desktop and web workflows by observing your demonstrations. Spend less time on repetitive tasks and more on work that truly matters. Boost team productivity in HR operations. Automate candidate sourcing using LinkedIn Recruiter, LinkedIn Talent Solutions, GetProspect, Reply.io, outreach.io, Gmail/Outlook, and... -
21
VulnClaw
Based on AI Agent + MCP toolchain + penetration Skill orchestration
VulnClaw is an AI-powered penetration testing agent that turns natural language security goals into structured testing workflows. It combines LLM agents, MCP toolchains, penetration testing skills, and command-line automation to support authorized security assessments. The project can guide information gathering, vulnerability discovery, validation, and report generation while keeping the workflow organized through sessions and tools. Its newer architecture uses a goal-driven solving engine... -
22
Windows Copilot API
Reverse engineered Windows Copilot into an OpenAI-compatible API
Windows Copilot API is an unofficial Python project that exposes the consumer Microsoft Copilot web experience through a programmable interface. It can be used directly as a Python library or launched as a local server compatible with the OpenAI API format. The client supports streamed responses, multi-turn conversations, and conversation identifiers. Browser-based sign-in stores a reusable session and refreshes it automatically for later requests. -
23
Flowly AI
Flowly is 100x faster than OpenClaw
Flowly is an open-source personal AI assistant that runs locally on your machine and connects to multiple communication platforms like Telegram, WhatsApp, Discord, and Slack. It acts as a centralized AI system that can perform tasks such as web browsing, file management, command execution, scheduling, and more—all while keeping your data private. Designed for flexibility, Flowly supports multiple AI providers and models through LiteLLM, allowing users to customize how their assistant... -
24
MolmoWeb
Open multimodal web agent built by Ai2
MolmoWeb is an open-source multimodal web agent designed to autonomously navigate and interact with web browsers using vision-language models, representing a significant step toward fully agentic AI systems that can operate in real-world digital environments. The system takes natural language instructions and translates them into sequences of browser actions such as clicking, typing, scrolling, and navigating, effectively performing tasks on behalf of the user. Unlike traditional automation... -
25
SWE-agent
SWE-agent takes a GitHub issue and tries to automatically fix it
SWE-agent turns LMs (e.g. GPT-4) into software engineering agents that can resolve issues in real GitHub repositories. On the SWE-bench, the SWE-agent resolves 12.47% of issues, achieving state-of-the-art performance on the full test set. We accomplish our results by designing simple LM-centric commands and feedback formats to make it easier for the LM to browse the repository, and view, edit, and execute code files. We call this an Agent-Computer Interface (ACI).
