Search Results for "web crawler source code"
Sort By:
Showing 236 open source projects for "web crawler source code"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
$300 in Free Credit Towards Top Cloud ServicesStart your project in minutes. After credits run out, 20+ products include free monthly usage. Only pay when you're ready to scale.
-
1
GPT Crawler
Crawl a site to generate knowledge files to create your own custom GPT
GPT Crawler is an open-source tool designed to automatically crawl websites and generate structured knowledge that can be used to build AI assistants and retrieval systems. It focuses on extracting high-quality textual content from web pages and preparing it in formats suitable for embedding, indexing, or fine-tuning workflows. The project is especially useful for teams that want to turn documentation sites or knowledge bases into conversational AI backends without building custom scrapers from scratch. ... -
2
Roo Code
Roo Code gives you a whole dev team of AI agents in your code editor
Roo Code is an AI-powered software engineering platform that works interactively in your IDE and autonomously in the cloud to help teams ship faster. It combines a powerful VS Code extension with cloud-based agents that can take on real development tasks across GitHub, Slack, and the web. Designed to work on your terms, Roo Code gives you full control locally while enabling delegation and parallel execution at scale. -
3
Every Code
Local AI coding agent CLI with multi-agent orchestration tools
Every Code (often referred to simply as Code) is a fast, local AI-powered coding agent designed to run directly in the terminal environment. It is a community-driven fork of the Codex CLI, with a strong emphasis on improving real-world developer ergonomics and workflows. Every Code enhances the traditional coding assistant model by introducing multi-agent orchestration, allowing multiple AI agents to collaborate, compare solutions, and refine outputs in parallel. It supports integration with... -
4
Visual Studio Code
Modern IDE and code editor from Microsoft for Mac, Windows, and Linux
Visual Studio Code combines the simplicity of a code editor with what developers need for their core edit-build-debug cycle. It provides comprehensive code editing, navigation, and understanding support along with lightweight debugging, a rich extensibility model, and lightweight integration with existing tools. Visual Studio Code is a distribution of the Code - OSS repository with Microsoft-specific customizations released under a traditional Microsoft product license. Visual Studio Code is... -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
5
Telegram Web A
Telegram Web A, GPL v3
Telegram Web A is a full‐featured web client implementation for Telegram, built from scratch as a lightweight, modern single-page application. It uses a custom front-end framework (called “Teact”) that re-implements React-style paradigms and pairs them with a custom version of the MTProto library (based on GramJS) to interact with Telegram’s backend infrastructure. The project achieved recognition (winning first prize in the Telegram Lightweight Client Contest) and serves as the code base... -
6
WebLLM
Bringing large-language models and chat to web browsers
WebLLM is a modular, customizable javascript package that directly brings language model chats directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU. We can bring a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration. WebLLM offers a minimalist and modular interface to access the chatbot in the browser. The WebLLM package itself does not come... -
7
Proton Web Clients
Monorepo hosting the proton web clients
Proton Web Clients is a monorepo hosting the web applications for Proton’s suite of privacy-focused services, including the core Proton Mail webmail interface and related web apps like Proton Calendar, Proton Drive, Proton Account, Proton VPN, Proton Pass, and other connected tools. It consolidates all web client code, shared modules, dependencies, and development tooling into a single repository, enabling unified maintenance, consistency of design patterns, and efficient evolution of... -
8
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
9
Fluent UI Web
Collection of utilities andcomponents for building web applications
A collection of UX frameworks for creating beautiful, cross-platform apps that share code, design, and interaction behavior. Build for one platform or for all. Everything you need is here. Build your own apps using the same open source components we do, with accessibility, internationalization, and performance included. From tutorials to a fun collection of API references, find what you need to design and develop your own Fluent experience. -
Enterprise-grade ITSM, for every businessFreshservice is an intuitive, AI-powered platform that helps IT, operations, and business teams deliver exceptional service without the usual complexity. Automate repetitive tasks, resolve issues faster, and provide seamless support across the organization. From managing incidents and assets to driving smarter decisions, Freshservice makes it easy to stay efficient and scale with confidence.
-
10
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
11
AWS Toolkit for Visual Studio Code
Local Lambda debug, CodeWhisperer, SAM/CFN syntax, etc.
The AWS Toolkit extension for Visual Studio Code enables you to interact with Amazon Web Services (AWS). Try the AWS Code Sample Catalog to start coding with the AWS SDK. The AWS Explorer provides access to the AWS services that you can work with when using the Toolkit. To see the AWS Explorer, choose the AWS icon in the Activity bar. The Developer Tools panel is a section for developer-focused tooling curated for working in an IDE. The Developer Tools panel can be found underneath the AWS... -
12
Happy Coder
Mobile and Web client for Codex and Claude Code, with realtime voice
Happy is an open-source, cross-platform mobile and web client designed to bring powerful AI coding agents such as Claude Code and Codex to your fingertips no matter where you are. At its core, Happy wraps existing AI coding tools with a unified interface, providing real-time voice interactions, encrypted communication, and seamless device switching between desktop and mobile. -
13
Llama Coder
Open source Claude Artifacts – built with Llama 3.1 405B
Llama Coder is an open-source tool that lets you generate small applications (often React or web apps) from a single natural-language prompt using the Llama 3 family of models. It’s framed as an open-source “Claude Artifacts”-style experience: you describe the app you want, the tool calls an LLM hosted on Together.ai, and you get back a runnable code artifact. -
14
Angular
The modern web developer’s platform
Learn one way to build applications with Angular and reuse your code and abilities to build apps for any deployment target. For web, mobile web, native mobile and native desktop. Achieve the maximum speed possible on the Web Platform today, and take it further, via Web Workers and server-side rendering. Angular puts you in control over scalability. Meet huge data requirements by building data models on RxJS, Immutable.js or another push model. Build features quickly with simple, declarative... -
15
Modern.js
The meta-framework suite designed from scratch for web development
The meta-framework suite was designed from scratch for frontend-focused modern web development. Both "server-side routing" and "client-side routing" are supported. Serverless paradigm of "dynamic and static integrated Web". Low-threshold, full-featured, integrated "BFF" development. Modern.js provides out-of-the-box support for mainstream IDEs such as VS Code, and has features such as automatic detection and repair of Lint problems, access checks before code submission, etc., which can make... -
16
AutoMaker
Start directing AI agents
Automaker is an autonomous AI development studio designed to transform how software is built by allowing developers to describe features, then watching AI agents implement code, tests, commits, and more with minimal manual typing. Instead of writing every line of code by hand, users add feature cards to a Kanban board with natural language descriptions, and AI agents powered by the Claude Agent SDK handle multi-step tasks such as planning, generating code, running tests, and committing to an... -
17
watercrawl
AI-ready web crawler that extracts and structures website content
WaterCrawl is an open source web crawling and data extraction platform designed to transform website content into structured data suitable for machine learning and AI workflows. It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website.... -
18
Pascal Editor
A 3D building editor built with React Three Fiber and WebGPU
Editor is a lightweight, browser-based code editing environment designed to provide a fast and minimal interface for writing and testing code directly in the browser. The project focuses on simplicity and performance, offering a distraction-free workspace that allows developers to quickly prototype ideas without the overhead of a full development environment. It leverages modern web technologies to deliver a responsive and interactive experience, making it suitable for quick experiments,... -
19
Vercel
Optimal workflow for frontend teams
Vercel combines the best developer experience with an obsessive focus on end-user performance. Our platform enables frontend teams to do their best work. Developers love Next.js, the open source React framework Vercel built together with Google and Facebook. Next.js powers the biggest websites like Twilio, for use cases in e-commerce, travel, news, and marketing. Vercel is the best place to deploy any frontend app. Start by deploying with zero configuration to our global edge network. Scale... -
20
Brokk
Brokk brings code intelligence to AI
Brokk is a code intelligence assistant framework designed to let large language models (LLMs) understand code semantically (not just as raw text) so that they can work effectively on large codebases that don’t fit wholly in a prompt context. It helps bridge the gap between LLMs and real-world engineering code by offering tooling to index, analyze, query, and augment code context, so that AI can meaningfully reason about existing code, suggest edits, and navigate across projects. Modular... -
21
Webstudio
Open source website builder and Webflow alternative
Webstudio is an open source visual development platform that enables developers, designers, and cross-functional teams to build modern websites through a powerful visual builder while maintaining full ownership of their data and infrastructure. The project positions itself as a Webflow alternative but emphasizes openness, portability, and deep control over the generated frontend code. -
22
Srcbook
TypeScript-centric app development platform
Srcbook is an open-source, local-first development environment that blends a TypeScript notebook with an AI-assisted app builder, giving developers a flexible space to prototype, explore, and build full web applications with AI support and real execution. It runs locally through a CLI tool backed by Node.js, launching a web interface where users can write and execute TypeScript code interactively, visualize results, and iterate quickly on ideas without leaving the browser. ... -
23
Groq AppGen
Project showcasing Llama 3.3 70B HTML codegen abilities
Groq AppGen is an interactive web application (built with Next.js and TypeScript) that uses Groq’s LLM API to generate or modify web application code based on natural-language prompts. Essentially, you tell the app what kind of web app or page you want (in plain English), and groq-appgen will produce HTML/JSX code scaffolding, layout, and optionally application logic accordingly. It supports iterative feedback: you can refine your prompt, adjust parameters or requirements, and have the app... -
24
The Vibe Companion
Open-source Claude Code/Codex Web UI
The Companion project by The-Vibe-Company is an open-source web-based UI for interacting with Claude Code, offering a fully graphical interface to launch sessions, stream responses, and approve or deny AI tool usage—all via web browsers on desktop or mobile. Instead of working solely through a command-line interface, developers and creators can start, manage, and visualize Claude Code sessions in a modern web environment built with TypeScript, enabling more natural interaction with AI models. ... -
25
Fragments by E2B
Open source template for AI-powered code generation apps w/ sandboxes
Fragments is an open source template designed for building applications where artificial intelligence generates and executes code directly from natural language prompts. It provides a ready-to-use web interface where users can chat with AI models to create code snippets, applications, or scripts in various programming environments. Generated code can be executed safely in isolated sandbox environments using the E2B SDK, which helps protect the host system while running AI-produced programs. ...
