Search Results for "web crawler source code" - Page 2
Sort By:
Showing 3459 open source projects for "web crawler source code"
View related business solutions-
Gemini 3 and 200+ AI Models on One PlatformBuild generative AI apps with Vertex AI. Switch between models without switching platforms.
-
$300 in Free Credit Towards Top Cloud ServicesStart your project in minutes. After credits run out, 20+ products include free monthly usage. Only pay when you're ready to scale.
-
1
whatsapp-web.js
WhatsApp library for NodeJS that connects through the browser app
A WhatsApp client library for NodeJS that connects through the WhatsApp Web browser app. Programmatically control WhatsApp whether you're running user or business accounts. It uses Puppeteer to run a real instance of Whatsapp Web to avoid getting blocked. Programmatically control WhatsApp whether you're running user or business accounts. Whatsapp-web.js connects to an official version of WhatsApp Web under the hood, reducing ban risks. The object-oriented approach makes it easy to get... -
2
claude-code-transcripts
Tools for publishing transcripts for Claude Code sessions
claude-code-transcripts is a command-line utility that takes session files exported from Claude Code (in JSON or JSONL format) and turns them into clean, navigable HTML transcripts that can be viewed in any modern web browser. It is designed to make the often dense and verbose outputs from AI coding sessions easier to read, share, and archive by breaking conversations into paginated, annotated pages with navigable timelines of prompts and responses. Users can run this tool locally or fetch... -
3
Web Experience Toolkit
Open source code library for building innovative websites
Web Experience Toolkit (WET): Open source code library for building innovative websites that are accessible, usable, interoperable, mobile-friendly and multilingual. This collaborative open source project is led by the Government of Canada. A collection of flexible and themeable templates and reusable components. A collaborative open source project led by the Government of Canada. -
4
PHP QR Code
A PHP QR Code generator and reader with a user-friendly API
chillerlan/php-qrcode is a modern, flexible PHP library for generating QR codes. It supports various customization options such as size, encoding, error correction, and logo embedding. The library is PSR-compliant and built for ease of use in modern PHP projects, making it suitable for generating QR codes in web apps, receipts, and authentication systems. -
Earn up to 16% annual interest with Nexo.Generate interest, access liquidity without selling, and execute trades seamlessly. All in one platform. Geographic restrictions, eligibility, and terms apply.
-
5
katana
Fast CLI web crawler for discovering endpoints in modern web apps
Katana is an open source command-line web crawling and spidering framework developed by ProjectDiscovery. It is designed to efficiently crawl websites and web applications in order to discover endpoints, resources, and other useful information that may not be easily visible through manual browsing. Katana focuses on speed and automation, making it suitable for use in security reconnaissance workflows and automated pipelines. Katana supports both standard HTTP crawling and headless browser... -
6
The Falcon Web Framework
The no-nonsense REST API and microservices framework
Falcon is a minimalist WSGI library for building speedy web APIs and app backends. We like to think of Falcon as the Dieter Rams of web frameworks. When it comes to building HTTP APIs, other frameworks weigh you down with tons of dependencies and unnecessary abstractions. Falcon cuts to the chase with a clean design that embraces HTTP and the REST architectural style. Highly optimized, extensible code base. Easy access to headers and bodies through request and response objects. DRY request... -
7
Lightning Web Components
A Blazing Fast, Enterprise-Grade Web Components Foundation
Lean, lightweight runtime optimized for performance, with minimal boilerplate code. Lightning Web Components uses standard HTML, modern JavaScript (ES6+), and the best of native Web Components. Leave abstractions behind and build custom elements with HTML, JavaScript, and CSS. Lightning web components are custom elements built using HTML and modern JavaScript. Ready to learn more? Check out the Developer Guide and write code in the Playground. You’ll be up and running in no time. Create... -
8
Python Code Tutorials
The Python Code Tutorials
Python Code Tutorials is a large educational repository that aggregates programming tutorials from the “The Python Code” website into a structured collection of Python projects and learning materials. The repository covers a wide range of programming topics including cybersecurity, networking, web scraping, machine learning, GUI development, and automation scripts. Each tutorial typically includes complete Python code examples and explanations that demonstrate how to build real tools and... -
9
Crawl4AI
Open-source LLM Friendly Web Crawler & Scraper
Crawl4AI is a high-performance, AI‑ready web crawler tailored for LLM data ingestion and RAG pipelines. It supports adaptive crawling heuristics (stopping when enough info is gathered), structured markdown output, and high-speed parallel execution. Designed to operate at scale with optional Docker deployment and framework integrations. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
Proton Web Clients
Monorepo hosting the proton web clients
Proton Web Clients is a monorepo hosting the web applications for Proton’s suite of privacy-focused services, including the core Proton Mail webmail interface and related web apps like Proton Calendar, Proton Drive, Proton Account, Proton VPN, Proton Pass, and other connected tools. It consolidates all web client code, shared modules, dependencies, and development tooling into a single repository, enabling unified maintenance, consistency of design patterns, and efficient evolution of... -
11
web-mode.el
Web template editing mode for emacs
web-mode.el is an autonomous emacs major mode for editing web templates. HTML documents can embed parts (CSS / JavaScript) and blocks (client / server side). web-mode.el is compatible with many template engines: PHP, JSP, ASP, Django, Twig, Jinja, Mustache, ERB, FreeMarker, Velocity, Cheetah, Smarty, CTemplate, Mustache, Blade, ErlyDTL, Go Template, Dust.js, Google Closure (soy), React/JSX, Angularjs, ejs, Nunjucks, etc. Compatibility with many template engines : php, jsp, gsp (grails), asp... -
12
fess
Open source enterprise search server for websites, files, and data
...Fess includes a built-in crawler that can collect content from sources such as databases, CSV files, and shared storage, making it suitable for centralized knowledge discovery. It supports indexing and searching across many document formats including office documents, PDFs, and compressed archives. It also provides a web-based administrative interface that allows administrators to configure crawling targets, manage indexing tasks, and adjust search settings from a graphical dashboard. -
13
IPFS Web UI
A frontend for an IPFS node
A web interface to IPFS, shipped with Kubo, and ipfs-desktop. Check on your node stats, explore the IPLD powered merkle forest, see peers around the world and manage your files, without needing to touch the CLI. When working on the code, run an ipfs daemon, the local dev server, the unit tests, and the storybook component viewer and see the results of your changes as you save files. If you need to run IPFS in a Docker container, you can still have Web UI available by exposing both the... -
14
QR Code generator library
High-quality QR Code generator library in Java, TypeScript/JavaScript
This project aims to be the best, clearest library for generating QR Codes. My primary goals are flexible options and absolute correctness. The secondary goals are compact implementation size and good documentation comments. This work is an independent implementation based on reading the official ISO specification documents. I believe that my library has a more intuitive API and shorter code length than competing libraries out there. The library is designed first in Java and then ported to... -
15
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. -
16
The Apache Struts web framework
Mirror of Apache Struts
The Apache Struts web framework is a free open-source solution for creating Java web applications. Web applications differ from conventional websites in that web applications can create a dynamic response. Many websites deliver only static pages. A web application can interact with databases and business logic engines to customize a response. Web applications based on JavaServer Pages sometimes commingle database code, page design code, and control flow code. ... -
17
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
18
Playwright Skill for Claude Code
Claude Code Skill for browser automation with Playwright
...The system supports a wide range of use cases, including testing web applications, validating user interfaces, automating workflows, and extracting data from websites. One of its key advantages is its ability to generate custom Playwright code tailored to each request, allowing flexible and context-aware automation. -
19
web-extension-starter
Web Extension starter to build "Write Once Run on Any Browser"
Web Extension starter to build "Write Once Run on Any Browser" extension. Cross Browser Support (Web-Extensions API). Browser Tailored Manifest generation. Automatic build on code changes. Auto packs browser-specific build files. SASS styling. TypeScript by default. ES6 modules support. React UI Library by default. Smart reload. Used by extensions in production that has over 100,000+ users. -
20
Jupyter Notebook
Jupyter Interactive Notebook
The notebook extends the console-based approach to interactive computing in a qualitatively new direction, providing a web-based application suitable for capturing the whole computation process: developing, documenting, and executing code, as well as communicating the results. The Jupyter notebook combines two components. A web application, which is a browser-based tool for interactive authoring of documents which combine explanatory text, mathematics, computations and their rich media... -
21
Python-Spider
Python3 web crawler practice
Python-Spider is a repository intended to teach or provide examples for writing web spiders / crawlers in Python — part of a broader learning and resource collection by its author. The code and documentation are oriented toward beginners or intermediate learners who want to learn how to fetch, parse, and extract data from websites programmatically. As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe... -
22
Web Dev for Beginners
About 24 Lessons, 12 Weeks, Get Started as a Web Developer
Web-Dev-For-Beginners is Microsoft’s open source, project-based curriculum for learning web development from scratch. Designed as a 12-week, 24-lesson course, it covers HTML, CSS, and JavaScript fundamentals through hands-on projects like terrariums, browser extensions, and space games. Each lesson includes a mix of pre-lecture quizzes, written content, assignments, challenges, and post-lecture quizzes to reinforce learning. -
23
spider_collection
Collection of Python web scraping scripts for data extraction tasks
spider_collection is a collection of Python web crawler scripts created primarily for experimentation, learning, and practical scraping tasks. spider_collection gathers multiple independent spiders designed to collect data from different platforms and services, demonstrating a variety of scraping techniques and workflows. These crawlers make use of common Python scraping tools such as requests, parsel, BeautifulSoup, and the Scrapy framework to extract structured information from web pages.... -
24
Tencent Cloud Code Analysis
Static code analysis
Tencent Cloud Code Analysis (TCA for short, used internally by the R&D code CodeDog ) is a cloud-native, distributed, high-performance comprehensive code analysis and tracking platform that integrates many analysis tools, including server, web and client The three components have integrated a number of self-developed tools, and also support the dynamic integration of analysis tools of various programming languages in the industry. Obtain the Tencent Cloud code analysis platform by... -
25
Bot Framework Web Chat
A highly-customizable web-based client for Azure Bot Services
This repository contains code for the Bot Framework Web Chat component. The Bot Framework Web Chat component is a highly-customizable web-based client for the Bot Framework V4 SDK. The Bot Framework SDK v4 enables developers to model conversation and build sophisticated bot applications. This repo is part of the Microsoft Bot Framework, a comprehensive framework for building enterprise-grade conversational AI experiences.
