Search Results for "python web crawler"
Sort By:
1587 projects for "python web crawler" with 1 filter applied:
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Get the most trusted enterprise browserDefend against security incidents with Chrome Enterprise. Create customizable controls, manage extensions and set proactive alerts to keep your data and employees protected without slowing down productivity.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
FastAPI Python
FastAPI framework, high performance, easy to learn, fast to code
FastAPI framework, high performance, easy to learn, fast to code, ready for production. FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints. -
3
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
4
NiceGUI
Create web-based user interfaces with Python
NiceGUI is a Python-based UI framework that enables developers to create interactive web applications using only Python code. It abstracts away the complexities of HTML, CSS, and JavaScript, allowing for rapid development of web interfaces directly from Python scripts. NiceGUI is suitable for building dashboards, control panels, and other web-based tools, especially in contexts like robotics and data visualization. -
Simply solve complex auth. Easy for devs to set up. Easy for non-devs to use.Custom auth drains 25% of dev time and risks 62% more breaches, stalling enterprise deals. Frontegg platform delivers a simple login box, seamless authentication (SSO, MFA, passwordless), robust multi-tenancy, and a customizable Admin Portal. Integrate fast with the React SDK, meet compliance needs, and focus on innovation.
-
5
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. Heritrix is designed to respect the robots.txt exclusion directives... -
6
Frappe
Low code web framework for real world applications
Frappe is a full-stack, low-code web framework written in Python and JavaScript, used to build scalable and modular enterprise applications. It powers ERPNext and includes tools for REST APIs, user management, document modeling, workflows, and real-time updates. Frappe uses a "model-view-controller" approach with its own ORM and frontend system, enabling rapid development without sacrificing control or performance. -
7
Glances
An eye on your system
Glances is an open source, cross-platform monitoring tool that aims to provide a significant amount of monitoring information through a curses or Web-based interface. Depending on the size of the user interface, this information can then dynamically adapt. Glances can work in client/server mode, and is also capable of remote monitoring. All systems statistics can be exported to files or external time/value databases. Glances gets information from your system through various libraries... -
8
StabilityMatrix
Multi-Platform Package Manager for Stable Diffusion
StabilityMatrix is a multi-platform package manager and inference UI for Stable Diffusion / AI image generation tools. It lets users install, update, and manage different Stable Diffusion based Web UIs (like Automatic1111, ComfyUI, SD-Next etc.), manage plugins/extensions, models, provide inference UI (generate images) with project/workspace support, drag & drop gallery, shared model directories etc. It is designed to be portable, embedded dependencies, cross-platform, with features like syntax... -
9
PyMySQL
MySQL client library for Python
PyMySQL is a 100% Python implementation of the MySQL client protocol, allowing Python applications to connect to MySQL and MariaDB databases without requiring binary extensions. It supports standard DB‑API 2.0 features, such as cursors, transactions, and parameterized queries. PyMySQL is versatile for web applications, scripts, and tools, offering compatibility with ORMs like SQLAlchemy and frameworks like Django. -
Test your software product anywhere in the worldGlobal App Testing is a managed pool of freelancers used by Google, Meta, Microsoft, and other world-beating software companies.
-
10
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
11
ChatGLM-6B
ChatGLM-6B: An Open Bilingual Dialogue Language Model
ChatGLM-6B is an open bilingual (Chinese + English) conversational language model based on the GLM architecture, with approximately 6.2 billion parameters. The project provides inference code, demos (command line, web, API), quantization support for lower memory deployment, and tools for finetuning (e.g., via P-Tuning v2). It is optimized for dialogue and question answering with a balance between performance and deployability in consumer hardware settings. -
12
Starlette
The little ASGI framework that shines
Starlette is a lightweight ASGI framework/toolkit, which is ideal for building async web services in Python. It is production-ready and gives you a lightweight, low-complexity HTTP web framework. WebSocket support. In-process background tasks. Startup and shutdown events. Test client built on httpx. CORS, GZip, Static Files, streaming responses. Session and Cookie support. 100% test coverage. 100% type annotated codebase. Few hard dependencies. Compatible with asyncio and trio backends. -
13
Gunicorn
WSGI HTTP Server for UNIX, fast clients and sleepy applications
Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX. It's a pre-fork worker model. The Gunicorn server is broadly compatible with various web frameworks, simply implemented, light on server resources, and fairly speedy. You can run Gunicorn by using commands or integrate with popular frameworks like Django, Pyramid, or TurboGears. For deploying Gunicorn in production see Deploying Gunicorn. After installing Gunicorn you will have access to the command line script gunicorn. Gunicorn... -
14
HunyuanDiT
Diffusion Transformer with Fine-Grained Chinese Understanding
HunyuanDiT is a high-capability text-to-image diffusion transformer with bilingual (Chinese/English) understanding and multi-turn dialogue capability. It trains a diffusion model in latent space using a transformer backbone and integrates a Multimodal Large Language Model (MLLM) to refine captions and support conversational image generation. It supports adapters like ControlNet, IP-Adapter, LoRA, and can run under constrained VRAM via distillation versions. -
15
asciinema
Open source terminal session recorder
asciinema is a free and open source terminal session recorder. It lets you easily record and play back terminal sessions in the terminal or in a web browser. Forget old screen recording methods and resulting blurry videos. asciinema lets you record your terminal sessions the right way, which is right where you work, in the terminal. Recording is as easy as running one command, and since it’s purely text-based you can copy and paste any content you want, simply pause the recording! You can... -
16
Basic Computer Games
An updated version of the classic "Basic Computer Games" book
Basic Computer Games is a modern revitalization of the classic “Basic Computer Games” book’s collection of games, ported and expanded into various modern, memory-safe and scripting languages. It includes illustrative code examples of many classic games (e.g. Blackjack, Bowling) in multiple languages, with the goal of making the historical games accessible and educational in safe modern environments. -
17
Reflex Dev
Web apps in pure Python
Reflex is a Python framework for building full-stack web apps entirely in Python—without writing JavaScript for the frontend. It provides fast live reloads, built-in state management, deployment tooling, and optional AI-powered scaffolding to accelerate development. -
18
Trame
Weave various components and technologies into a Web App
Developed by Kitware, trame is a Python-based framework that allows developers to create web applications with desktop-like functionality. It enables the integration of various components and technologies, such as VTK and ParaView, into web applications written entirely in Python. With best-in-class platforms at its core, trame provides complete control of 3D visualizations and data processing. Developers benefit from a write-once environment from trame. trame is an open source project licensed... -
19
DrissionPage
Python based web automation tool. Powerful and elegant
DrissionPage is a Python-based automation framework that blends the capabilities of Selenium for browser automation with Requests-HTML for fast, headless web data extraction. It enables seamless switching between browser-controlled and headless HTTP sessions within the same interface. Ideal for web scraping, testing, and automation, DrissionPage is lightweight and highly efficient, offering more flexibility than standard Selenium or Requests usage alone. -
20
Odigos
Distributed tracing without code changes
Odigos supports any application written in Java, Python, .NET, Node.js and Go. Historically, compiled languages like Go have been difficult to instrument without code changes. Odigos solves this problem by uniquely leveraging eBPF. Odigos currently supports all the popular managed and open source destinations. By producing data in the OpenTelemetry format, Odigos can be used with any observability tool that supports OTLP. Odigos automatically scales OpenTelemetry collectors based... -
21
Rio
WebApps in pure Python. No JavaScript, HTML and CSS needed
Rio is a Python framework designed to build web applications without the need for HTML, CSS, or JavaScript. Inspired by frameworks like Flutter and React, Rio offers a declarative interface and reusable components, enabling developers to create dynamic web apps entirely in Python. It streamlines the development process by managing both frontend and backend seamlessly. -
22
PyScript is a framework that allows users to create rich Python applications in the browser using HTML's interface and the power of Pyodide, MicroPython and WASM, and modern web technologies. PyScript is a meta project that aims to combine multiple open technologies into a framework that allows users to create sophisticated browser applications with Python. It integrates seamlessly with the way the DOM works in the browser and allows users to add Python logic in a way that feels natural both...
-
23
ReactPy
It's React, but in Python
ReactPy provides a React-style component model for Python developers to build web frontends without JavaScript. Components and hooks create declarative, reactive UIs, working across frameworks like Flask, Django, Jupyter, and more. With WebAssembly support for performance, ReactPy aims to empower Python-native frontend development. -
24
Bottle
bottle.py is a fast and simple micro-framework for python applications
Bottle is a minimalist web framework for building small web applications and APIs in Python. It is distributed as a single file with no external dependencies, making it perfect for rapid development, prototyping, or embedded use. Despite its small size, Bottle supports routing, templates, request handling, and plugin support, offering a full-featured toolkit in an extremely compact package. -
25
redis-py
Redis Python client
redis-py is the official Python client for interacting with Redis, the in-memory data structure store. It supports all Redis commands and data types, making it easy to build caching, messaging, or real-time analytics features in Python applications. With both synchronous and asyncio support, redis-py is suited for modern Python projects and integrates smoothly into web frameworks, task queues, and backend services.
