Search Results for "python web crawler"
Sort By:
Showing 76 open source projects for "python web crawler"
View related business solutions-
Grafana: The open and composable observability platformGrafana is the open source analytics & monitoring solution for every database.
-
Keep company data safe with Chrome EnterpriseMake AI work your way with Chrome Enterprise. Block unapproved sites and set custom data controls that align with your company's policies.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
Best-of Web Development with Python
A ranked list of awesome python libraries for web development
...If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! A ranked list of awesome python libraries for web development. Updated weekly. -
3
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. -
4
Best-of Python
A ranked list of awesome Python open-source libraries
...If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! Ranked list of awesome python libraries for web development. Correctly generate plurals, ordinals, indefinite articles; convert numbers. Libraries for loading, collecting, and extracting data from a variety of data sources and formats. Libraries for data batch- and stream-processing, workflow automation, job scheduling, and other data pipeline tasks. -
Cloud-based help desk software with ServoDeskWhat if You Could Automate 90% of Your Repetitive Tasks in Under 30 Days? At ServoDesk, we help businesses like yours automate operations with AI, allowing you to cut service times in half and increase productivity by 25% - without hiring more staff.
-
5
Selenium-python Helium
Selenium-python but lighter: Helium is the best Python library
Under the hood, Helium forwards each call to Selenium. The difference is that Helium's API is much more high-level. In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by user-visible labels. As a result, Helium scripts are typically 30-50% shorter than similar Selenium scripts. What's more, they are easier to read and more stable with respect to changes in the underlying web page. Selenium-python is great for web automation. Helium makes it easier to use. ... -
6
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
7
redis-py
Redis Python client
redis-py is the official Python client for interacting with Redis, the in-memory data structure store. It supports all Redis commands and data types, making it easy to build caching, messaging, or real-time analytics features in Python applications. With both synchronous and asyncio support, redis-py is suited for modern Python projects and integrates smoothly into web frameworks, task queues, and backend services. -
8
notebooker
Productionise & schedule your Jupyter Notebooks
Productionise and schedule your Jupyter Notebooks, just as interactively as you wrote them. Notebooker is a webapp which can execute and parametrise Jupyter Notebooks as soon as they have been committed to git. The results are stored in MongoDB and searchable via the web interface, essentially turning your Jupyter Notebook into a production-style web-based report in a few clicks. -
9
PyMySQL
MySQL client library for Python
PyMySQL is a 100% Python implementation of the MySQL client protocol, allowing Python applications to connect to MySQL and MariaDB databases without requiring binary extensions. It supports standard DB‑API 2.0 features, such as cursors, transactions, and parameterized queries. PyMySQL is versatile for web applications, scripts, and tools, offering compatibility with ORMs like SQLAlchemy and frameworks like Django. -
Print management system for direct buyers, brokers, in-plants and printers.P3Software's affordable print management system, is ideally suited for corporate, non-profit and educational print buyers, print managers, in-plants and print manufacturers. Designed by print experts, this easy-to-use print procurement management system helps users manage the print sourcing and buying workflow, from initial job specification to project delivery. Core features include bid and buy or direct buy, customer proposal (estimate), customer direct ordering, enhanced CRM, powerful reporting, easy access to current and historical data, and outstanding training and support.
-
10
WeasyPrint
The awesome document factory
WeasyPrint is a smart solution helping people to create PDF documents. You can generate gorgeous statistical reports, invoices, tickets, and anything you want as long as you have some webdesign skills! Design your documents just as you design your websites! WeasyPrint follows the widely used HTML and CSS specifications from the W3C. You can use your usual web tools, languages and frameworks, but for print. Creating high-quality digital documents requires features that you love to use as... -
11
Helium
Lighter web automation with Python
Helium is a Python library built on top of Selenium to make browser automation more intuitive and human-friendly. It replaces verbose boilerplate code with natural language-like API calls such as click("Login") or write("hello", into="Name"). Helium manages browser setup, waits, and teardown, enabling quick development of scripts for testing, scraping, or task automation without requiring deep Selenium knowledge. -
12
Parsera
Lightweight library for scraping web-sites with LLMs
Scrape data from any website with only a link and column descriptions. Parsera is a tool designed to scrape web content, specifically handling poorly structured or messy websites. -
13
WTForms
A flexible forms validation and rendering library for Python
WTForms is a flexible forms validation and rendering library for Python web development. It can work with whatever web framework and template engine you choose. It supports data validation, CSRF protection, internationalization (I18N), and more. There are various community libraries that provide closer integration with popular frameworks. WTForms is designed to work with any web framework and template engine. -
14
Metaflow
A framework for real-life data science
Metaflow is a human-friendly Python library that helps scientists and engineers build and manage real-life data science projects. Metaflow was originally developed at Netflix to boost productivity of data scientists who work on a wide variety of projects from classical statistics to state-of-the-art deep learning. -
15
Tortoise ORM
Familiar asyncio ORM for python, built with relations in mind
Tortoise ORM is an easy-to-use asyncio ORM (Object Relational Mapper) for Python, inspired by Django's ORM. It is designed to work with asynchronous frameworks, providing a simple and familiar API for interacting with databases. Tortoise ORM supports various relational databases and is suitable for building high-performance web applications. -
16
Google CTF
Google CTF
Google CTF is the public repository that houses most of the challenges from Google’s Capture-the-Flag competitions since 2017 and the infrastructure used to run them. It’s a learning and practice archive: competitors and educators can replay tasks across categories like pwn, reversing, crypto, web, sandboxing, and forensics. The code and binaries intentionally contain vulnerabilities—by design—so users can explore exploit chains and patching in realistic settings. The repo also includes... -
17
django-viewflow
Reusable workflow library for Django
Viewflow is a lightweight reusable workflow library that helps to organize people collaboration business logic in Django applications. In conjunction with Django-material, they could be used as the framework to build ready-to-use business applications in minutes. Django web framework solves only technical problems related to the client-server interaction on top of the stateless HTTP protocol. Model-View-Template separation pattern helps to maintain simple CRUD-based logic. Viewflow is the... -
18
Awesome Free ChatGPT
List of free ChatGPT mirror sites, continuously updated
This is a curated directory of freely accessible ChatGPT-style services and mirror sites that offer AI chatbot interfaces without login or payment requirements. Resources often support multiple models like GPT-4, Claude, Gemini, and more. Data collected from multiple independent sites with descriptions and tags. Includes services with image upload and drawing capabilities. Aggregates free, no-login-required ChatGPT-like web services. Continually updated mirror list to maintain availability. -
19
Werkzeug
The comprehensive WSGI web application library
Werkzeug is a comprehensive WSGI web application library. It began as a simple collection of various utilities for WSGI applications and has become one of the most advanced WSGI utility libraries. Werkzeug doesn’t enforce any dependencies. It is up to the developer to choose a template engine, database adapter, and even how to handle requests. Includes an interactive debugger that allows inspecting stack traces and source code in the browser with an interactive interpreter for any frame in... -
20
Shumai
Fast Differentiable Tensor Library in JavaScript & TypeScript with Bun
...Built on Bun and Flashlight, with ArrayFire as its numerical backend, Shumai brings GPU-accelerated tensor operations, automatic differentiation, and scientific computing tools directly to JavaScript developers. It allows seamless integration of machine learning, deep learning, and custom differentiable programs into web-based or server-side environments without relying on Python frameworks. The library supports matrix operations, gradient computation, and tensor conversions with intuitive APIs and near-native speed, thanks to Bun’s low-overhead FFI bindings. It can automatically leverage GPU acceleration on Linux (via CUDA) and CPU computation on macOS. -
21
aws-devops-zero-to-hero
AWS zero to hero repo for devops engineers to learn AWS in 30 Days
aws-devops-zero-to-hero is a 30-day AWS learning roadmap aimed squarely at DevOps engineers who want both conceptual understanding and hands-on projects. The README is structured as a day-by-day syllabus, starting with “Day 1: Introduction to AWS” and moving through IAM, EC2, VPC networking, security, DNS (Route 53), storage (S3), and many other core services. Each day mixes explanation with at least one concrete project or lab, such as deploying applications on EC2, designing secure VPCs,... -
22
Graph Notebook
Library extending Jupyter notebooks to integrate with Apache TinkerPop
The graph notebook provides an easy way to interact with graph databases using Jupyter notebooks. Using this open-source Python package, you can connect to any graph database that supports the Apache TinkerPop, openCypher or the RDF SPARQL graph models. These databases could be running locally on your desktop or in the cloud. Graph databases can be used to explore a variety of use cases including knowledge graphs and identity graphs. This project includes many examples of Jupyter notebooks.... -
23
Basic Computer Games
An updated version of the classic "Basic Computer Games" book
Basic Computer Games is a modern revitalization of the classic “Basic Computer Games” book’s collection of games, ported and expanded into various modern, memory-safe and scripting languages. It includes illustrative code examples of many classic games (e.g. Blackjack, Bowling) in multiple languages, with the goal of making the historical games accessible and educational in safe modern environments. Definitely use the most recent versions and features of the target language, but also try to... -
24
Playwright for .NET
.NET version of the Playwright testing and automation library
Playwright for .NET is the official language port of Playwright, the library to automate Chromium, Firefox and WebKit with a single API. Playwright is built to enable cross-browser web automation that is ever-green, capable, reliable and fast. Cross-browser. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. Cross-platform. Test on Windows, Linux, and macOS, locally or on CI, headless or headed. Cross-language. Use the Playwright API in TypeScript, JavaScript, Python, .NET, Java. ... -
25
QR Code generator library
High-quality QR Code generator library in Java, TypeScript/JavaScript
This project aims to be the best, clearest library for generating QR Codes. My primary goals are flexible options and absolute correctness. The secondary goals are compact implementation size and good documentation comments. This work is an independent implementation based on reading the official ISO specification documents. I believe that my library has a more intuitive API and shorter code length than competing libraries out there. The library is designed first in Java and then ported to...
