Search Results for "python web crawler"
Sort By:
Showing 96 open source projects for "python web crawler"
View related business solutions-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
Earn up to 16% annual interest with Nexo.Generate interest, access liquidity without selling, and execute trades seamlessly. All in one platform. Geographic restrictions, eligibility, and terms apply.
-
1
Spatie Crawler
An easy to use, powerful crawler implemented in PHP
Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites. -
2
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
3
Best-of Web Development with Python
A ranked list of awesome python libraries for web development
...If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! A ranked list of awesome python libraries for web development. Updated weekly. -
4
Best-of Python
A ranked list of awesome Python open-source libraries
...If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! Ranked list of awesome python libraries for web development. Correctly generate plurals, ordinals, indefinite articles; convert numbers. Libraries for loading, collecting, and extracting data from a variety of data sources and formats. Libraries for data batch- and stream-processing, workflow automation, job scheduling, and other data pipeline tasks. -
Fully Managed MySQL, PostgreSQL, and SQL ServerCloud SQL handles your database ops end to end, so you can focus on your app.
-
5
Python for OSINT
In this repository you will find sample code files for each day
Python for OSINT: 21-Day Course is a beginner-oriented training repository focused on automating repetitive open-source intelligence tasks with Python. It accompanies a free course PDF and provides sample code for lessons organized across three weeks. Topics progress from basic syntax and command-line tools to HTTP requests, APIs, JSON, CSV, databases, scraping, regular expressions, proxies, and file handling. -
6
Heritrix
Internet Archive's open-source, web-scale, web crawler project
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt. -
7
Selenium-python Helium
Selenium-python but lighter: Helium is the best Python library
Under the hood, Helium forwards each call to Selenium. The difference is that Helium's API is much more high-level. In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by user-visible labels. As a result, Helium scripts are typically 30-50% shorter than similar Selenium scripts. What's more, they are easier to read and more stable with respect to changes in the underlying web page. Selenium-python is great for web automation. Helium makes it easier to use. ... -
8
Comprehensive Python Cheatsheet
Comprehensive Python Cheatsheet
...It covers a broad range of topics including data structures, control flow, functions, object-oriented programming, standard library usage, and common patterns. The repository includes both web and printable versions, allowing users to access the material in multiple formats depending on their workflow. Because it is continuously maintained, the cheatsheet reflects modern Python usage and practical conventions. Overall, it serves as a fast lookup companion for everyday Python development. -
9
learn_python3_spider
Python crawler tutorial series, learning Python crawlers from 0 to 1
learn_python3_spider is a tutorial and example-code collection for learning web scraping with Python 3 from beginner to advanced topics. It introduces request capture, browser and mobile traffic inspection, and core scraping libraries such as Requests, BeautifulSoup, Selenium, Appium, and Scrapy. Lessons also cover JSON parsing, regular expressions, concurrency, proxy usage, login handling, and CAPTCHA-related techniques. -
Go from Code to Production URL in SecondsSkip the Kubernetes configs. Cloud Run handles HTTPS, scaling, and infrastructure automatically. Two million requests free per month.
-
10
My Python Eggs
Python Examples
My Python Eggs, commonly associated with the geekcomputers Python repository, is a large collection of practical Python scripts and small programs created primarily for experimentation, automation, and educational purposes. Rather than being a single cohesive application, it functions as a repository of utilities that demonstrate how Python can be used to solve everyday problems and automate repetitive tasks. -
11
Python-Spider
Python3 web crawler practice
...As part of the author’s public learning-path repositories, python-spider likely includes examples of HTTP requests, HTML parsing, maybe concurrency or scheduling to crawl multiple pages, and techniques to handle common web-scraping issues. For people wanting to get hands-on with building scrapers, collecting data, or learning how to navigate web programming in Python, this repository acts as a didactic reference or starting point. -
12
X-Crawl
Flexible Node.js AI-assisted crawler library
A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction. -
13
PyMySQL
MySQL client library for Python
PyMySQL is a 100% Python implementation of the MySQL client protocol, allowing Python applications to connect to MySQL and MariaDB databases without requiring binary extensions. It supports standard DB‑API 2.0 features, such as cursors, transactions, and parameterized queries. PyMySQL is versatile for web applications, scripts, and tools, offering compatibility with ORMs like SQLAlchemy and frameworks like Django. -
14
WTForms
A flexible forms validation and rendering library for Python
WTForms is a flexible forms validation and rendering library for Python web development. It can work with whatever web framework and template engine you choose. It supports data validation, CSRF protection, internationalization (I18N), and more. There are various community libraries that provide closer integration with popular frameworks. WTForms is designed to work with any web framework and template engine. -
15
redis-py
Redis Python client
redis-py is the official Python client for interacting with Redis, the in-memory data structure store. It supports all Redis commands and data types, making it easy to build caching, messaging, or real-time analytics features in Python applications. With both synchronous and asyncio support, redis-py is suited for modern Python projects and integrates smoothly into web frameworks, task queues, and backend services. -
16
interview_python
Interview questions about Python
...Additional sections cover operating systems, networking, database transactions, and computer architecture. Readers can study HTTP protocols, TCP connections, SQL indexing, Redis, and common web security concepts. The repository includes code examples, conceptual comparisons, and references to external educational resources. It also discusses differences between Python 2 and Python 3, reflecting the historical development of the language. The collection is intended for interview preparation and technical knowledge review. -
17
MediaManager
A modern selfhosted media management system for your media library
...It is designed for ease of deployment with Docker, supports standardized metadata sources such as TMDB and TVDB, and integrates OAuth/OIDC for secure authentication. Users can browse, search, and manage their media with a responsive web frontend while developers benefit from a clean codebase that uses Python and modern web technologies. Its holistic approach toward acquisition, tracking, and library maintenance reduces duplication, improves media discovery workflows, and simplifies long-term management of large media collections. -
18
Tortoise ORM
Familiar asyncio ORM for python, built with relations in mind
Tortoise ORM is an easy-to-use asyncio ORM (Object Relational Mapper) for Python, inspired by Django's ORM. It is designed to work with asynchronous frameworks, providing a simple and familiar API for interacting with databases. Tortoise ORM supports various relational databases and is suitable for building high-performance web applications. -
19
WeasyPrint
The awesome document factory
WeasyPrint is a smart solution helping people to create PDF documents. You can generate gorgeous statistical reports, invoices, tickets, and anything you want as long as you have some webdesign skills! Design your documents just as you design your websites! WeasyPrint follows the widely used HTML and CSS specifications from the W3C. You can use your usual web tools, languages and frameworks, but for print. Creating high-quality digital documents requires features that you love to use as... -
20
Parsera
Lightweight library for scraping web-sites with LLMs
Scrape data from any website with only a link and column descriptions. Parsera is a tool designed to scrape web content, specifically handling poorly structured or messy websites. -
21
notebooker
Productionise & schedule your Jupyter Notebooks
Productionise and schedule your Jupyter Notebooks, just as interactively as you wrote them. Notebooker is a webapp which can execute and parametrise Jupyter Notebooks as soon as they have been committed to git. The results are stored in MongoDB and searchable via the web interface, essentially turning your Jupyter Notebook into a production-style web-based report in a few clicks. -
22
how-to-live-better
Online single-page reading version of the 528-item "High-Value Life"
How to Live Better is a self-contained web reader for the open-source High Cost-Performance Life Guide. It renders all 32 sections and 528 recommendations from the upstream project into a single searchable HTML page. Each recommendation is presented as a card with evidence grade, value tier, and cost labels. Users can search the full guide, filter for top-grade evidence, or show only simplified explanations. A synchronized table of contents, responsive mobile layout, dark mode, and print... -
23
Helium
Lighter web automation with Python
Helium is a Python library built on top of Selenium to make browser automation more intuitive and human-friendly. It replaces verbose boilerplate code with natural language-like API calls such as click("Login") or write("hello", into="Name"). Helium manages browser setup, waits, and teardown, enabling quick development of scripts for testing, scraping, or task automation without requiring deep Selenium knowledge. -
24
Claude-Red
claude-red is a curated library of offensive security skills
Claude Red is a curated collection of offensive-security skill files for Claude's skill system. Each skill provides structured domain guidance for a specific area of authorized security testing or research. The library covers web applications, identity, Active Directory, wireless systems, cloud environments, mobile platforms, embedded devices, reconnaissance, exploit research, and fuzzing. Skills load on demand from conversational triggers instead of occupying context continuously. Users can... -
25
Playwright for .NET
.NET version of the Playwright testing and automation library
Playwright for .NET is the official language port of Playwright, the library to automate Chromium, Firefox and WebKit with a single API. Playwright is built to enable cross-browser web automation that is ever-green, capable, reliable and fast. Cross-browser. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. Cross-platform. Test on Windows, Linux, and macOS, locally or on CI, headless or headed. Cross-language. Use the Playwright API in TypeScript, JavaScript, Python, .NET, Java. ...
