Search Results for "python web crawler" - Page 7
Sort By:
Showing 2276 open source projects for "python web crawler"
View related business solutions-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
Crowdtesting That Delivers | TesteumTesteum connects your software, app, or website to a worldwide network of testers, delivering detailed feedback in under 48 hours. Ensure functionality and refine UX on real devices, all at a fraction of traditional costs. Trusted by startups and enterprises alike, our platform streamlines quality assurance with actionable insights. Click to perfect your product now.
-
1
Playwright for .NET
.NET version of the Playwright testing and automation library
..., JavaScript, Python, .NET, Java. Test Mobile Web. Native mobile emulation of Google Chrome for Android and Mobile Safari. The same rendering engine works on your Desktop and in the Cloud. Auto-wait. Playwright waits for elements to be actionable prior to performing actions. It also has a rich set of introspection events. The combination of the two eliminates the need for artificial timeouts - the primary cause of flaky tests. -
2
PyMySQL
MySQL client library for Python
PyMySQL is a 100% Python implementation of the MySQL client protocol, allowing Python applications to connect to MySQL and MariaDB databases without requiring binary extensions. It supports standard DB‑API 2.0 features, such as cursors, transactions, and parameterized queries. PyMySQL is versatile for web applications, scripts, and tools, offering compatibility with ORMs like SQLAlchemy and frameworks like Django. -
3
Requests for PHP
Requests for PHP is a humble HTTP request library
Requests is a HTTP library written in PHP, for human beings. It is roughly based on the API from the excellent Requests Python library. Requests is ISC Licensed (similar to the new BSD license) and has no dependencies, except for PHP 5.6+. Despite PHP’s use as a language for the web, its tools for sending HTTP requests are severely lacking. cURL has an interesting API, to say the least, and you can’t always rely on it being available. Sockets provide only low-level access and require you... -
4
Lexbor
Lexbor is development of an open source HTML Renderer library
Lexbor is the development of a web browser engine available as a software library; it ships with a free license and has no extra dependencies. For us, speed is an absolute must-have. In our development process, we focus on fastest parsing techniques for HTML, CSS, and fonts, fastest data processing methods, and fastest ways to serve content to end users. Whether you are building a backend that handles millions of HTML documents or a UI-heavy user app, your software’s response rate always... -
MongoDB Atlas | Run databases anywhereMongoDB Atlas allows you to build and run modern applications across 125+ cloud regions, spanning AWS, Azure, and Google Cloud. Its multi-cloud clusters enable seamless data distribution and automated failover between cloud providers, ensuring high availability and flexibility without added complexity.
-
5
Tock
Tock, the open source conversational AI toolkit
Complete and autonomous NLU solution leveraging opensource libs, such as OpenNLP, Stanford, Duckling and more. Web, mobile, social networks, smart speakers and more. Create your bot once, connect it progressively to multiple channels as you need them. Simple graphical interfaces to build stories and models, manage multilingual and multichannel bots, better understand users with analytics. Program complex stories using Kotlin, Python or Node.js provided components, or integrate with any language... -
6
Agent360
360 monitoring agent
360 Monitoring is a web service that monitors and displays statistics of your server performance. Agent360 is OS-agnostic software compatible with Python 3.7 and 3.8. It's been optimized to have low CPU consumption and comes with an extendable set of useful plugins. -
7
Textual
Textual is a TUI (Text User Interface) framework for Python
Textual is a Python framework for creating interactive applications that run in your terminal. Textual adds interactivity to Rich with a Python API inspired by modern web development. On modern terminal software (installed by default on most systems), Textual apps can use 16.7 million colors with mouse support and smooth flicker-free animation. A powerful layout engine and re-usable components makes it possible to build apps that rival the desktop and web experience. Textual runs on Linux... -
8
Mercury
Convert Python notebook to web app and share with non-technical users
Turn Python notebooks to web applications with open-source Mercury framework. Hide code and add interactive widgets. Non-technical users can tweak widgets and execute notebook with new parameters. The core of Mercury is Open Source under AGPLv3. We provide Mercury Pro with additional features, dedicated support and friendly commercial license. Mercury is a perfect tool to convert Python notebook to interactive web application and share with non-programmers. You define interactive widgets... -
9
Advanced Alchemy
A carefully crafted, thoroughly tested, optimized companion library
advanced-alchemy is an opinionated ORM toolkit built on SQLAlchemy and designed for integration with the Litestar web framework. It simplifies common ORM patterns such as CRUD, pagination, and async support while providing extensibility and best practices out of the box. It’s designed to accelerate backend development for modern Python web apps. -
No-Nonsense Code-to-Cloud Security for Devs | AikidoAikido provides a unified security platform for developers, combining 12 powerful scans like SAST, DAST, and CSPM. AI-driven AutoFix and AutoTriage streamline vulnerability management, while runtime protection blocks attacks.
-
10
finvizfinance
Finviz analysis python library
finvizfinance is a package that collects financial information from FinViz website. Stock charts, fundamental & technical information, insider information and stock news. Forex charts and performance. Crypto charts and performance. Screener and Group provide data frames for comparing stocks according to different filters and trading signals. Getting information (fundament, description, outer rating, stock news, inside trader) of an individual stock. -
11
redis-py
Redis Python client
redis-py is the official Python client for interacting with Redis, the in-memory data structure store. It supports all Redis commands and data types, making it easy to build caching, messaging, or real-time analytics features in Python applications. With both synchronous and asyncio support, redis-py is suited for modern Python projects and integrates smoothly into web frameworks, task queues, and backend services. -
12
DrissionPage
Python based web automation tool. Powerful and elegant
DrissionPage is a Python-based automation framework that blends the capabilities of Selenium for browser automation with Requests-HTML for fast, headless web data extraction. It enables seamless switching between browser-controlled and headless HTTP sessions within the same interface. Ideal for web scraping, testing, and automation, DrissionPage is lightweight and highly efficient, offering more flexibility than standard Selenium or Requests usage alone. -
13
ScrapeGraphAI
Python scraper based on AI
Extracting content from websites and local documents using LLM. ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). Just say which information you want to extract and the library will do it for you. -
14
Dulwich
Pure-Python Git implementation
Dulwich is a Python implementation of the Git file formats and protocols, which does not depend on Git itself. All functionality is available in pure Python. Optional C extensions can be built for improved performance. Dulwich takes its name from the area in London where the friendly Mr. and Mrs. Git once attended a cocktail party. Supported Python versions are Python 3.5 and later. Versions of Dulwich prior to 0.20 also supported Python 2.7. Supported platforms include Linux, Mac OS X... -
15
Notte
Opensource browser using agents
Notte is an open-source browser framework that enables the development and deployment of web-based AI agents. It introduces a perception layer that transforms web pages into structured, navigable maps described in natural language, allowing agents to interact with the internet more effectively. Notte is designed for building scalable and efficient browser-based AI applications. -
16
Browser Use MCP Server
Browse the web, directly from Cursor etc.
A browser automation server implementing the Model Context Protocol, designed to allow AI assistants to browse the web directly from applications like Cursor. It supports natural language commands for web navigation and interaction. -
17
socketify.py
Bringing Http/Https and WebSockets High Performance servers for PyPy3
Socketify.py is a reliable, high-performance Python web framework for building large-scale app backends and microservices. With no precedents websocket performance and a really fast HTTP server that can delivery encrypted TLS 1.3 quicker than most alternative servers can do even unencrypted, cleartext messaging. -
18
Trafilatura
Python & command-line tool to gather text on the Web
Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text-processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first... -
19
ScrapydWeb
Web app for Scrapyd cluster management
Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization. Make sure that Scrapyd has been installed and started on all of your hosts. Start ScrapydWeb via command scrapydweb. (a config file would be generated for customizing settings on the first startup.) Add your Scrapyd servers, both formats of string and tuple are supported, you can attach basic auth for accessing the Scrapyd server, as well as a string for grouping or labeling. You can select any number... -
20
img2dataset
Easily turn large sets of image urls to an image dataset
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine. Also supports saving captions for url+caption datasets. Opt-out directives: Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers. -
21
dude uncomplicated data extraction
dude uncomplicated data extraction: A simple framework
Dude is a very simple framework for writing web scrapers using Python decorators. The design, inspired by Flask, was to easily build a web scraper in just a few lines of code. Dude has an easy-to-learn syntax. Dude is currently in Pre-Alpha. Please expect breaking changes. You can run your scraper from terminal/shell/command-line by supplying URLs, the output filename of your choice and the paths to your python scripts to dude scrape command. -
22
text-dedup
All-in-one text de-duplication
text-dedup is a Python library that enables efficient deduplication of large text corpora by using MinHash and other probabilistic techniques to detect near-duplicate content. This is especially useful for NLP tasks where duplicated training data can skew model performance. text-dedup scales to billions of documents and offers tools for chunking, hashing, and comparing text efficiently with low memory usage. It supports Jaccard similarity thresholding, parallel execution, and flexible... -
23
Basketball Reference
NBA Stats API via Basketball Reference
Basketball Reference is a great site (especially for a basketball stats nut like me), and hopefully, they don't get too pissed off at me for creating this. I initially wrote this library as an exercise for creating my first PyPi package, hope you find it valuable! This library was created for another Python project where I was trying to estimate an NBA player's productivity. A lot of sports-related APIs are expensive - luckily, Basketball Reference provides a free service which can be scraped... -
24
Mara Pipelines
A lightweight opinionated ETL framework, halfway between plain scripts
This package contains a lightweight data transformation framework with a focus on transparency and complexity reduction. Data integration pipelines as code: pipelines, tasks and commands are created using declarative Python code. PostgreSQL as a data processing engine. Extensive web ui. The web browser as the main tool for inspecting, running and debugging pipelines. GNU make semantics. Nodes depend on the completion of upstream nodes. No data dependencies or data flows. No in-app data... -
25
Auto Bangumi
Automated Bangumi episode downloader and organizer with Web UI
Auto_Bangumi is a fully automated tool for downloading, organizing, and tracking anime (Bangumi) episodes using RSS feeds and download clients like qBittorrent. It offers a modern Web UI for managing subscriptions, custom filtering rules, automatic file renaming, and subtitle matching. Designed for anime fans, it streamlines the process of staying up-to-date with seasonal shows by integrating feed parsing, downloading, and library organization.
