Search Results for "python web crawler" - Page 7
Sort By:
Showing 3076 open source projects for "python web crawler"
View related business solutions-
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Dulwich
Pure-Python Git implementation
Dulwich is a Python implementation of the Git file formats and protocols, which does not depend on Git itself. All functionality is available in pure Python. Optional C extensions can be built for improved performance. Dulwich takes its name from the area in London where the friendly Mr. and Mrs. Git once attended a cocktail party. Supported Python versions are Python 3.5 and later. Versions of Dulwich prior to 0.20 also supported Python 2.7. Supported platforms include Linux, Mac OS X and... -
2
Django MarkdownX
Comprehensive Markdown plugin built for Django
Django MarkdownX is a comprehensive Markdown plugin built for Django, the renowned high-level Python web framework, with flexibility, extensibility, and ease-of-use at its core. -
3
XY
Ultra-fast and customizable Python charts
...Interactive capabilities include panning, zooming, hovering, selections, crosshairs, callbacks, and linked charts. Visual styling can be controlled with Python, CSS, or Tailwind, and charts can appear in notebooks or web applications. Finished visualizations can also be exported as HTML, PNG, SVG, or PDF. -
4
Peewee-async
Asynchronous interface for peewee ORM powered by asyncio
peewee-async is an async extension for the Peewee ORM, enabling non-blocking database access in asyncio-powered Python applications. It allows you to use familiar Peewee models while benefiting from asynchronous I/O, making it ideal for web applications or services that require concurrency. -
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
5
MechanicalSoup
A Python library for automating interaction with websites
A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn't do JavaScript. MechanicalSoup was created by M Hickford, who was a fond user of the Mechanize library. Unfortunately, Mechanize was incompatible with Python 3 until 2019 and its development stalled for several years. MechanicalSoup provides a similar API, built on Python giants Requests (for HTTP sessions) and... -
6
granary
The social web translator
The social web translator. Fetches and converts data between social networks, HTML and JSON with microformats2, ActivityStreams/ActivityPub, Atom, JSON Feed, and more. Granary is a library and REST API that fetches and converts between a wide variety of social data sources and formats. Free yourself from silo API chaff and expose the sweet social data foodstuff inside in standard formats and protocols. -
7
Django OAuth Toolkit
OAuth2 goodies for the Djangonauts!
...OAuth is an open standard for access delegation, commonly used as a way for Internet users to grant websites or applications access to their information on other websites but without giving them the passwords. Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. Your Django app exposes a web API you want to protect with OAuth2 authentication. You need to implement an OAuth2 authorization server to provide tokens management for your infrastructure. -
8
pagodo
Automate Google Hacking Database scraping and searching
pagodo automates Google searching for potentially vulnerable web pages and applications on the Internet. It replaces manually performing Google dork searches with a web GUI browser. There are 2 parts. The first is ghdb_scraper.py that retrieves the latest Google dorks and the second portion is pagodo.py that leverages the information gathered by ghdb_scraper.py. This version of pagodo also supports native HTTP(S) and SOCKS5 application support, so no more wrapping it in a tool like... -
9
FlaskBB
A classic Forum Software in Python using Flask
...Our goal is to develop a forum system with modern technologies and sticking to already well-tested and approved layouts. FlaskBB is a simple and extensible forum software that should be seen as the foundation for building communities. It is written in Python using the web framework Flask. FlaskBB is being distributed under the BSD 3-Clause License. -
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
10
Reflex Dev
Web apps in pure Python
Reflex is a Python framework for building full-stack web apps entirely in Python—without writing JavaScript for the frontend. It provides fast live reloads, built-in state management, deployment tooling, and optional AI-powered scaffolding to accelerate development. -
11
Masonite
The Modern And Developer Centric Python Web Framework
Stop using old frameworks with just a few confusing features. Masonite is the developer-focused dev tool with all the features you need for the rapid development you deserve. Masonite is perfect for beginners getting their first web app deployed or advanced developers and businesses that need to reach for the full fleet of features available. Mail support for sending emails quickly. Queue support to speed your application up by sending jobs to run on a queue or asynchronously. Notifications... -
12
AGiXT
AGiXT is a dynamic AI Automation Platform
AGiXT is a dynamic Artificial Intelligence Automation Platform engineered to orchestrate efficient AI instruction management and task execution across a multitude of providers. Our solution infuses adaptive memory handling with a broad spectrum of commands to enhance AI's understanding and responsiveness, leading to improved task completion. The platform's smart features, like Smart Instruct and Smart Chat, seamlessly integrate web search, planning strategies, and conversation continuity,... -
13
PythonPark
Python open source project "The Road to Self-Study Programming"
PythonPark is a large, curated “learning playground” for Python — essentially a comprehensive self-study meta-repository aimed at helping learners progress in Python programming, data science, machine learning, web scraping, and software engineering practices. It aggregates tutorials, learning guides, project examples, and resources across topics: from Python basics and data structures to machine learning, web scraping, and even interview preparation and “programmer life” guidance. ... -
14
LLaMA Efficient Tuning
Easy-to-use LLM fine-tuning framework (LLaMA-2, BLOOM, Falcon
Easy-to-use LLM fine-tuning framework (LLaMA-2, BLOOM, Falcon, Baichuan, Qwen, ChatGLM2) -
15
videodl
Lightweight Python tool for downloading videos from many platforms
Videodl is a lightweight video downloader implemented entirely in Python that allows users to retrieve videos from a wide range of online media platforms. It focuses on providing a fast and simple way to parse video pages and download media files, often prioritizing high-definition versions without watermarks when available. It supports numerous video platforms across both Chinese and international streaming ecosystems, enabling users to fetch content from many popular services through a... -
16
Bili23 Downloader
Cross platform GUI tool for downloading videos from Bilibili sites
Bili23-Downloader is an open source desktop application designed for downloading video content from the Bilibili platform. It provides a graphical interface that allows users to download various types of media including user-uploaded videos, series episodes, movies, and other hosted content. It focuses on ease of use with a zero-configuration setup, making it accessible to both beginners and experienced users. It supports high performance downloads through multi-threading and includes resume... -
17
Mercury Browser
Privacy-focused web browser fork of Firefox
Mercury Browser is an optimized, privacy-focused web browser that is a fork of Mozilla Firefox. It incorporates compiler optimizations such as AVX, AES, LTO, and PGO to enhance performance and security. With features derived from projects like LibreWolf, Waterfox, and Ghostery, Mercury disables telemetry and debugging elements by default, ensuring a more private browsing experience. It also includes usability patches that bring back features like the classic top bar and supports unsigned... -
18
CommunityScrapers
This is a public repository containing scrapers
Stash Community Scrapers is a large open-source collection of metadata extraction tools designed to work with the Stash media management platform, enabling automated scraping of content information from various online sources. The repository contains hundreds of scraper definitions written primarily in YAML and Python, each tailored to extract structured metadata such as titles, performers, tags, and media details from specific websites. These scrapers integrate directly into Stash, allowing... -
19
Full Stack FastAPI and PostgreSQL
Full stack, modern web application generator
Generate a backend and frontend stack using Python, including interactive API documentation. Production ready Python web server using Uvicorn and Gunicorn. Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic). Great editor support. Completion everywhere. Less time debugging. Designed to be easy to use and learn. Less time reading docs. Minimize code duplication. -
20
Ungoogled Chromium
A lightweight approach to removing Google web service dependency
In descending order of significance (i.e. most important objective first), ungoogled-chromium is Google Chromium, sans dependency on Google web services, ungoogled-chromium retains the default Chromium experience as closely as possible. Unlike other Chromium forks that have their own visions of a web browser, ungoogled-chromium is essentially a drop-in replacement for Chromium. ungoogled-chromium features tweaks to enhance privacy, control, and transparency. However, almost all of these... -
21
AnySearch Skill
Unified real-time search engine skill for AI agents
AnySearch Skill is a real-time search engine skill for AI agents. It gives agents a structured way to search the web, run vertical searches, perform parallel batch searches, and extract full-page content. The project is packaged as a skill rather than a standalone search application, so it is meant to be installed into compatible AI-agent environments. It supports multiple domain-specific search categories, making it useful when general web search is too broad. The skill can also fetch page... -
22
Windrecorder
Windrecorder is a memory search app by records everything
Windrecorder is an open-source personal memory search engine that continuously records on-screen activity in a highly optimized and storage-efficient format. It captures screen content locally and builds a searchable database using OCR and image understanding, allowing users to rewind and rediscover anything they have previously seen. The system indexes only meaningful visual changes, extracting text, browser data, and contextual information to improve search accuracy and reduce storage... -
23
Starlette
The little ASGI framework that shines
Starlette is a lightweight ASGI framework/toolkit, which is ideal for building async web services in Python. It is production-ready and gives you a lightweight, low-complexity HTTP web framework. WebSocket support. In-process background tasks. Startup and shutdown events. Test client built on httpx. CORS, GZip, Static Files, streaming responses. Session and Cookie support. 100% test coverage. 100% type annotated codebase. -
24
Grab Framework Project
Web Scraping Framework
Grab is a python framework for building web scrapers. With Grab you can build web scrapers of various complexity, from simple 5-line scripts to complex asynchronous website crawlers processing millions of web pages. Grab provides an API for performing network requests and for handling the received content e.g. interacting with DOM tree of the HTML document. -
25
Payloads All The Things
A list of useful payloads and bypass for Web Application Security
A list of useful payloads and bypasses for Web Application Security. Feel free to improve with your payloads and techniques. The API key is a unique identifier that is used to authenticate requests associated with your project. Some developers might hardcode them or leave it on public shares.
