Search Results for "python web crawler" - Page 5
Sort By:
Showing 3076 open source projects for "python web crawler"
View related business solutions-
Demo Series - Small Business Backup By VeeamWatch this on-demand demo series and learn how to protect your Microsoft 365 data with clear, simple, actionable steps that are easy to implement for businesses of all sizes.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Certbot
Get free HTTPS certificates forever from Let's Encrypt
Certbot is a fully-featured, easy-to-use, extensible client for the Let's Encrypt CA. It fetches a digital certificate from Let’s Encrypt, an open certificate authority launched by the EFF, Mozilla, and others. This certificate then lets browsers verify the identity of web servers and ensures secure communication over the Web. Obtaining and maintaining a certificate is usually such a hassle, but with Certbot and Let’s Encrypt it becomes automated and hassle-free. With just a few simple... -
2
Docker Compose
Define and run multi-container applications with Docker
Docker Compose is an open source tool for defining and running multi-container applications with Docker. Compose lets you use a YAML file to configure your application’s services, and then create and start all the services from your configuration with just a single command. Compose works great in all environments: production, staging, testing, development, and on CI workflows. Compose has commands for every stage of your application lifecycle, from starting, stopping and rebuilding services,... -
3
Bottle
bottle.py is a fast and simple micro-framework for python applications
Bottle is a minimalist web framework for building small web applications and APIs in Python. It is distributed as a single file with no external dependencies, making it perfect for rapid development, prototyping, or embedded use. Despite its small size, Bottle supports routing, templates, request handling, and plugin support, offering a full-featured toolkit in an extremely compact package. -
4
Substra
Low-level Python library used to interact with a Substra network
An open-source framework supporting privacy-preserving, traceable federated learning and machine learning orchestration. Offers a Python SDK, high-level FL library (SubstraFL), and web UI to define datasets, models, tasks, and orchestrate secure, auditable collaborations. -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
redis-py
Redis Python client
redis-py is the official Python client for interacting with Redis, the in-memory data structure store. It supports all Redis commands and data types, making it easy to build caching, messaging, or real-time analytics features in Python applications. With both synchronous and asyncio support, redis-py is suited for modern Python projects and integrates smoothly into web frameworks, task queues, and backend services. -
6
Parsera
Lightweight library for scraping web-sites with LLMs
Scrape data from any website with only a link and column descriptions. Parsera is a tool designed to scrape web content, specifically handling poorly structured or messy websites. -
7
Uplink
A Declarative HTTP Client for Python
A Declarative HTTP Client for Python. Inspired by Retrofit. Uplink is in beta development. The public API is still evolving, but we expect most changes to be backward compatible at this point. Uplink turns your HTTP API into a Python class. Build an instance to interact with the web service. Then, executing an HTTP request is as simply as invoking a method. Use decorators and type hints to describe each HTTP request. -
8
Dash
Build beautiful web-based analytic apps, no JavaScript required
Dash is a Python framework for building beautiful analytical web applications without any JavaScript. Built on top of Plotly.js, React and Flask, Dash easily achieves what an entire team of designers and engineers normally would. It ties modern UI controls and displays such as dropdown menus, sliders and graphs directly to your analytical Python code, and creates exceptional, interactive analytics apps. -
9
PostHog
PostHog provides open-source web & product analytics
PostHog is an all‑in‑one open‑source platform for product and web analytics—offering event-based analytics, session recording, feature flagging, A/B testing, cohorts, and more—that you can self‑host, with full support for data privacy and enterprise compliance. Sync data from external tools like Stripe, Hubspot, your data warehouse, and more. Query it alongside your product data. Run custom filters and transformations on your incoming data. Send it to 25+ tools or any webhook in real time or... -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
10
Frappe
Low code web framework for real world applications
Frappe is a full-stack, low-code web framework written in Python and JavaScript, used to build scalable and modular enterprise applications. It powers ERPNext and includes tools for REST APIs, user management, document modeling, workflows, and real-time updates. Frappe uses a "model-view-controller" approach with its own ORM and frontend system, enabling rapid development without sacrificing control or performance. -
11
gemini-web2api
Convert Google Gemini web into OpenAI-compatible API
gemini-web2api is a Python bridge that exposes Google Gemini web access through OpenAI-compatible API endpoints. It is designed to let OpenAI-style clients connect to Gemini-like models through routes such as chat completions, models, responses, and native Gemini-compatible endpoints. The project can run as a simple local server and uses a mostly single-file design with an optional dependency for streaming. -
12
Scweet
Scrape tweets, profiles, followers and following from Twitter/X
Scweet is a Python-based Twitter/X scraping library and CLI designed to collect tweets, profile timelines, followers, following lists, and user profile data without requiring the official Twitter/X API or a developer account. Instead of depending on deprecated unauthenticated scraping methods, it works by using X’s web GraphQL API together with authenticated browser cookies, which gives it a more current and practical approach for data extraction. -
13
Selectolax
Python binding to Modest and Lexbor engines
A fast HTML5 parser with CSS selectors using Modest and Lexbor engines. Selectolax supports two backends: Modest and Lexbor. By default, all examples use the Modest backend. Most of the features between backends are almost identical, but there are still some differences. Currently, the Lexbor backend is in beta and missing some of the features. To use lexbor, just import the parser and use it in the similar way to the HTMLParser. -
14
finvizfinance
Finviz analysis python library
finvizfinance is a package that collects financial information from FinViz website. Stock charts, fundamental & technical information, insider information and stock news. Forex charts and performance. Crypto charts and performance. Screener and Group provide data frames for comparing stocks according to different filters and trading signals. Getting information (fundament, description, outer rating, stock news, inside trader) of an individual stock. -
15
Pydoll
Async Python library in automating Chromium browsers without WebDriver
Pydoll is a Python library designed for automating Chromium-based web browsers such as Chrome and Edge without relying on a traditional WebDriver layer. Instead of using external drivers, it connects directly to the Chrome DevTools Protocol through WebSocket, allowing scripts to control browser behavior more efficiently and with fewer compatibility issues. -
16
ArchiveBox
Open source self-hosted web archiving
ArchiveBox is a powerful, self-hosted internet archiving solution to collect, save, and view websites offline. Without active preservation effort, everything on the internet eventually disappears or degrades. Archive.org does a great job as a centralized service, but saved URLs have to be public, and they can't save every type of content. ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data.... -
17
Astropy
Repository for the Astropy core package
The Astropy Project is a community effort to develop a common core package for Astronomy in Python and foster an ecosystem of interoperable astronomy packages. Astropy is a Python library for use in astronomy. Learn Astropy provides a portal to all of the Astropy educational material through a single dynamically searchable web page. It allows you to filter tutorials by keywords, search for filters, and make search queries in tutorials and documentation simultaneously. ... -
18
Sanic
Async Python 3.6+ web server/framework
Build fast, run fast with Sanic! Sanic is a Python 3.6+ web server and web framework designed to go fast. It provides a way to get a highly performant HTTP server up and running fast, while also making it easy to build, expand, and eventually scale. Sanic aspires to be as simple as possible while delivering the performance that you require. It allows the usage of the async/await syntax added in Python 3.5, so your code is guaranteed to be non-blocking and speedy. ... -
19
CadQuery
A python parametric CAD scripting framework based on OCCT
...Provide a non-proprietary, plain text model format that can be edited and executed with only a web browser. The scripts use a standard programming language, Python, and thus can benefit from the associated infrastructure. This includes many standard libraries and IDEs. CadQuery's CAD kernel Open CASCADE Technology (OCCT) is much more powerful than the CGAL used by OpenSCAD. Features supported natively by OCCT include NURBS, splines, surface sewing, STL repair, STEP import/export, and other complex operations. -
20
Mopidy
Mopidy is an extensible music server written in Python
Mopidy plays music from local disk, Spotify, SoundCloud, TuneIn, and more. You can edit the playlist from any phone, tablet, or computer using a variety of MPD and web clients. Vanilla Mopidy only plays music from files and radio streams. Through extensions, Mopidy can play music from cloud services like Spotify, SoundCloud, and TuneIn. With Mopidy's extension support, you can easily add backends for new music sources. Mopidy is a Python application that runs in a terminal or in the background on Linux computers or Macs that have network connectivity and audio output. ... -
21
Auto Bangumi
Automated Bangumi episode downloader and organizer with Web UI
Auto_Bangumi is a fully automated tool for downloading, organizing, and tracking anime (Bangumi) episodes using RSS feeds and download clients like qBittorrent. It offers a modern Web UI for managing subscriptions, custom filtering rules, automatic file renaming, and subtitle matching. Designed for anime fans, it streamlines the process of staying up-to-date with seasonal shows by integrating feed parsing, downloading, and library organization. -
22
furl
The easiest way to parse and modify URLs in Python
furl is a small Python library that makes it easier to parse, inspect, build, and modify URLs without relying on tedious low-level URL handling code. It wraps URL components into convenient objects, so developers can work directly with schemes, usernames, passwords, hosts, ports, paths, queries, and fragments. The library supports simple path editing, query argument changes, fragment manipulation, inline method chaining, and URL joining. It also handles encoding automatically, including... -
23
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. -
24
WAFW00F
WAFW00F allows one to identify and fingerprint Web App Firewall
The Web Application Firewall Fingerprinting Tool. Sends a normal HTTP request and analyses the response; this identifies a number of WAF solutions. If that is not successful, it sends a number of (potentially malicious) HTTP requests and uses simple logic to deduce which WAF it is. If that is also not successful, it analyses the responses previously returned and uses another simple algorithm to guess if a WAF or security solution is actively responding to our attacks. For further details,... -
25
Blender GIS
Blender addons to make the bridge between Blender and geographic data
Import in Blender most commons GIS data format, Shapefile vector, raster image, geotiff DEM, OpenStreetMap XML. There are a lot of possibilities to create a 3D terrain from geographic data with BlenderGIS, check the Flowchart to have an overview. Display dynamics web maps inside Blender 3d view, requests for OpenStreetMap data (buildings, roads, etc.), get true elevation data from the NASA SRTM mission. Manage georeferencing information of a scene, compute a terrain mesh by Delaunay...
