Search Results for "python web crawler"
Sort By:
Showing 247 open source projects for "python web crawler"
View related business solutions-
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
Cut Data Warehouse Costs by 54%BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.
-
1
MDServer Web
Simple Linux Panel
MDServer-Web is an open-source, web-based control panel for managing web servers and hosting environments. It supports popular web servers like Nginx and Apache, along with databases such as MySQL and Redis. The panel provides a user-friendly interface to manage websites, databases, SSL certificates, and more, making server administration accessible even to those with limited technical knowledge. -
2
Zappa - Serverless Python
Serverless Python
Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as "serverless" web hosting for your Python apps. That means infinite scaling, zero downtime, zero maintenance - and at a fraction of the cost of your current deployments! With a traditional HTTP server, the server is online 24/7, processing requests one by one as they come in. ... -
3
Pholcus
Distributed high-concurrency crawler software written in pure golang
Pholcus is a high-concurrency crawler software written in pure Go language that supports distributed, only used for programming learning and research. It supports three operating modes of stand-alone, server and client, and has three operating interfaces, Web, GUI, and command line; simple and flexible rules, concurrent batch tasks, and rich output methods (mysql/mongodb/kafka/csv/excel, etc.); In addition, it also supports horizontal and vertical grabbing modes, and a series of advanced functions such as simulated login and task suspension and cancellation. ... -
4
BotCity Framework Core Python
BotCity Framework - Python
Recognize and interact with UI elements using state-of-art computer vision module. Operate any UI interface independent of the technology or platform (desktop, web, terminal). BotCity is a platform to develop, deploy, manage and maintain automation. Automation can be developed in Python or Java using open-source libraries that are market standard. Develop, deploy, manage and scale your Automation Ops using All in One platform that provides task queue, runtime environment management, reports, alerts, logs and much more. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
5
Pyodide
Pyodide is a Python distribution for the browser and Node.js
Pyodide brings the Python runtime to the browser by compiling Python and its scientific libraries to WebAssembly. It allows developers to run Python code directly in web browsers without a server, supporting packages like NumPy, Pandas, and Matplotlib. Pyodide opens up new possibilities for interactive data analysis, scientific computing, and educational tools in web environments, all while integrating seamlessly with JavaScript. -
6
Locust
Scalable open source load testing tool
Locust is an open source user load testing tool written in Python. The idea behind Locust is to swarm your web site or other systems with attacks from simulated users during a test, with each user behavior defined by you using Python code. This swarming process is then monitored from a web UI in real-time, and will help identify any bottlenecks in your code before real users can come in. As it is completely event-based, Locust can have thousands or even millions of simultaneous users distributed over multiple machines swarming your system. ... -
7
NetBox
The premiere source of truth powering network automation
...By combining the traditional disciplines of IP address management (IPAM) and datacenter infrastructure management (DCIM) with powerful APIs and extensions, NetBox provides the ideal "source of truth" to power network automation. Available as open source software under the Apache 2.0 license, NetBox is employed by thousands of organizations around the world. Netbox is written in Python and uses the Django web framework. It is a web-based application that can be used to manage IP addresses and the devices and cables connected to them, as well as providing a data center infrastructure management (DCIM) tool. It supports virtualization, inventory management, and cable management. It has a web-based user interface and RESTful API, to easily integrate with other tools and automate tasks. -
8
ArchiveBox
Open source self-hosted web archiving
ArchiveBox is a powerful, self-hosted internet archiving solution to collect, save, and view websites offline. Without active preservation effort, everything on the internet eventually disappears or degrades. Archive.org does a great job as a centralized service, but saved URLs have to be public, and they can't save every type of content. ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data.... -
9
pyLoad
The free and open-source Download Manager written in pure Python
pyLoad is a lightweight, open-source download manager written entirely in Python that automates downloading files from file hosting services, cloud drives, and other online sources. The software is designed to run efficiently on a wide range of systems, including desktops, servers, routers, and NAS devices, making it suitable for both personal and self-hosted environments. It uses a plugin-driven architecture that supports hundreds of hosters, link decrypters, and extensions that extend its capabilities. pyLoad includes a modern web-based interface that allows users to remotely manage downloads from a browser, enabling full control over queues, links, and download settings. ... -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
Shelfmark
Web interface for searching and downloading books and audiobooks
Shelfmark is a self-hosted web application that serves as a unified interface for searching, discovering, and downloading ebooks and audiobooks from multiple sources into a personal digital library. Formerly known as Calibre Web Automated Book Downloader, the platform aggregates content from web archives, torrents, Usenet, IRC, and metadata providers to create a centralized acquisition hub. It provides both direct search and metadata-driven discovery modes, enabling richer results and... -
11
AnySearch Skill
Unified real-time search engine skill for AI agents
AnySearch Skill is a real-time search engine skill for AI agents. It gives agents a structured way to search the web, run vertical searches, perform parallel batch searches, and extract full-page content. The project is packaged as a skill rather than a standalone search application, so it is meant to be installed into compatible AI-agent environments. It supports multiple domain-specific search categories, making it useful when general web search is too broad. The skill can also fetch page... -
12
Nautobot
Network Source of Truth & Network Automation Platform
Nautobot is an open-source network source of truth and automation platform designed to manage network infrastructure data effectively. Initially built as a fork of NetBox, Nautobot extends its capabilities by offering flexible data modeling, powerful REST and GraphQL APIs, and built-in automation tools. It enables network engineers and operators to store, query, and integrate network infrastructure data with external systems, making it a key component in modern network automation workflows.... -
13
Scalene
High-performance CPU, GPU, and memory profiler for Python
Scalene is a high-performance CPU, GPU and memory profiler for Python that does a number of things that other Python profilers do not and cannot do. It runs orders of magnitude faster than other profilers while delivering far more detailed information. Once Scalene has profiled your program, it will launch a web browser with an interactive user interface (all processing is done locally). Hover over bars to see breakdowns of CPU and memory consumption, and click on underlined column headers to sort the columns. ... -
14
Glances
An eye on your system
Glances is an open source, cross-platform monitoring tool that aims to provide a significant amount of monitoring information through a curses or Web-based interface. Depending on the size of the user interface, this information can then dynamically adapt. Glances can work in client/server mode, and is also capable of remote monitoring. All systems statistics can be exported to files or external time/value databases. Glances gets information from your system through various libraries,... -
15
Ulauncher
Feature rich application Launcher for Linux
Type in an application name without worrying about spelling. Ulauncher will figure out what you meant. It also remembers your previous choices and automatically selects the best option for you. Ulauncher provides 4 themes built in. But if you need something different you can always create a custom color theme. Improve your workflow with customizable shortcuts and extensions. Create a shortcut for web search or your scripts or install a 3rd party extension. -
16
Roxy-WI
Web interface for managing Haproxy, Nginx, Apache and Keepalived
For those who need a convenient interface for managing all services in one place. Roxy-WI was created for people who want to have a fault-tolerant infrastructure, but do not want to plunge deep into the details of setting up and creating a cluster based on HAProxy, NGINX, Apache, and Keepalived. Use Roxy-WI to build a high available cluster for a couple of clicks: install HAProxy, NGINX, Apache, Keepalived, and its exporters, and carry out the initial configuration for the services. Collect... -
17
Modoboa
Mail hosting made simple
Modoboa is a mail hosting and management platform including a modern and simplified Web User Interface. It provides useful components such as an administration panel and webmail. Modoboa integrates with well known software such as Postfix or Dovecot. A SQL database (MySQL, PostgreSQL or SQLite) is used as a central point of communication between all components. Modoboa is developed with modularity in mind, expanding it is really easy. -
18
Meli-Action
Download files from direct links, YouTube, Telegram, Google Play
Meli-Action is a GitHub Actions-based downloading and web archiving project designed for heavily restricted internet environments. It uses GitHub-hosted workflow runners to fetch content from direct links, YouTube, Telegram, Google Play, SoundCloud, and web pages. Downloaded files are committed into the user’s repository so they can be retrieved later through normal GitHub access. The project includes workflows for different content sources, plus a Python script that can render and save web pages as MHTML archives using a headless browser. ... -
19
Grafana
Leading open-source visualization and observability platform
Grafana OSS is the leading open-source platform for visualization and observability. It enables teams to query, visualize, alert on, and explore telemetry data from multiple sources in a single interface. With support for 100+ data source plugins—including Prometheus, Loki, Elasticsearch, InfluxDB, SQL/NoSQL databases, and OpenTelemetry—Grafana helps teams correlate metrics, logs, and traces across applications and infrastructure. Users can build interactive dashboards with rich... -
20
JupyterLab
JupyterLab computational environment
JupyterLab is the next-generation web-based user interface for Project Jupyter. Try it on Binder. JupyterLab follows the Jupyter Community Guides. JupyterLab enables you to work with documents and activities such as Jupyter notebooks, text editors, terminals, and custom components in a flexible, integrated, and extensible manner. You can arrange multiple documents and activities side by side in the work area using tabs and splitters. Documents and activities integrate with each other,... -
21
Flower
Real-time monitor and web admin for Celery distributed task queue
Flower is an open-source web application that provides real-time monitoring and administrative control for Celery distributed task queues. It exposes detailed visibility into worker status, task execution history, and queue metrics through an interactive web dashboard. Developers and operators can remotely manage workers by restarting instances, adjusting pool sizes, revoking tasks, or applying rate limits without direct server access. -
22
Wagtail
A Django content management system focused on flexibility & UX
Wagtail is a powerful, open source content management system that’s focused on flexibility and user experience. Built on Django, Wagtail offers precise control and flexibility for designers, developers and editors. Designed by developers for developers, Wagtail plays nicely with everything else in your tech stack so you can do more and focus on perfecting your site. Designers will find Wagtail’s simple templating system ideal for building beautiful websites just the way they want, without... -
23
CapRover
Scalable PaaS (automated Docker+nginx), aka Heroku on Steroids
CapRover is an extremely easy-to-use app/database deployment & web server manager for your NodeJS, Python, PHP, ASP.NET, Ruby, MySQL, MongoDB, Postgres, WordPress (and etc...) applications! It's blazingly fast and very robust as it uses Docker, Nginx, LetsEncrypt and NetData under the hood behind its simple-to-use interface. For a developer who does not like spending hours and days setting up a server, building tools, sending code to the server, building it, getting an SSL certificate, installing it, update nginx over and over again. ... -
24
Hyperledger Cello
Operating System for Enterprise Blockchain
Hyperledger Cello is a blockchain operation and provisioning system designed to automate the deployment, management, and scaling of Hyperledger Fabric networks. As part of the Hyperledger project under the Linux Foundation, Cello aims to offer Blockchain-as-a-Service (BaaS) by abstracting the complexity of infrastructure setup for consortiums and enterprises. It provides a dashboard, APIs, and orchestration tools to help users create, monitor, and manage blockchain nodes, ledgers, and... -
25
Wizarr
User invitation and management system for Jellyfin, Plex, Emby etc.
Wizarr is an open-source system focused on simplifying user invitation, onboarding, and management for personal media servers like Jellyfin, Plex, and Emby, and it aims to evolve into a more comprehensive server administration tool. Initially conceived to enable administrators to create unique invite links that automatically register new users on their media servers, Wizarr abstracts many of the manual account-creation tasks typical of media server setups. It features a web interface and...
