Search Results for "python web crawler"
Sort By:
Showing 489 open source projects for "python web crawler"
View related business solutions-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
1
crawler
Collection of JS reverse engineering examples for web scraping study
crawler is a collection of web scraping and JavaScript reverse engineering examples designed for learning how modern websites protect their data and how those protections can be analyzed. It contains many case studies that demonstrate how to analyze and replicate request parameters, cookies, and encryption logic used by real websites. Each directory in the project focuses on a specific target service or scenario, showing how browser network requests and JavaScript code can be studied to reproduce API calls programmatically. ... -
2
Node Crawler
Web Crawler/Spider for NodeJS + server-side jQuery
Most powerful, popular and production crawling/scraping package for Node, happy hacking. -
3
EasySpider
A visual no-code/code-free web crawler/spider
A visual code-free/no-code web crawler/spider, supporting both Chinese and English. -
4
AUTOMATIC1111 Stable Diffusion web UI
Stable Diffusion web UI
AUTOMATIC1111's stable-diffusion-webui is a powerful, user-friendly web interface built on the Gradio library that allows users to easily interact with Stable Diffusion models for AI-powered image generation. Supporting both text-to-image (txt2img) and image-to-image (img2img) generation, this open-source UI offers a rich feature set including inpainting, outpainting, attention control, and multiple advanced upscaling options. With a flexible installation process across Windows, Linux, and... -
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
5
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. DouyinCrawler supports both automated scraping and batch operations to process multiple targets efficiently. It also integrates with the Aria2 download utility to enable large-scale downloading of videos and images associated with collected content. ... -
6
Web Dev for Beginners
About 24 Lessons, 12 Weeks, Get Started as a Web Developer
Web-Dev-For-Beginners is Microsoft’s open source, project-based curriculum for learning web development from scratch. Designed as a 12-week, 24-lesson course, it covers HTML, CSS, and JavaScript fundamentals through hands-on projects like terrariums, browser extensions, and space games. Each lesson includes a mix of pre-lecture quizzes, written content, assignments, challenges, and post-lecture quizzes to reinforce learning. The course also offers global accessibility with translations in... -
7
katana
Fast CLI web crawler for discovering endpoints in modern web apps
Katana is an open source command-line web crawling and spidering framework developed by ProjectDiscovery. It is designed to efficiently crawl websites and web applications in order to discover endpoints, resources, and other useful information that may not be easily visible through manual browsing. Katana focuses on speed and automation, making it suitable for use in security reconnaissance workflows and automated pipelines. Katana supports both standard HTTP crawling and headless browser... -
8
Stable Diffusion web UI for AMDGPUs
Stable Diffusion WebUI optimized for AMD GPUs with editing tools
Stable Diffusion WebUI AMDGPU is a browser-based interface for generating images using Stable Diffusion, built with Gradio and adapted for AMD graphics hardware. It provides both text-to-image and image-to-image workflows, allowing users to create, refine, and upscale visuals within a single interface. It includes tools such as inpainting and outpainting for editing specific areas of an image, along with features like prompt matrix generation and attention controls to fine-tune outputs.... -
9
fess
Open source enterprise search server for websites, files, and data
...Fess is built on top of OpenSearch and offers an integrated solution for crawling, indexing, and searching documents from websites, file systems, and various data stores. Fess includes a built-in crawler that can collect content from sources such as databases, CSV files, and shared storage, making it suitable for centralized knowledge discovery. It supports indexing and searching across many document formats including office documents, PDFs, and compressed archives. It also provides a web-based administrative interface that allows administrators to configure crawling targets, manage indexing tasks, and adjust search settings from a graphical dashboard. -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
10
diskover-community
Open source file indexing & storage analytics powered by Elasticsearch
Diskover Community Edition is an open source file system indexing and storage analytics platform designed to help organizations understand and manage large volumes of file data. It crawls file systems and indexes metadata using Elasticsearch, enabling fast search, analysis, and organization of files stored across different storage systems. It allows administrators and users to explore file structures, monitor storage usage, and gain insights into how data is distributed across... -
11
WanGP
AI video generator optimized for low VRAM and older GPUs use
Wan2GP is an open source AI video generation toolkit designed to make modern generative models accessible on consumer-grade hardware with limited GPU memory. It acts as a unified interface for running multiple video, image, and audio generation models, including Wan-based models as well as other systems like Hunyuan Video, Flux, and Qwen. A key focus of the project is reducing VRAM requirements, enabling some workflows to run on as little as 6 GB while still supporting older Nvidia and... -
12
Mercury Browser
Privacy-focused web browser fork of Firefox
Mercury Browser is an optimized, privacy-focused web browser that is a fork of Mozilla Firefox. It incorporates compiler optimizations such as AVX, AES, LTO, and PGO to enhance performance and security. With features derived from projects like LibreWolf, Waterfox, and Ghostery, Mercury disables telemetry and debugging elements by default, ensuring a more private browsing experience. It also includes usability patches that bring back features like the classic top bar and supports unsigned... -
13
videodl
Lightweight Python tool for downloading videos from many platforms
Videodl is a lightweight video downloader implemented entirely in Python that allows users to retrieve videos from a wide range of online media platforms. It focuses on providing a fast and simple way to parse video pages and download media files, often prioritizing high-definition versions without watermarks when available. It supports numerous video platforms across both Chinese and international streaming ecosystems, enabling users to fetch content from many popular services through a... -
14
UFONet
UFONet - Denial of Service Toolkit
UFONet is a powerful and controversial Python-based toolkit for testing and conducting Distributed Denial of Service (DDoS) attacks using unconventional methods, such as leveraging third-party web applications as attack vectors. It automates the discovery of vulnerable targets and enables attackers or researchers to launch large-scale amplification attacks without directly using botnets. -
15
Search with Lepton
Lightweight demo to build a conversational AI search engine quickly
...It includes both a backend service written in Python and a web interface that allows users to interact with the search engine in a conversational format. Developers can configure different search providers and language models through environment variables, making it flexible for experimentation and prototyping. -
16
Seeker
Accurately Locate Smartphones using Social Engineering
Seeker is an open source project that demonstrates how to obtain precise location information from devices using social engineering and web-based techniques. The tool sets up a phishing page that asks for location permissions, allowing GPS and other device data to be shared if the user consents. It can capture latitude, longitude, accuracy, altitude, direction, and even speed, with results displayed in a terminal. The project supports both manual deployment and tunneling services like Ngrok... -
17
JS Beautifier
Beautifier for javascript
...The beautifier can be added on your page as web library. JS Beautifier is hosted on two CDN services: cdnjs and rawgit. You can beautify javascript using JS Beautifier in your web browser, or on the command-line using node.js or python. -
18
Pydoll
Async Python library in automating Chromium browsers without WebDriver
Pydoll is a Python library designed for automating Chromium-based web browsers such as Chrome and Edge without relying on a traditional WebDriver layer. Instead of using external drivers, it connects directly to the Chrome DevTools Protocol through WebSocket, allowing scripts to control browser behavior more efficiently and with fewer compatibility issues. -
19
Odoo
Odoo. Open Source Apps To Grow Your Business
Odoo is a comprehensive suite of open source business applications designed to manage and streamline various organizational operations. It provides an integrated ecosystem of tools that cover core business functions such as CRM, accounting, eCommerce, inventory, HR, project management, and manufacturing. Each Odoo app can be deployed individually to meet specific business needs or combined to form a powerful all-in-one ERP system. The platform’s modular architecture allows businesses to... -
20
SwarmUI
Modular AI image and video generation web UI with extensible tools
SwarmUI is a modular web-based user interface designed for AI-driven image generation, with a strong focus on usability, performance, and extensibility. It serves as a unified environment for working with multiple AI models, including Stable Diffusion and newer image and video generation systems, allowing users to create and manage outputs through a browser interface. SwarmUI is built to accommodate both beginners and advanced users by offering a simple “Generate” interface alongside more... -
21
Scalene
High-performance CPU, GPU, and memory profiler for Python
Scalene is a high-performance CPU, GPU and memory profiler for Python that does a number of things that other Python profilers do not and cannot do. It runs orders of magnitude faster than other profilers while delivering far more detailed information. Once Scalene has profiled your program, it will launch a web browser with an interactive user interface (all processing is done locally). Hover over bars to see breakdowns of CPU and memory consumption, and click on underlined column headers to sort the columns. ... -
22
FastRTC
The python library for real-time communication
FastRTC is a Python library designed to simplify real-time communication (RTC), especially for audio and video streaming applications. It abstracts away much of the complexity that typically comes with implementing WebRTC by providing a simple interface — e.g. a Stream class — that can be mounted within a web backend (for example a FastAPI application). -
23
owllook
Vertical novel search engine with unified reading and tracking tools
Owllook is an open source vertical search engine designed for discovering and reading online novels from multiple sources. Instead of redirecting users to different sites, the system parses content from many novel platforms and presents it in a unified reading interface. It focuses on providing a simple and comfortable reading experience with features such as searching for books, following updates, bookmarking chapters, and maintaining a personal bookshelf. It aggregates results from... -
24
Grafana
Leading open-source visualization and observability platform
Grafana OSS is the leading open-source platform for visualization and observability. It enables teams to query, visualize, alert on, and explore telemetry data from multiple sources in a single interface. With support for 100+ data source plugins—including Prometheus, Loki, Elasticsearch, InfluxDB, SQL/NoSQL databases, and OpenTelemetry—Grafana helps teams correlate metrics, logs, and traces across applications and infrastructure. Users can build interactive dashboards with rich... -
25
reNgine
Automated framework for web application reconnaissance and scanning
reNgine is an automated reconnaissance framework designed to simplify and enhance the process of gathering information about web applications during security assessments. It provides a streamlined workflow for penetration testers, bug bounty hunters, and security teams who need to perform reconnaissance efficiently and at scale. The platform integrates multiple open-source reconnaissance tools into a unified environment with a configurable scanning engine and an intuitive web interface....
