Search Results for "websites" - Page 2
Sort By:
Showing 78 open source projects for "websites"
View related business solutions-
Forever Free Full-Stack Observability | Grafana CloudBuilt on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.
-
Gemini 3 and 200+ AI Models on One PlatformBuild generative AI apps with Vertex AI. Switch between models without switching platforms.
-
1
dirhunt
Web crawler that finds hidden web directories without brute force
Dirhunt is an open source security tool designed to discover web directories and analyze website structures without relying on brute-force techniques. Instead of sending large numbers of guess-based requests, it operates as a specialized crawler that intelligently explores websites to identify accessible or hidden directories. Dirhunt can detect directories that expose “Index Of” listings, which may reveal files and other resources that were not intended to be publicly visible. It can also identify situations where directories are intentionally hidden through empty index files or servers that return misleading responses such as fake 404 errors. ... -
2
RPA for Python
Python package for doing RPA
Python package for doing RPA. RPA for Python's simple and powerful API makes robotic process automation fun! You can use it to quickly automate away repetitive time-consuming tasks on websites, desktop applications, or the command line. See sample Python script, the RPA Challenge solution, and RedMart groceries example. To send a Telegram app notification, simply look up @rpapybot to allow receiving messages. To automate Chrome browser invisibly, use headless mode. To run 10X faster instead of normal human speed, use turbo mode (read the caveats!). ... -
3
DecryptLogin
Python library providing APIs for automated website login workflows
DecryptLogin is a Python library designed to simplify automated login processes for many popular websites by providing ready-to-use APIs that simulate authentication behavior. It focuses on implementing login mechanisms through HTTP requests, allowing developers to programmatically authenticate with supported services without manually replicating complex login flows. It includes modules that handle different authentication modes such as PC login, mobile login, and QR code login depending on what the target platform supports. ... -
4
grab-site
Web crawler for archiving and backing up sites into WARC archives
grab-site is an open source web crawling tool designed to archive and back up websites by recursively downloading their content. It works by taking a starting URL and systematically following links across the site, capturing pages and resources and saving them into WARC archive files for long-term preservation. Internally, the crawler uses a fork of the wpull engine to fetch and process web pages efficiently during large-scale crawls. grab-site includes a built-in dashboard that displays real-time crawl activity, including which URLs are currently being processed and how many remain in the queue. ... -
8 Monitoring Tools in One APM. Install in 5 Minutes.AppSignal works out of the box for Ruby, Elixir, Node.js, Python, and more. 30-day free trial, no credit card required.
-
5
lxspider
Educational Python web scraping case collection for many sites
lxSpider is a collection of web scraping examples designed primarily for learning and experimentation with data extraction techniques. It gathers numerous crawler implementations that demonstrate how to collect data from a wide range of websites and online services. It focuses heavily on practical cases that illustrate how different platforms handle requests, authentication parameters, and anti-scraping protections. lxSpider includes examples targeting areas such as e-commerce platforms, social media services, content sites, research databases, and information portals. ... -
6
googler
Google from the terminal
...However, it has grown into a very handy and flexible utility that delivers much more. For example, fetch any number of results or start anywhere, limit the search by any duration, define aliases to google search any number of websites, and switch domains easily, all of this in a very clean interface without ads or stray URLs. The shell completion scripts make sure you don't need to remember any options. -
7
ruia
Async Python framework for fast and flexible web scraping spiders
Ruia is an asynchronous web scraping micro-framework built for Python that focuses on simplicity, speed, and flexibility when creating web crawlers. Ruia is powered by Python’s asyncio library along with aiohttp, enabling developers to perform concurrent network requests efficiently and scrape data from websites with minimal overhead. Ruia follows a “write less, run faster” philosophy, emphasizing concise code and streamlined spider development. It provides a structured approach to building scraping projects through components such as data items, spiders, middleware, and plugins. Developers can define structured fields to extract information from HTML content and process responses asynchronously to improve crawling performance. ... -
8
googler
Google Search, Google Site Search, Google News from the terminal
...However, it has grown into a very handy and flexible utility that delivers much more. For example, fetch any number of results or start anywhere, limit search by any duration, define aliases to google search any number of websites, switch domains easily... all of this in a very clean interface without ads or stray URLs. The shell completion scripts make sure you don't need to remember any options. googler isn't affiliated to Google in any way. Demo: https://asciinema.org/a/85019 -
9
XMLTV (http://xmltv.org/) is for grabbing TV listings primarily from websites. It has a grabber for Danish Television that grabs from http://tv.tv2.dk, but here we maintain serveral others. You can find documentation on http://niels.dybdahl.dk/xmltvdk
-
Try Google Cloud Risk-Free With $300 in CreditUse your credit across every product. Compute, storage, AI, analytics. When it runs out, 20+ products stay free. You only pay when you choose to.
-
10
GoogleScraper
Python tool for scraping search engine results from many providers
...It provides two different scraping approaches: sending direct HTTP requests that simulate browser traffic or controlling real browsers through automation frameworks. By running automated queries and collecting results in bulk, the project can assist with tasks such as SEO research, trend discovery, or building datasets of websites related to specific keywords. GoogleScraper also includes capabilities for running multiple scraping tasks concurrently to improve performance and increase the amount of collected data. -
11
AET
Detects visual changes on websites and performs page health checks
AET is a system that detects visual changes on websites and performs basic page health checks (like w3c compliance, accessibility, HTTP status codes, JS Error checks and others). AET is designed as a flexible system that can be adapted and tailored to the regression requirements of a given project. The tool has been developed to aid front-end client-side layout regression testing of websites or portfolios, in essence assessing the impact or change of a website from one snapshot to the next. -
12
ECommerceCrawlers
Collection of Python ecommerce and website crawler examples projects
ECommerceCrawlers is a collection of practical Python web crawler projects designed to gather data from a variety of ecommerce platforms, websites, and online services. It aggregates many independent crawler examples created by contributors and organized into separate subprojects that target specific sites or data sources. These examples demonstrate how to build and operate web scrapers capable of collecting structured information such as product listings, news content, job postings, social media data, and other publicly available web data. ... -
13
Toapi
Convert websites into structured APIs automatically with Python tool
Toapi is a Python library designed to transform ordinary websites into usable API services. Instead of building a traditional web crawler that collects and stores data before exposing it through an API, Toapi simplifies the process by allowing developers to define data structures that automatically generate an API layer from existing web pages. It works by parsing HTML content from a source site and mapping selected elements into structured data that can be returned as JSON through API endpoints. ... -
14
gain
Asyncio-based Python framework for building fast web crawling spiders
...It is built on top of asynchronous technologies such as asyncio, aiohttp, and uvloop to support high-performance crawling with concurrent network requests. It provides a structured framework for creating spiders that can navigate websites, extract structured data, and process the collected results. Developers define crawlers using components such as spiders, parsers, and items, allowing them to organize crawling logic and data extraction rules clearly. Gain supports CSS selectors and XPath expressions for parsing page content and extracting specific elements. Gain also allows developers to configure headers, concurrency levels, and proxy settings to control how crawlers interact with target websites. ... -
15
Offline Websites
Website2Pdf application helps to get offline form of webpages.
Favorite webpages can be made available offline as pdf files. Enter your favorite website url, with just one click pdf files will be created without loss of any css, styling of html. All the web files will be retained. Please make sure to use help button before you convert webpages to offline files. -
16
sqliv
Massive SQL injection vulnerability scanner for automated web testing
...SQLiv also supports reverse domain scanning to locate other websites hosted on the same server, which can then be examined for similar vulnerabilities. -
17
huhamhire-hosts
hosts for Internet Freedom
...Users could use these tool to modify the hosts to visit specified websites or services blocked by ISP/government. Functions which help users to backup/restore hosts files are also provided. For more detailed information, please visit the website of this project. -
18
Web browser plug-in for hosting real 3D on the websites.
-
19
"the maker" is a Content Management System for websites. It is client side and interacts seamlessly with a remote FTP server. It features a powerful text editor, has a build in FTP client and a tool for basic image manipultaion.
-
20
awb combines simple but powerful AsciiDoc markup with templates, blog and image gallery generation, and sitemap.xml generation to allow you to easily maintain and update a website.
-
21
IP Proxy Scraper - Linux
Extracts multiple proxies from a list of websites
Lightweigh and easy to use tool to extract multiple proxies from a list of websites. IP Proxy Scraper is also available for windows, check it out here: https://sourceforge.net/projects/ipproxyscraper/ -
22
dynamide is a dynamic web application framework for handling the presentation and business layers in a traditional web app. See http://dynamide.com
-
23
webgeno is a light-weight Content Management System (CMS) for generating websites offline. It is primarily designed for personal websites where the user doesn't have access to the server except to upload files.
-
24
-
25
LinkChecker
check links in web documents or full websites
New Homepage: http://wummel.github.io/linkchecker/ Linkchecker features: - recursive and multithreaded checking and site crawling - output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats - HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support - restrict link checking with regular expression filters for URLs - proxy support -...
