Search Results for "websites"
Sort By:
Showing 173 open source projects for "websites"
View related business solutions-
Try Google Cloud Risk-Free With $300 in CreditUse your credit across every product. Compute, storage, AI, analytics. When it runs out, 20+ products stay free. You only pay when you choose to.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
SiteDorks
Automate search engine dorking across hundreds of websites
SiteDorks is a command line tool designed to automate advanced search queries across multiple search engines and websites. It allows users to perform search engine “dork” queries against a large set of predefined domains, making it easier to discover publicly available information across different platforms. SiteDorks supports several major search engines including Google, Bing, Brave, Ecosia, DuckDuckGo, Yahoo, and Yandex. Instead of manually running the same query for many sites, SiteDorks generates and executes the queries automatically using lists of “dorkable” websites. ... -
2
Browser Use
Make websites accessible for AI agents
Browser-Use is a framework that makes websites accessible for AI agents, enabling automated interactions and data extraction from web pages. -
3
MechanicalSoup
A Python library for automating interaction with websites
A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn't do JavaScript. MechanicalSoup was created by M Hickford, who was a fond user of the Mechanize library. Unfortunately, Mechanize was incompatible with Python 3 until 2019 and its development stalled for several years. -
4
ScrapeGraphAI
Python scraper based on AI
Extracting content from websites and local documents using LLM. ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). Just say which information you want to extract and the library will do it for you. -
Full-stack observability with actually useful AI | Grafana CloudBuilt on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.
-
5
qBittorrent RuTracker plugin
qBittorrent search engine plugin for rutracker
...The plugin follows qBittorrent’s official search plugin architecture and is implemented as a Python script that communicates with the RuTracker website to retrieve and display search results. By embedding this functionality into the client, it streamlines the torrent discovery process and reduces the need for manual browsing across external websites. The plugin requires users to authenticate with RuTracker credentials, enabling access to content that may otherwise be restricted or hidden behind login requirements. It is designed to be simple to install and configure, typically involving placing the script in the appropriate directory and editing credentials. -
6
Maigret
Collect a dossier on a person by username from thousands of sites
Maigret is an open-source OSINT tool designed to collect a dossier on a person by username from thousands of sites. It automates the process of checking for a user's presence across various platforms, aiding in digital investigations. -
7
django CMS
Easy-to-use and developer-friendly enterprise CMS powered by Django
Create modern websites that content editors love. django CMS was originally conceived by web developers frustrated with the technical and security limitations of other systems. Its lightweight core makes it easy to integrate with other software and put to use immediately, while its ease of use makes it the go-to choice for content managers, content editors and website admins. -
8
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. -
9
Streamlink
Streamlink is a CLI utility which pipes video streams
Streamlink is a command-line utility that pipes video streams from various services into a video player, such as VLC. The main purpose of Streamlink is to avoid resource-heavy and unoptimized websites, while still allowing the user to enjoy various streamed content. There is also an API available for developers who want access to the stream data. Streamlink is built upon a plugin system that allows support for new services to be easily added. Most of the big streaming services are supported. Streamlink is made up of two parts, a cli and a library API. ... -
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
10
crawler
Collection of JS reverse engineering examples for web scraping study
crawler is a collection of web scraping and JavaScript reverse engineering examples designed for learning how modern websites protect their data and how those protections can be analyzed. It contains many case studies that demonstrate how to analyze and replicate request parameters, cookies, and encryption logic used by real websites. Each directory in the project focuses on a specific target service or scenario, showing how browser network requests and JavaScript code can be studied to reproduce API calls programmatically. ... -
11
Tookie-OSINT
Username OSINT tool for discovering accounts across many websites
Tookie-OSINT is an open source intelligence tool designed to help security researchers, ethical hackers, and investigators discover online accounts associated with a specific username. It automates the process of searching for usernames across multiple websites, making it easier to identify a person's presence on different platforms. By entering a target username, Tookie-OSINT scans a list of supported sites and checks whether the username exists on those platforms. This approach removes the need for manual checks and significantly speeds up OSINT investigations. It is similar in concept to tools such as Sherlock, focusing on identifying user profiles across social media and other online services. ... -
12
Index
The SOTA Open-Source Browser Agent
Index is an open-source browser automation agent designed to autonomously perform complex tasks across websites by transforming web interfaces into programmable APIs. The system enables developers to instruct an AI agent to interact with web pages using natural language rather than traditional automation scripts. Instead of writing detailed browser automation code, users can describe the desired task and allow the agent to interpret the page structure, interact with elements, and complete multi-step workflows automatically. ... -
13
gallery-dl
Command-line program to download image galleries and collections
gallery-dl is a powerful command-line tool designed to download image galleries and collections from a wide range of image hosting and media websites. Built with Python, it works across multiple operating systems including Windows, Linux, and macOS. The program allows users to download images, manga chapters, or media collections by simply providing a supported website URL. It includes extensive configuration options that allow users to control download behavior, file naming, and directory structure. gallery-dl also supports authentication methods such as usernames, cookies, and OAuth to access restricted or private content. ... -
14
JobFunnel
Scrape job websites into a single spreadsheet with no duplicates.
Scrape job websites into a single spreadsheet with no duplicates. Automated tool for scraping job postings into a .csv file. You can search for jobs with YAML configuration files or by passing command arguments. By performing regular scraping and reviewing, you can cut through the noise of even the busiest job markets. Run funnel with your settings YAML to populate your master CSV file with jobs from available providers. -
15
LaVague
Framework for building AI agents that automate complex web tasks
...It can use browser automation tools such as Selenium or Playwright to interact with websites programmatically. Developers can integrate various language models and configure the agent’s reasoning and execution behavior to suit different automation scenarios. -
16
Podcastfy.ai
Transforming Multimodal Content into Captivating Multilingual Audio
Podcastfy is an open-source Python package that transforms multi-modal content (text, images) into engaging, multi-lingual audio conversations using GenAI. Input content includes websites, PDFs, youtube videos as well as images. Unlike UI-based tools focused primarily on note-taking or research synthesis (e.g. NotebookLM), Podcastfy focuses on the programmatic and bespoke generation of engaging, conversational transcripts and audio from a multitude of multi-modal sources enabling customization and scale. -
17
ArchiveBox
Open source self-hosted web archiving
...It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more. ArchiveBox is an open-source, self-hosted web archiving tool for saving websites offline. It helps organizations and individuals preserve bookmarks, research papers, and social media content, among others. -
18
Coomer Downloader App
Coomer downloader
CoomerDL is an open-source command-line tool designed to download and archive content from websites supported by the Coomer platform, which aggregates publicly available content from various creators. The tool enables users to automate the retrieval of media such as images, videos, and metadata associated with creator profiles. It is built to handle large volumes of content efficiently, allowing users to download entire collections or selectively retrieve specific posts. -
19
news-please
Python tool for crawling and extracting structured data from news site
news-please is an open source news crawler and information extraction tool designed to collect and structure articles from online news websites. It provides an integrated pipeline that crawls news sites, retrieves article pages, and extracts structured information such as headlines, authors, publication dates, and article text. news-please can recursively follow internal links and read RSS feeds to gather both recent and archived articles from a news outlet when given only the root URL of a site. ... -
20
LinkChecker
Check links in web documents or full websites
LinkChecker is a free, GPL licensed website validator. LinkChecker checks links in web documents or full websites. It runs on Python 3 systems, requiring Python 3.8 or later. The version in the pip repository may be old, to find out how to get the latest code, plus platform-specific information and other advice see doc/install.txt in the source code archive. If you do not want to install any additional libraries/dependencies you can use the Docker image which is published on GitHub Packages. -
21
Blackbird
OSINT tool for finding accounts across 600+ sites by username or email
...The project focuses on helping investigators, researchers, and security professionals quickly discover where a specific identity appears on the internet. It performs reverse searches across more than 600 websites by leveraging data from the community-driven WhatsMyName project, which improves detection accuracy and reduces false positives. The tool operates primarily through a command line interface, allowing users to run automated searches and gather results from many platforms in a single process. Blackbird also includes an optional AI-powered profiling feature that analyzes discovered sites to generate behavioral and technical insights about a user’s online presence. ... -
22
holehe
Check if the mail is used on different sites
...The tool works by leveraging password-reset mechanisms and other public account-existence checks to infer whether an email is associated with accounts on major platforms. It supports more than a hundred websites and is commonly used during reconnaissance, digital investigations, and account exposure assessments. holehe is designed to operate quickly and quietly, emphasizing efficiency and minimal footprint during enumeration tasks. The project can be used both as a standalone command-line tool and as a library embedded into larger automation pipelines. ... -
23
INS
Inspiration database for Internet practitioners with no ads
INS is described as a kind of “inspiration database for internet workers” — a repository that collects and curates interesting websites, tools, links, or resources that might inspire developers, designers, or any knowledge workers. It aims to operate without ads, focusing purely on the content and resource quality, and leverages automation (e.g. GitHub Actions) to check link validity or site load speed, ensuring that listed resources remain accessible over time. -
24
newspaper4k
Python library for scraping and analyzing online news articles easily
Newspaper4k is a Python library designed for extracting, processing, and analyzing news articles from websites. It is a continuation and active fork of the original newspaper3k library, which had stopped receiving updates, with the goal of keeping the ecosystem maintained while adding improvements and bug fixes. It provides developers with tools to automatically download web pages, extract the main article content, and collect associated metadata such as titles, authors, images, and publication dates. ... -
25
User Scanner
Scan usernames and emails across many platforms from the CLI
user-scanner is a command-line OSINT tool designed to analyze the presence and availability of usernames and email addresses across many online platforms. It helps users quickly determine whether a specific username or email is already associated with accounts on social networks, developer platforms, creator communities, gaming services, and other sites. user-scanner is useful for security researchers, investigators, and analysts performing open source intelligence, as well as individuals or...
