Search Results for "collect"
Sort By:
Showing 131 open source projects for "collect"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
1
Maigret
Collect a dossier on a person by username from thousands of sites
Maigret is an open-source OSINT tool designed to collect a dossier on a person by username from thousands of sites. It automates the process of checking for a user's presence across various platforms, aiding in digital investigations. -
2
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. DouyinCrawler supports both automated scraping and batch operations to process multiple targets efficiently. It also integrates with the Aria2 download utility to enable large-scale downloading of videos and images associated with collected content. ... -
3
Weibo Crawler
Python crawler for collecting and downloading Sina Weibo user data
...Collected data can be exported to structured formats such as CSV or JSON or stored in databases for further analysis and research. It supports incremental crawling so users can periodically collect only newly published posts, making it useful for ongoing monitoring or dataset updates. -
4
MediaCrawler
Self-media platform crawler
...It obtains required request signatures from the active browser context, avoiding complex JavaScript reverse engineering. Users can search by keyword, crawl specific posts, collect nested comments, and retrieve creator profiles. It also supports cached login states, proxy pools, and comment word-cloud generation. The repository is intended for learning and research, with clear restrictions against illegal, large-scale, or commercial scraping. -
Earn up to 16% annual interest with Nexo.Put idle assets to work with competitive interest rates, borrow without selling, and trade with precision. All in one platform. Geographic restrictions, eligibility, and terms apply.
-
5
X-osint
Open source OSINT tool for gathering data on emails, phones, and IPs
X-osint is an open source intelligence framework designed to collect and analyze publicly available information from multiple sources. It focuses on gathering useful and credible data about entities such as phone numbers, email addresses, and IP addresses using a range of automated OSINT techniques. It provides investigators and researchers with a centralized interface for running information-gathering tasks that would normally require multiple separate tools. -
6
jrnl
Collect your thoughts and notes without leaving the command line
Collect your thoughts and notes without leaving the command line. jrnl has a natural-language interface so you don't have to remember cryptic shortcuts when you're writing down your thoughts. Your journals are stored in plain-text files that will still be readable in 50 years when all your fancy iPad apps will have gone the way of the Dodo. -
7
Argilla
The open-source data curation platform for LLMs
...In real-world projects, data collection is a key activity of the iterative process of ML model development. Once a model goes into production, you want to monitor and analyze its predictions, and collect more data to improve your model over time. Argilla is designed to close this gap, enabling you to iterate as much as you need. -
8
Robin
AI-powered tool for dark web OSINT search and investigation
...It combines automated dark web search capabilities with large language models (LLMs) to analyze and summarize information discovered across hidden services and Tor-based search engines. The tool helps refine investigative queries, collect results from multiple dark web sources, and filter relevant intelligence using AI-driven processing. Robin also performs scraping of discovered pages through Tor sessions, allowing users to gather additional context from dark web sites while maintaining the required network routing. By integrating AI models, the platform can interpret results, highlight key information, and produce summaries that help analysts understand findings faster. ... -
9
Microsoft Azure CLI
Azure command-line interface
...Run the current command and show its output in a side-by-side editor. Show documentation on mouse hover. Display current subscription and defaults in the status bar. The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
DouK-Downloader
TikTok releases/likes/compilations/live streams/videos/atlases/music
DouK-Downloader is a fully open-source data acquisition and media downloading tool designed to extract, collect, and download content from TikTok and its Chinese counterpart Douyin at scale. Built using Python and modern asynchronous networking libraries such as HTTPX, it enables batch downloading of videos, images, live streams, and metadata from accounts, playlists, and individual links. The software goes beyond simple downloading by offering comprehensive data collection features, including comments, user statistics, and trending data such as hot boards and search results. ... -
11
Tandoor Recipes
Application for managing recipes, planning meals, building shopping
Application for managing recipes, planning meals, building shopping lists and much much more! This application is meant for people with a collection of recipes they want to share with family and friends or simply store them in a nicely organized way. A basic permission system exists but this application is not meant to be run as a public page. Tandoor is developed by volunteers in their free time just because its fun. That said earning some money with the project allows us to spend more time... -
12
watercrawl
AI-ready web crawler that extracts and structures website content
...It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website. WaterCrawl supports customizable extraction rules so users can focus only on relevant elements while ignoring unnecessary page components. WaterCrawl also offers real-time monitoring capabilities, allowing users to track crawling progress, performance metrics, and errors during large data collection jobs. ... -
13
BBOT
The recursive internet scanner for hackers
BBOT is an advanced open-source reconnaissance automation framework designed to streamline large-scale OSINT and attack surface discovery workflows. It operates as a modular and recursive scanning tool that can enumerate subdomains, perform port scans, gather metadata, and collect web intelligence through a unified command-line interface. The project emphasizes extensibility, allowing users to create or integrate custom modules that expand the scope of reconnaissance tasks without modifying the core engine. BBOT is particularly valuable for security researchers and red teamers who need to automate multi-stage discovery processes across complex infrastructures. ... -
14
autocrawler
Multiprocess Selenium crawler for downloading images by keywords
AutoCrawler is a Python-based image crawling tool designed to automatically download large numbers of images from search engines using automated browser interaction. It uses Selenium and a Chrome browser driver to navigate image search pages and collect image sources based on keywords provided by the user. AutoCrawler supports multiprocess and multithreaded downloading, which allows it to retrieve images faster by running several tasks simultaneously. Users provide search terms through a simple keyword file, and the crawler organizes downloaded images into directories for each keyword. ... -
15
Open Wearables
Self-hosted platform to unify wearable health data
Open Wearables is an open-source initiative that aims to provide a community-driven ecosystem for wearable device software and interoperability by connecting sensor data, activity tracking, and health insights across multiple platforms and devices. Instead of relying on closed vendor ecosystems, the project provides standardized data models and APIs that let developers and hobbyists collect, sync, and analyze biometric and environmental data from wearables, DIY sensors, and open hardware projects. This approach allows users to break free from manufacturer lock-in while enabling richer, customizable dashboards, real-time visualizations, and personalized health analytics that match real-world needs rather than a one-size-fits-all model. ... -
16
Argus
Python toolkit for OSINT and reconnaissance with 135+ modules
Argus is a Python-based open source toolkit designed to simplify information gathering and reconnaissance tasks in cybersecurity. It provides an integrated command-line environment that consolidates numerous reconnaissance utilities into a single framework. The tool enables users to collect data about networks, domains, web applications, and infrastructure in an organized and efficient manner. Argus includes a modular architecture with more than 130 modules that support activities such as DNS analysis, port scanning, web application inspection, and threat intelligence lookups. Its interactive CLI allows users to browse available modules, configure targets, run scans, and review results from within a unified interface. ... -
17
Roxy-WI
Web interface for managing Haproxy, Nginx, Apache and Keepalived
...Use Roxy-WI to build a high available cluster for a couple of clicks: install HAProxy, NGINX, Apache, Keepalived, and its exporters, and carry out the initial configuration for the services. Collect download statistics in one place. Choose one of the three available monitoring options or use them all. If there are any problems, you will be informed immediately. -
18
ArchiveBox
Open source self-hosted web archiving
ArchiveBox is a powerful, self-hosted internet archiving solution to collect, save, and view websites offline. Without active preservation effort, everything on the internet eventually disappears or degrades. Archive.org does a great job as a centralized service, but saved URLs have to be public, and they can't save every type of content. ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. ... -
19
OpenWPM
A web privacy measurement framework
OpenWPM is a web privacy measurement framework that makes it easy to collect data for privacy studies on a scale of thousands to millions of websites. OpenWPM is built on top of Firefox, with automation provided by Selenium. It includes several hooks for data collection. Check out the instrumentation section below for more details. OpenWPM is tested on Ubuntu 18.04 via TravisCI and is commonly used via the docker container that this repo builds, which is also based on Ubuntu. -
20
HelloGitHub
Share interesting, entry-level open source projects on GitHub
...The content includes interesting, entry-level open-source projects, open-source books, practical projects, enterprise-level projects, etc., so that you can feel the charm of open source in a short time and fall in love with open source! At first, I just wanted to collect interesting, high-quality, and easy-to-use projects that I found in the process of browsing GitHub, so that it would be easier to find and learn later. Later, I plan to share these interesting and valuable open source projects with you. I ended up writing this website for easy viewing and sharing. Open source projects in various languages, tools to make life better, books, study notes, tutorials, and more. ... -
21
lightning AI
The most intuitive, flexible, way for researchers to build models
...Download the code and type 'lightning run app'. Feel free to ssh into any machine and run from there as well. In research, we often have multiple separate scripts to train models, finetune them, collect results and more. -
22
FinalRecon
All-in-one Python web reconnaissance tool for fast target analysis
FinalRecon is an all-in-one web reconnaissance tool written in Python that helps security professionals gather information about a target website quickly and efficiently. It combines multiple reconnaissance techniques into a single command-line utility so users do not need to run several separate tools to collect similar data. FinalRecon focuses on providing a fast overview of a web target while maintaining accuracy in the collected results. It includes modules for gathering server information, analyzing SSL certificates, performing WHOIS lookups, and crawling website resources. FinalRecon can also enumerate DNS records, discover subdomains, search for directories and files, and scan common network ports. ... -
23
Shynet
Modern, privacy-friendly, and detailed web analytics
...They require handing all of your visitors' info to a third-party company They use cookies to track visitors across sessions, so you need to have those annoying cookie notices. They collect so much personal data that even the NSA is jealous. They are closed source and/or expensive, often with limited data portability. They are hard to use. Shynet has none of these caveats. You host it yourself, so the data is yours. It works without cookies, so you don't need any intrusive cookie notices. It collects just enough data to be useful, but not enough to be creepy. ... -
24
GRR
GRR Rapid Response, remote live forensics for incident response
GRR Rapid Response is an incident response framework focused on remote live forensics. It consists of a python client (agent) that is installed on target systems, and python server infrastructure that can manage and talk to clients. The goal of GRR is to support forensics and investigations in a fast, scalable manner to allow analysts to quickly triage attacks and perform analysis remotely. GRR client is deployed on systems that one might want to investigate. On every such system, once... -
25
Xianyu Intelligent Monitor Bot
AI tool for real-time monitoring and analysis of Goofish listings
ai-goofish-monitor is an open source automation tool designed to monitor listings on the Goofish second-hand marketplace and analyze them using artificial intelligence. It combines browser automation with AI-based analysis to automatically search, collect, and evaluate newly posted items that match a user’s purchase criteria. It uses Playwright to simulate real user interactions with the marketplace, allowing the system to retrieve product data and track updates in near real time. ai-goofish-monitor can run multiple monitoring tasks simultaneously, each configured with specific keywords, price ranges, and filtering conditions. ...
