Search Results for "collect"
Sort By:
Showing 111 open source projects for "collect"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Veeam Data Platform v13.1Try Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
1
douyin
Open source Douyin crawler for collecting and downloading public data
DouyinCrawler is an open source data collection tool designed to gather publicly available information from the Douyin platform. It demonstrates how to build a Python-based web crawler combined with a graphical interface and command line functionality. It allows users to collect data from various types of Douyin content, including user profiles, videos, hashtags, and music pages. DouyinCrawler supports both automated scraping and batch operations to process multiple targets efficiently. It also integrates with the Aria2 download utility to enable large-scale downloading of videos and images associated with collected content. ... -
2
Sub-Store
Advanced Subscription Manager for QX, Loon, Surge, Stash, etc.
Advanced subscription manager for QX, Loon, Surge, Stash and ShadowRocket. Conversion among various formats. Subscription formatting. Collect multiple subscriptions in one URL. -
3
Weibo Crawler
Python crawler for collecting and downloading Sina Weibo user data
...Collected data can be exported to structured formats such as CSV or JSON or stored in databases for further analysis and research. It supports incremental crawling so users can periodically collect only newly published posts, making it useful for ongoing monitoring or dataset updates. -
4
Listen 1
One for all free music in china (chrome extension)
Listen 1 can search and play songs from Netease Cloud Music, Xiami, QQ Music, Kugou Music, Kuwo Music, Bilibili, Migu Music website, making your music library more comprehensive. It also supports the playlist function, you can easily play, collect and create your own playlist. Download the Chrome plugin version zip package and unzip it. Open the Chrome extension page and check "Developer Mode" in the upper right corner. Select "Load unpacked extension..." and select the folder you just unpacked. Download the Windows zip file and choose the 32-bit or 64-bit version according to the system. ... -
Build Securely on Azure with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
5
Floorp Browser
All of source code of version 10 or later of Floorp Browser
...In addition to the usual Firefox theme customization, Floorp allows you to switch between five different interface designs, including OS-specific themes. Floorp is based on Firefox ESR. Floorp will be updated every 4 weeks, with security updates provided before each Firefox release. We don't collect personal information from users. We don't track users. We don't sell user data. We have no affiliation with any advertising companies. Floorp's source code is entirely open, allowing anyone to view it and contribute to the project. Not only is the browser itself open source, but the build environment is as well. -
6
Adguard Browser Extension
AdGuard browser extension
...Unlike its standalone counterparts (AG for Windows, Mac), the browser extension is completely free and open source. You can learn more about the difference here. AdGuard does not collect any information about you, and does not participate in any acceptable ads program. The only source of income we have is selling premium versions of our software, and we intend to keep it that way. AdGuard for Windows is not just another ad blocker, it is a multipurpose tool that combines all necessary features for the best web experience. ... -
7
Translate Web Pages
Translate your page in real time using Google or Yandex
...To translate any website it is necessary to access and modify the text of the web pages. And the extension can only do that, with that permission. The pages are translated using the Google or Yandex translation engine (you choose). We do not collect any information. However, to translate, the contents of the web pages will be sent to Google or Yandex servers. You can also install via crx file, download the file using a download manager/or firefox. Activate developer mode and drag the file into the chromium extension manager. It doesn't work on Chrome/Edge. -
8
Jellyfin Android TV
Android TV Client for Jellyfin
...Jellyfin is the volunteer-built media solution that puts you in control of your media. Stream to any device from your own server, with no strings attached. Your media, your server, your way. Jellyfin enables you to collect, manage, and stream your media. Run the Jellyfin server on your system and gain access to the leading free-software entertainment system, bells and whistles included. -
9
Microsoft Azure CLI
Azure command-line interface
...Run the current command and show its output in a side-by-side editor. Show documentation on mouse hover. Display current subscription and defaults in the status bar. The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. -
Build Securely on AWS with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
10
watercrawl
AI-ready web crawler that extracts and structures website content
...It enables developers and researchers to crawl web pages, extract meaningful information, and convert it into formats that are easier to process and analyze. It provides a modern crawling system that can automatically navigate links, control crawl depth, and collect content from targeted sections of a website. WaterCrawl supports customizable extraction rules so users can focus only on relevant elements while ignoring unnecessary page components. WaterCrawl also offers real-time monitoring capabilities, allowing users to track crawling progress, performance metrics, and errors during large data collection jobs. ... -
11
fess
Open source enterprise search server for websites, files, and data
...Fess is built on top of OpenSearch and offers an integrated solution for crawling, indexing, and searching documents from websites, file systems, and various data stores. Fess includes a built-in crawler that can collect content from sources such as databases, CSV files, and shared storage, making it suitable for centralized knowledge discovery. It supports indexing and searching across many document formats including office documents, PDFs, and compressed archives. It also provides a web-based administrative interface that allows administrators to configure crawling targets, manage indexing tasks, and adjust search settings from a graphical dashboard. -
12
scrawler
Desktop tool for downloading media from many social platforms
SCrawler is a desktop application designed to download media content from a wide range of online platforms and social media services. It allows users to add profiles, channels, or posts and automatically collect images, videos, and other media associated with them. It provides tools for organizing downloaded content locally, including feeds, profile folders, and customizable file naming rules. SCrawler includes advanced configuration options that allow users to control download behavior, manage concurrent jobs, and customize output paths. ... -
13
autocrawler
Multiprocess Selenium crawler for downloading images by keywords
AutoCrawler is a Python-based image crawling tool designed to automatically download large numbers of images from search engines using automated browser interaction. It uses Selenium and a Chrome browser driver to navigate image search pages and collect image sources based on keywords provided by the user. AutoCrawler supports multiprocess and multithreaded downloading, which allows it to retrieve images faster by running several tasks simultaneously. Users provide search terms through a simple keyword file, and the crawler organizes downloaded images into directories for each keyword. ... -
14
Linkwarden
Self-hosted collaborative bookmark manager
Linkwarden is a self-hosted, open-source bookmark manager built to help individuals and teams collect, organize, and preserve important web content in a way that stays useful long after the original pages change or disappear. Instead of saving only a URL, it captures durable archived formats so your saved knowledge remains accessible even when link rot happens. The experience is designed to feel like a modern “read-it-later” tool, with a reader view that makes long articles easier to consume and built-in highlighting and annotations so you can keep context on why a link mattered. ... -
15
Telegraf
The plugin-driven server agent for collecting and reporting metrics
...Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics. Telegraf is plugin-driven and has the concept of 4 distinct plugin types. Input Plugins collect metrics from the system, services, or 3rd party APIs. Processor Plugins transform, decorate, and/or filter metrics. Aggregator Plugins create aggregate metrics (e.g. mean, min, max, quantiles, etc.). Output Plugins write metrics to various destinations. New plugins are designed to be easy to contribute, pull requests are welcomed and we work to incorporate as many pull requests as possible. ... -
16
GoToSocial
Fast, fun, small ActivityPub server
GoToSocial is an ActivityPub social network server, written in Golang. With GoToSocial, you can keep in touch with your friends, post, read, and share images and articles. All without being tracked or advertised to. It is already deployable and useable, and it federates cleanly with many other Fediverse servers (not yet all). However, many things are not yet implemented, and there are plenty of bugs! We left the alpha stage around September/October 2024, and we intend to exit beta sometime... -
17
NSFW Filter
Google Chrome extension that blocks NSFW images
A Google Chrome extension that blocks NSFW images from the web pages that you load using TensorFlow JS. NSFW Filter web extension blocks NSFW content using AI. NSFW Filter allows you to block inappropriate, Not-Safe-For-Work content, protecting you online. A browser extension that blocks NSFW images from the web pages that you load using TensorFlowJS. When a web page is loaded, all the images remain hidden until they are found to be NSFW or not. If they are found to be NSFW, they remain... -
18
AWS Amplify
A declarative JavaScript library for application development
The Amplify open-source client libraries provide use-case-centric, opinionated, declarative, and easy-to-use interfaces across different categories of cloud-powered operations enabling mobile and web developers to easily interact with their backends. These libraries are powered by the AWS cloud and offer a pluggable model which can be extended to use other providers. The libraries can be used with both new backends created using the Amplify CLI and existing backend resources. The Amplify... -
19
skycaiji
Open source web scraping system for automated data collection tasks
SkyCaiji is an open source web scraping and data collection system designed to gather information from websites through configurable extraction rules. It focuses on simplifying the process of building crawlers by allowing users to visually define scraping rules rather than writing complex code. It can collect structured or unstructured data from many types of webpages and automate the extraction process for large datasets. SkyCaiji is designed to run on a variety of hosting environments including local machines, shared hosting environments, and cloud servers. It integrates with content management systems so collected data can be published automatically without manual intervention. ... -
20
OpenWPM
A web privacy measurement framework
OpenWPM is a web privacy measurement framework that makes it easy to collect data for privacy studies on a scale of thousands to millions of websites. OpenWPM is built on top of Firefox, with automation provided by Selenium. It includes several hooks for data collection. Check out the instrumentation section below for more details. OpenWPM is tested on Ubuntu 18.04 via TravisCI and is commonly used via the docker container that this repo builds, which is also based on Ubuntu. -
21
Spider
High-performance Rust web crawler and scraper for large-scale data
...It focuses on speed, concurrency, and reliability by using asynchronous and multi-threaded processing to handle large volumes of web pages. It can rapidly crawl websites to collect links, retrieve page content, and extract structured information from HTML documents. Spider can operate concurrently across many pages, allowing it to gather large datasets in a short period of time. Spider also provides mechanisms for subscribing to crawl events so developers can process page data such as URLs, status codes, or HTML content as it is discovered. ... -
22
xhs-spider
Desktop tool for collecting and exporting Xiaohongshu post data
XHS-Spider is a desktop data collection tool designed to gather content and metadata from the Xiaohongshu platform. It provides a graphical interface that allows users to explore posts, collect information, and download media such as images and videos from individual notes or search results. It was developed primarily as a learning project to demonstrate approaches to building web crawlers and experimenting with technologies such as WebView2 and WPF UI. It supports multiple ways to locate content, including keyword searches, user searches, and parsing individual post links. ... -
23
Shynet
Modern, privacy-friendly, and detailed web analytics
...They require handing all of your visitors' info to a third-party company They use cookies to track visitors across sessions, so you need to have those annoying cookie notices. They collect so much personal data that even the NSA is jealous. They are closed source and/or expensive, often with limited data portability. They are hard to use. Shynet has none of these caveats. You host it yourself, so the data is yours. It works without cookies, so you don't need any intrusive cookie notices. It collects just enough data to be useful, but not enough to be creepy. ... -
24
SPX
A simple & straight-to-the-point PHP profiling extension
...Using a dedicated browser extension or command line launcher. Multi metrics capable: 22 are currently supported (various time & memory metrics, included files, objects in use, I/O...). Able to collect data without losing context. For example Xhprof (and potentially its forks) aggregates data per caller / callee pairs, which implies the loss of the full call stack and forbids timeline or Flamegraph based analysis. -
25
newspaper4k
Python library for scraping and analyzing online news articles easily
...It is a continuation and active fork of the original newspaper3k library, which had stopped receiving updates, with the goal of keeping the ecosystem maintained while adding improvements and bug fixes. It provides developers with tools to automatically download web pages, extract the main article content, and collect associated metadata such as titles, authors, images, and publication dates. Newspaper4k also includes natural language processing capabilities that can generate summaries and identify keywords from extracted article text. Newspaper4k supports both single-article extraction and full news site processing, allowing users to build sources representing entire publications and iterate through their articles. ...
