Search Results for "duplicate files"

...Diskover also helps identify outdated or unused files, duplicate data, and inefficient storage usage that can waste resources or increase operational costs. A Python-based indexing engine performs the scanning and indexing tasks.

tumblr-crawler is an open source Python-based utility designed to download media content from Tumblr blogs. It provides a script that automatically retrieves photos and videos from specified Tumblr sites and saves them locally for offline access. Users can specify one or multiple blogs to crawl by editing a configuration file or by passing parameters through the command line. Once executed, the script fetches media from the Tumblr API and stores the downloaded files in folders named after...

...Additional mechanisms like duplicate page detection help avoid re-crawling identical content.

proxypool is an open source proxy aggregation tool that automatically collects proxy node information from publicly available sources on the internet. It crawls different sources such as Telegram channels, subscription links, and publicly accessible web pages to gather proxy configurations. After collecting these nodes, proxypool processes them by removing duplicates and verifying whether each node is functional. proxypool then provides a usable list of proxy nodes that have passed...

...It's one solution to centrally handling uploads from a number of users, and extensive logging is provided by a Perl+MySQL combination. The key feature of the software is that duplicate files should be detected, and incoming files have an MD5 checksum automatically generated. All upload details, including timestamp, uploaded-by, checksum, filename and location are all stored in a 2-table database. The software should not impact the FTP server software as it is run separately. Coming soon will be a complementary application that will run as a daemon and scan the logging database on a regular basis. ...