Search Results for "large files"

s3fs allows Linux, macOS, and FreeBSD to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, allowing use of other tools like AWS CLI. Large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes. Compatible with Amazon S3, and other S3-based object stores. Allows random writes and appends, large files via multi-part upload, and renames via server-side copy. User-specified regions, including Amazon GovCloud. ...

...It is developped in C and available under the BSD license. It splits a list of directories and file trees into a certain number of partitions, trying to produce partitions with the same size and number of files. It can also produce partitions with a given number of files or a limited size. Once generated, partitions are either printed as file lists to stdout (default) or to files. Those lists can then be used by third party programs. Fpart also includes a live mode, which allows it to crawl very large filesystems and produce partitions in live. ...

...It supports any POSIX filesystem and implements advanced features for Lustre filesystems (list/purge files per OST or pool, read MDT changelogs...)

cryptmount allows management and user-mode mounting of encrypted filing systems on GNU/Linux systems. Filesystems & crypto-swap can be on either raw block devices or ordinary files, with device-mapper (dm-crypt) and loopback devices setup automatically.

diskover is a file system crawler and disk space usage software that uses Elasticsearch to index your file metadata. diskover crawls and indexes your files on a local computer or remote storage server over network mounts. diskover helps manage your storage by identifying old and unused files and give better insights into data change "hotfiles", file duplication "dupes" and wasted space. It is designed to help deal with managing large amounts of data growth and provide detailed storage analytics. ...

Retools is a set of modifications to the commonly used open source utilities bzip2, gzip, rsync, and tar that automatically selects the Lustre stripe count for created and/or extracted files according to the sizes of the files involved. By striping large files over a higher number of physical disks and small files over a lower number, aggregate I/O bandwidth for large files is maximized and the impact to the file system due to small files is minimized.

...File mapping allows the process to use both random input and output (I/O) and sequential I/O. It also allows the process to work efficiently with a large data file, such as a database, without having to map the whole file into memory. Multiple processes can also use memory-mapped files to share data.

Folder bookmarks allows you to quickly open a file or folder you use on a regular basis. No need to navigate large folder directories, just double-click the bookmark to open it! Will run on any operating system with .NET Framework 4.0+ installed.

Filesystems can avoid fragmentation if they know how large files will be. This library provides a wrapper for the fallocate() syscall (like glibc-2.10+), but also knows the xfs/ocfs2 ioctl and provides fallbacks better than posix_fallocate().

virtual800 is a FUSE based filesystem. It provides a big filesystem with big files for testing of large file transfers.

PicoStorage, a lightweight "filesystem in a file" (Structure Storage) library, features efficient and compact storage of both tiny and large files and directories, compression, and transactions. It is highly scalable, simple and easy to use.

Latest iteration in the Bonnie family of benchmarks. Tests large file IO and creation/deletion of small files.

Jibte is an alternative system exploring program like Microsoft Windows Explorer. Using Jibte, the user will always see the full size of the directories she is exploring, allowing users to search for 'hidden' large files that they wish to remove.

The Abstract Large File (ALF) project is a portable library for writing files that can be larger than 2GB or contain holes on systems that don't natively support one or both properties.