Search Results for "memory read"

Easy everyday parallelism with a file tree abstraction. Read a directory structure as a Julia data structure, (lazy-)load the files, apply map and reduce operations on the data while not exceeding available memory if possible. Make up a file tree in memory, create some data to go with each file (in parallel), write the tree to disk (in parallel). FileTrees is a set of tools to lazy-load, process and save file trees.

Feather is a fast, interoperable binary data frame storage format designed for efficient data exchange between analysis tools. It provides columnar serialization for data frames, making read and write operations faster than many text-based formats. The project was built to make sharing data across languages such as Python and R easier. Feather is powered by the Apache Arrow columnar memory specification, which helps it represent numeric, string, categorical, date, timestamp, boolean, and binary data efficiently. It also supports null and missing values across column types. ...

xsv is a command line program for indexing, slicing, analyzing, splitting and joining CSV files. Commands should be simple, fast and composable. Simple tasks should be easy. Performance trade offs should be exposed in the CLI interface. Composition should not come at the expense of performance. Let's say you're playing with some of the data from the Data Science Toolkit, which contains several CSV files. Maybe you're interested in the population counts of each city in the world. So grab the...