#37 [Feature request] Support for more data formats

[Anonymous]

Originally created by: mr-september

Currently, AFAIK, roboquant only supports .csv local historical data.

However, this is not great in terms of storage efficiency, and there are many alternatives (e.g pickle, feather, parquet...) with various combinations of compressions (e.g. zip, gzip...).

Would it be possible to add native support for some of these other formats?

(Personally I use zip-compressed pickle as the best compromise between size and read/write times)

df.to_pickle('foo.pkl', compression='zip')

pd.read_pickle('foo.pkl', compression='zip')

[Anonymous]

Originally posted by: jbaron

Actually there is support for Avro files (with snappy compression) and that is what I personally use all the time. The AvroFeed keeps minimal data in memory, so it works fine on low end machines. Avro is row based, which is great for random access like in a Monte-Carlo back testing. So I would suggest to give it a try.

It is easy to transform other Feeds into AvroFeed.

val feed = CSVFeed("someDir")
AvroFeed.record(feed, "/tmp/alpaca.avro")

// And then later you can use
val feed = AvroFeed("/tmp/alpaca.avro")
roboquant.run(feed)

To illustrate, a bit more complex example I used to get data from Alpaca and store it locally:

```kotlin
val feed = AlpacaHistoricFeed()
val tf = Timeframe.past(100.days) - 15.minutes
tf.split(1.days).forEach {
feed.retrieveStockPriceBars("AAPL", "IBM", timeframe = it, barPeriod = BarPeriod.MINUTE)
// Sleep a little to not exceed API call limits
Thread.sleep(1000)
}

// store the result in an Avro feed for future usage: val feed = AvroFeed("/tmp/alpaca.avro")
AvroFeed.record(feed, "/tmp/alpaca.avro")

I did notice people using indeed also using the formats you mention, but in general random (time-based) access is not that great and even a bit slower (but only limited testing on my site).

But still, if it is used a lot, I would consider adding support. I noticed Parquet used in a few places and you can download complete historic set of prices. And I assume pickle support would make it easier for Python users to use roboquant.

[Anonymous]

Originally posted by: mr-september

Oh! Thanks, I did not know about Avro, just thought it was an arbitrary name given to the class/function of loading data locally. From your introduction and with further reading, I think it makes a lot of sense.

I will look into converting my data to Avro. I would consider this issue closed, but I'll leave that up to you.

[Anonymous]

Originally posted by: jbaron

I'll leave it open for now, since it does make sense in the future to look at other formats is popular. Will put it on release 2.0 and based on popular demand or PR see to add some other binary formats.