Re: [Cs-studio-users] Multiple ArchiveEngines

[zunbeltz <zun...@es...>]

Dear Martin,

Thanks for the information and avices.
Indeed my idea was to store the sort-time data in a table/DB that  I 
could just simple drop or remove.

Best regards,

Zunbeltz


az., 2014.eko iraren 17a 17:42(e)an, Konrad, Martin igorleak idatzi zuen:
> Hi Zunbeltz,
>> Do you take some care on not trying to write at the same time with
>> different engines? Or some ohter measure?
> You don't need to do that. The DBMS takes care of that. You simply write
> values with a channel name and a timestamp into the table. When you
> retrieve data you get everything that matches your request (e.g. a
> certain channel name and within a certain time span).
>
>>> In principle, the channel configuration contains a "retention"
>>> parameter. You could configure that to for example set the
>>> retention of some channels to "1 year", and then you can write a
>>> tool that deletes data for those channels that's older than one
>>> year. For now, nobody has written such a tool.
> Be aware thought, that deleting data usually does not free up disk space
> with most RDBMS. You also need to run VACUUM which can take a long time.
> That's why some labs use different RDBMS archivers, e.g.
>
> Archiver 1. Stores the full data (e.g. 10 Hz), keeps data for 1 week,
> uses fast SSDs
> Archiver 2. Reduces data to 1 sample per minute, keeps data forever,
> uses slow hard drives
>
> If you partition your database over time (1 partition per week) you can
> simply keep two tables (in other words: DROP one table every week -
> which is very fast). Partitioning can also speed up data retrieval a
> bit. See [1] for details.
>
>>> The SLAC Archive appliance has the idea of
>>> short-term/middle-term/long-term
> Since the (SLAC) Archiver Appliance is not using a RDBMS for data
> storage (only for meta data) it can handle this in a different way: It
> saves data to short-term storage (usually a RAM disk) first. After a
> certain time it moves the files to mid-term storage (usually local disk)
> and after an even longer time it moves the data to long-term storage
> (usually NFS). This ETL process does not reduce the amount of data _by
> default_ but I think you _can_ configure filters [2] that reduce the
> amount of data along with the ETL process.
>
> HTH,
>
> Martin
>
> [1] http://accelconf.web.cern.ch/AccelConf/pcapac2012/papers/wepd03.pdf
> [2] http://epicsarchiverap.sourceforge.net/userguide.html#post_processing
>


-- 
Dr. Zunbeltz Izaola
Accelerator Physics Group
ESS-Bilbao