Re: [Cs-studio-users] Multiple ArchiveEngines

[Konrad, M. <ko...@fr...>]

Hi Zunbeltz,
> Do you take some care on not trying to write at the same time with
> different engines? Or some ohter measure?
You don't need to do that. The DBMS takes care of that. You simply write
values with a channel name and a timestamp into the table. When you
retrieve data you get everything that matches your request (e.g. a
certain channel name and within a certain time span).

>> In principle, the channel configuration contains a "retention"
>> parameter. You could configure that to for example set the
>> retention of some channels to "1 year", and then you can write a
>> tool that deletes data for those channels that's older than one
>> year. For now, nobody has written such a tool.
Be aware thought, that deleting data usually does not free up disk space 
with most RDBMS. You also need to run VACUUM which can take a long time. 
That's why some labs use different RDBMS archivers, e.g.

Archiver 1. Stores the full data (e.g. 10 Hz), keeps data for 1 week, 
uses fast SSDs
Archiver 2. Reduces data to 1 sample per minute, keeps data forever, 
uses slow hard drives

If you partition your database over time (1 partition per week) you can 
simply keep two tables (in other words: DROP one table every week - 
which is very fast). Partitioning can also speed up data retrieval a 
bit. See [1] for details.

>> The SLAC Archive appliance has the idea of
>> short-term/middle-term/long-term
Since the (SLAC) Archiver Appliance is not using a RDBMS for data 
storage (only for meta data) it can handle this in a different way: It 
saves data to short-term storage (usually a RAM disk) first. After a 
certain time it moves the files to mid-term storage (usually local disk) 
and after an even longer time it moves the data to long-term storage 
(usually NFS). This ETL process does not reduce the amount of data _by 
default_ but I think you _can_ configure filters [2] that reduce the 
amount of data along with the ETL process.

HTH,

Martin

[1] http://accelconf.web.cern.ch/AccelConf/pcapac2012/papers/wepd03.pdf
[2] http://epicsarchiverap.sourceforge.net/userguide.html#post_processing

-- 
Martin Konrad
Control System Engineer
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. 517-908-7253
Email: ko...@fr...