Re: [Cs-studio-users] Multiple ArchiveEngines

[zunbeltz <zun...@es...>]

Hi,


az., 2014.eko iraren 17a 15:28(e)an, Kasemir, Kay igorleak idatzi zuen:
>
> That's handled by the relational database.
> A channel is only within one group, within one engine.
> So there aren't two engines writing to the same channel.
> All the rest, the transactional integrity, is handled by the RDB, which is why this archive setup is so dependable.

But as far as I understood, it is possible to configure the same channel 
in different ArchiveEngine instances, isn't it? Sure you can not put the 
same twice in the configuration file for an engine, but I don't see what 
forbid it Archive Engines are running in different machines.

>>> There is no short-term/middle-term/long-term, though. All go to the same RDB tables.
>>>
>>> In principle, the channel configuration contains a "retention" parameter.
>>> You could configure that to for example set the retention of some channels to "1 year", and then you can write a tool that deletes data for those channels that's older than one year.
>>> For now, nobody has written such a tool.
>> In our installation we have just 290 PVs (4 of them are 2500 arrays). Most of the PV are sotred on change (this means that ACCT noise data is stored at the moment). This amounts to ~7 GB of data per day. I can image that at SNS you have a much bigger data. What do you cope with that?
> We have close to 90000 PVs, generating only about 1200 values per second. It amounts to about 1.4 TB per year.
> Right now the machine is off, and we have about 350 values per second.
> See also attached stats from the web report, generated by the Archive Reports package you can get from https://ics-web.sns.ornl.gov/css/products.html .
>
Obviously we are doing something wrong here. I guess our IOC servers are 
updating values to much or we are storing to much "noise". Is not 
reasonable at a small ion source + lebt is generating twice as much data 
as whole SNS accelerator!


> The SLAC Archive appliance is very promising:
> People simple add PV names, and it figures out how to store them: Sample at what period, which engine to use from a list of available ones, storing the data first in a RAM disk, then on a local disk, then on some NFS-mounted drive for longer storage.
> It's very fast, because it's using its own file format, with its own code to read and write that, no RDB overhead.
> I worry a little because that's what the original Channel Archiver did. As the data grew, we had to add custom index files to speed up access to the growing number of files, then add tools to manage the data, and eventually it looked much easier to just use an RDB which already has all those features, well tested - but certainly slower.
> By now, I believe SLAC and a few other sites have been using the SLAC Archive appliance for a while, so it seems to work OK.
>
> Than

My idea of the short-term database, was something like this:
Store as much as possible data in case  something breaks. If something 
breaks you can do the post-morten analysis. If everything is fine delete 
the data daily (or every two days).

Regards,

Zunbeltz

> ks,
> Kay
>


-- 
Dr. Zunbeltz Izaola
Accelerator Physics Group
ESS-Bilbao