Menu
▾
▴
PS Re: Service design drafts for NSLS2.
|
From: White, G. <gr...@sl...> - 2011-11-04 11:39:02
|
PS: Forgot to add the other 2 tables that are in the LCLS schema that are not in yours:
symbols_upload - contains the original output from the MAD tape file. It describes
each accelerator device, or MAD "symbol". It is completely
denormalized (that is, each row has many blank columns because
the set of columns is the union of those needed for all devices).
initial_conditiions - points to a file while describes is the injection condition
for tracking. Ie, the twiss params going into the tracking (after
the space-charge dominated part).
Greg
On 4 Nov 2011, at 12:10, White, Greg wrote:
> Hi Guobao,
>
> I see, things are a lot clearer to me now. The first paragraph of the Online Calculator Service document
> got on the wrong track. But now I think I understand the architecture you have in mind.
>
> To confirm, very basically, the Twiss service is the interface that physics applications which want to
> use model data, get that model data from. For instance a bump calculator or orbit fitter would
> go to the Twiss service for Rmats. The Twiss service would get that data from IRMIS. The Twiss
> service does no tracking, it just gets data from the db and can do Rmat A to B calculations. Whereas
> the calculator service can track models, or accept externally tracked models, and it puts the
> results of that tracking into IRMIS.
>
>
> Ok, that sounds good. I can follow that architecture for SwissFEL. I'm going to do
> the Twiss Service first because, I imagine at SwissFel at least, we can iron out tracking
> issues by trial and error using a script system much more easily than one involving a server.
> Trial and error is my favourite pattern for development. Then, when it's looking right, we'll put it
> in a server and write a nice client for controlling it.
>
> Please find the ERD (Entity Relationship Diagram) for our model database enclosed. You'll see it's
> essentially the same, just bigger, may be because of some meeting we had at BNL?
>
> Your "element" table = LCLS "lcls_elements" table
> Your "model" table = LCLS "runs" table
> Your "twiss" table = LCLS "element_models" table
> Your "lattice" table = LCLS "modelled_devices" table
>
> We have a couple of other tables too:
> hardware_settings - It was intended to record the hardware setpoints
> that resulted in a given tracked model. But in fact we don't use it.
> xml_docs - Keeps track of which XAL input files (xdxf in particular) described the
> accel lattice from which the model was tracked.
>
> I'll be able to turn more to this next week. Looking forward to it!
>
> Cheers
> Greg
>
>
> On 3 Nov 2011, at 21:32, Shen, Guobao wrote:
>
>> Hi Greg,
>> First of all, thanks for your input.
>> Please see inline.
>>
>> On 11/3/11 9:24 AM, White, Greg wrote:
>>> Hi Guobao,
>>>
>>> As you asked, please find comments on the Twiss Service, Online Calc, and LogScore requirements
>>> docs below. I'll respond to these and then the followup comments later.
>>>
>>> Some global comments.
>>> =====================
>>>
>>> 1) To confirm - you intend the Twiss service to return essentially design model data,
>>> whereas the Calculator Service to return present machine model data - is that right?
>> Yes.
>>> So, I would expect their user interfaces to be completely identical. But you listed only the Twiss
>>> Service as returning Rmat A to B. Suggest you add Rmat A->B to Calc service also.
>> OK, I added R-mat A->B to Calc Service.
>>
>> For the interface to user, yes, it is almost same to access either Twiss Service or online Calc, but not exactly for now.
>> One exception is that: our physicist is asking for 6x6x6 for 2nd order R-mat with the basic 6x6 linear R-mat.
>> But they are not asking for non-linear R-mat from online calc.
>>
>> However, it is worth put a placeholder there in case they change their ideas later.
>>>
>>> 2) You use the construct "lattice/model" a lot. What distinction are you making between
>>> a lattice and a model?
>> A good question. This 2 terms are so close, and we mix to use them often.
>> However, it is better to make it clear. A lattice is an element geometric + magnetic strength.
>> A model is a lattice + simulation code + algorithm used in simulation + result.
>>
>> In another hand, for a lattice, you can have many different models.
>>
>>> What data will you return for each?
>> As described above.
>>>
>>> 3) Both the Twiss and Calc service use the same ERD (as expected),
>> Sorry, what is the ERD?
>>> but I don't see
>>> where in that ERD you record whether a lattice was derived by the Twiss service
>>> (Design model) or the calc service (extant machine model).
>>> If indeed not there yet, I suggest that is added to the "model" table.
>> The Twiss service returns simulation result not only for design model.
>> You can save your online calculation result into your RDB, IRMIS for us, if you wish, and retrieve it later.
>> I DO have a place in the model table for user to capture that info in the format of comment/description.
>> Exactly, I have that place in both lattice table and model table.
>>
>>> 4) What language will you use for the services?
>> For the service itself, it is in C++.
>> The client api is in Python during initial release.
>> The access to RDB is in Python also.
>>
>>> 5) All the requirements documents would be made much clearer if you added use cases.
>> I agree. Now we do have use cases for logscore service, which is in much better shape than my word doc.
>> I will try to revise the the doc for TWISS and online Calc, and add use cases.
>>
>>> That is,
>>> examples of what a user may want to do, what then they would ask the service for (with a precise
>>> description of what they actually type, or a programmer codes), and what precisely they would get
>>> back. For instance, give a complete example of how the Calc Service would be used to implement a
>>> 4 corrector bump in python or matlab. And how the Twiss Service would be used to plot the
>>> design beta in x an y, by z down the machine. Just those 2 examples would explain a lot.
>> Agree.
>>> Specific service comments
>>> =========================
>>>
>>> Twiss Service.
>>> --------------
>>>
>>> Re: Schema ERD:
>>>
>>> Ok cool [in fact this schema looks exactly like a subset of the one I did for LCLS,
>>> just with tables renamed and a few fewer columns, and addition of global orbit params for a ring!].
>> There still has room to improve the schema.
>> Could you please share your schema? Either sql or picture are good.
>>> Some comments on the one in your diagram:
>>>
>>> a) I don’t see where the filename of a “model” is recorded.
>> First, we do not have a "file". The mode name is capture in model_name.
>>> b) Is model.creation_date the time that the model was run by the service?
>> No. It is the date when the data is loaded into RDB.
>> The Twiss service itself does not run simulation inside, so it is hard to say when the data is produced.
>> But we can say when the data is loaded. A user can provide some rough information when it is produced,
>> and record it in model_desc.
>>
>>> c) What's the cardinality of the lattice–element relationship? At slac it’s M:1, so
>>> thick devices can be modeled at more than one place?
>> What is the element? The element in element table, or a physical hardware component?
>> For front, it is 1:1, right? For later, it is M:1.
>>> Specifically at SLAC we
>>> limit it to 1:1 for thin devices, 1:3 for split quads, and 1:(M>=1) for RF cavity
>>> sets (we record the model (twiss and Rmats) at the end of each cavity in a structure).
>>> For SwissFEL I think I'll continue that.
>>>
>>>
>>> Re: Function on Server, item 3:
>>>> Service control. This part response the service initialization (lattice
>>>> configuration) and request from client.
>>> I'm not sure what this means. What is involved in the initialization of the service with respect to
>>> "lattice configuration"?
>> This statement is wrong. For a twiss service, there is nothing to do with lattice configuration.
>>> Re: Function on Server, item 3 con't:
>>>> It requests service to get a particular model/lattice, or a full
>>>> list of model/service. According request from client, it provides simulation data to
>>>> client, and/or save data into IRMIS thru service.
>>> This sentence is the core of the functionality as far as the user in concerned isn't it. Can you give use cases?
>> Your examples say everything. I would like just follow it. :-)
>>> Some comments:
>>> a) Is the interface device centric, element centric, or whole deck centric? That is,
>>> would a user ask for say the Twiss of a device, element, or accelerator? Egs
>>>
>>> get quad:sectiinname:id:twiss PV centric/device centric,
>>> and would return just 1 row of the twiss table.
>>>
>>> Q5:twiss Element name centric,
>>> and would return just 1 row of the twiss table
>>>
>>> nsls2:twiss Accelerator centric, and return all twiss params of
>>> every element in one go.
>>>
>>> I suggest all 3. That's what I'm aiming at for SwissFEL.
>> Agree. A user can choose what he wants, and get back as a NTTable on pvData/pvAccess layer.
>>>
>>> b) How would a user ask for "a full list of model/service."
>>> What is the distinction you're making between a model and a service? Full list of what?
>> A typo. It should be model/lattice. What I am saying is in either model table lattice table, you have many entries.
>> A user wants to get a list of what we have in model table and/or lattice table before retrieve data.
>> It returns a list of, let's say, model name, description and created date, which satisfy his search constrains.
>>
>>> c) The interface to get model data for an element or device or accelerator - whatever - will be
>>> identical whether you're getting it for Design or extant machine (from the calculator service).
>>> So, how will a user make the distinction when asking for design or extant?
>> It is from different services. So the distinction is made in nature.
>> If all are from TWISS server, the model_desc holds the information from the user when he/she loads it.
>>
>>>
>>> For SwissFEL I'll do this by a user interface parameter. That is, a user might ask for:
>>>
>>> Q5:twiss type=design
>>>
>>> d) Re "and/or save data into IRMIS thru service". What data are you expecting it to save. Are you
>>> implying the Twiss service both calculates design models (that is, it does the tracking), and also
>>> responds to user requests for model data?
>> The Twiss service does not do the tracking, and only responds to user request to save or retrieve data.
>> It saves data whatever it gets.
>>
>> The tracking can either be done by user offline, or online calc.
>> The online calc service does not save data directly into IRMIS, it ships data to Twiss service to save it.
>>
>>> If so, I suggest you remove the tracking and data upload
>>> requirement altogether. For SwissFEL, we're going to do the tracking and db upload offline initially.
>>> The service is only going to respond to user queries for data. Later, we will make a service to do
>>> tracking, but it'll be a completely different service executable. That is, the same "service,"
>>> if you like, but 2 completely different services. Same as if you talk to a bank's computers
>>> about your balance or a loan - 2 completely different server side programs will take care of that.
>>>
>>> e) In general I don't see any support for getting model data. Are you planning that a separate rdb
>>> service actually gets model data out of the db for a client?
>> ??? This is the responsibility of Twiss service.
>> Again, Twiss service takes care of data into/from IRMIS, do some basic R-mat A-> B calculation,
>> and dedicates to that.
>> Tracking is done either by a user offline, or online calc.
>>>
>>> Basically, we need Use Cases. Examples of what a physicist ask for precisely, and what is returned
>>> in what form it is returned in. I'll be doing this soon for PSI so I can propose some stuff.
>> Thanks.
>> I suggest we can have a written down doc before we start coding.
>> This is what I did exact for logscore service by having Marty on-site.
>> It helps a lot, and makes us more clear what we really want, and how to do.
>>
>>> Online Calculator Comments.
>>> ---------------------------
>>>
>>> As mentioned above, I don't see in the ERD whether a model is calculated by the Twiss or Calc service.
>> Still, I do not know what is ERD.
>> If you are mentioning a field to record the simulation is done offline or online calc, we have a place to record it.
>>
>>> Can't help thinking a Unit Conversion Service is the wrong place to do unit conversion. We've had that
>>> discussion before. As you know, I'd put that at IOC level, as we do at SLAC. Maybe I'm biased to what's
>>> familiar.
>>>
>>> As mentioned above, at first I'd do the tracking outside the service, or in a different service.
>> Yes, it is my design to have Twiss service, and online calc to separate them.
>>
>>> The first
>>> priority of the "extant model service" is to get the model out of the db, and return it it to a user. An
>>> offline system can track models and put them into the db.
>> Yes, this is the role of Twiss Service.
>>
>>> Again, need Use Cases to illustrate the online calc service functions. Eg you mentioned that your
>>> guys want the 2nd order system as well as 6x6. So, in what
>>> form is that going to be returned? I suggest not making that a priority. Just return 1st order to get going.
>> Yes. We do push it back to our AP guys that we will not do the 6x6x6 R-mat at the first day.
>> I am glad you are in the same with us.
>>
>>> As you know, I wouldn't use a python layer to access the DB. The SQL calls can be agreed with *Don*
>>> beforehand and he can bless them, so just write the straight JDBC or ODBC. But I know you guys don't agree
>>> so I'll shut up.
>>>
>>>
>>>
>>> LogScore Comments
>>> -----------------
>>>
>>> Looks good, but ambitious. Suggest that the first incarnation only does "get" from IRMIS. Leave "put" for later.
>>> That is, leave snapshot taking and upload to a GUI like SCORE to begin with.
>> We are focusing only 2 functions: getSnapshot from IRMIS, and putSnapshot into IRMIS.
>> All others will come later.
>>
>> Guobao
>>
>>>
>>>
>>>
>>>
>>> On 27 Oct 2011, at 16:10, Shen, Guobao wrote:
>>>
>>>> Hi all colleagues,
>>>> As I mentioned in previous email, I have a draft for a couple of services needed by NSLS2 project.
>>>> Since it is very preliminary, I am sending to a small group instead of whole mailing-list.
>>>> One more service for physics-engineering unit conversion is under writing.
>>>> Those services will be under V4 framework.
>>>>
>>>> Since as we discussed during yesterday's meeting, Greg will give a initial definition for NTTable before Monday,
>>>> one topic when Marty is here is to discussion on how to apply NTTable to those services, rise a topic for next
>>>> weekly meeting.
>>>>
>>>> Could you please give your comments?
>>>>
>>>> Guobao
>>>>
>>>> --
>>>> Guobao Shen
>>>> Bldg. 902-B, 17 Cornell Avenue
>>>> National Synchrotron Light Source II
>>>> Brookhaven National Laboratory
>>>> Upton, New York 11973
>>>> Tel. : +1 (631) 344 7540
>>>> Fax. : +1 (631) 344 8085
>>>> http://www.bnl.gov/nsls2
>>>>
>>>> <logscore.docx><online-calculator.docx><twiss.docx>
>>>
>>
>> --
>> Guobao Shen
>> Bldg. 902-B, 17 Cornell Avenue
>> National Synchrotron Light Source II
>> Brookhaven National Laboratory
>> Upton, New York 11973
>> Tel. : +1 (631) 344 7540
>> Fax. : +1 (631) 344 8085
>> http://www.bnl.gov/nsls2
>>
>
> <SWBGB_037_111110411510.pdf>------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
|