BASE

Quoting Carl Troein <carl@...>:

> 
> David Martin writes:
> 
> > One of the problems with deleting data is that it is possible to make
> mistakes
> > on a drastic scale. Physically throwing away an item is a very
> definite process
> > that is checked at many levels. Electronically one can delete a city
> in a
> > microsecond.
> 
> How very true. rm -rf / => Major OOPS.
> 
> > Can we look at actual space requirements? One can have both worlds,
> hiding data
> > and deleting it.
> 
> I'm working on getting the data hiding into as much of BASE as
> possible now. In the version I'm poking around in, it's possible
> to 'delete' and 'undelete' samples, extracts etc. The user can
> choose to see deleted items as well as 'existing' ones.
> If you delete a sample that has extracts, the sample will still
> show up when you look at the extract, but it will be marked as
> deleted. If you delete an extract, it won't show up when you
> look at the sample it comes from, unless you choose to see deleted
> things as well.
> In the future there should be some way to shut down BASE so that
> nothing is writing to the database. Then it'll be possible to
> safely make backups and run a script that removes 'deleted' items
> (after checking that nothing depends on them, of course).

Just to be semantically clear.. We should use 'hidden' rather than 'deleted'
when an item is in the database but not visible. I personally can see no reason
why I would want to actually delete data (some I would be happy to archive). 

I'll be back into BASE hacking in a week or so.. the rest of my life is being a
bit more pressing right now..

..d


> 
> //Carl
> 
> -- 
>  Carl Troein - carl@...
>  http://www.thep.lu.se/~carl/
>  BASE developer - http://base.thep.lu.se
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> basedb-devel mailing list
> basedb-devel@...
> https://lists.sourceforge.net/lists/listinfo/basedb-devel
> 



Dr David Martin
Bioinformatics Scientific Officer
Post-Genomics and Molecular Interactions Centre
University of Dundee
+44 1382 348704

David Martin writes:

> Just to be semantically clear.. We should use 'hidden' rather than 'deleted'
> when an item is in the database but not visible.

OK.

> I personally can see no reason
> why I would want to actually delete data (some I would be happy to archive). 

As long as things can be renamed. Can you imagine being stuck with
a sample called "my boss is a prick" or something like that? ;-)

//Carl - wouldn't use that name for a sample. Possibly.

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se

Quoting Carl Troein <carl@...>:

> 
> David Martin writes:
> 
> > Just to be semantically clear.. We should use 'hidden' rather than
> 'deleted'
> > when an item is in the database but not visible.
> 
> OK.
> 
> > I personally can see no reason
> > why I would want to actually delete data (some I would be happy to
> archive). 
> 
> As long as things can be renamed. Can you imagine being stuck with
> a sample called "my boss is a prick" or something like that? ;-)


As the name is merely a label and the actual identity is the sample id # then
there shouldn't be too much of a problem in hiding your frustrations.. ;-)

regards

..d

> 
> //Carl - wouldn't use that name for a sample. Possibly.
> 
> -- 
>  Carl Troein - carl@...
>  http://www.thep.lu.se/~carl/
>  BASE developer - http://base.thep.lu.se
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> basedb-devel mailing list
> basedb-devel@...
> https://lists.sourceforge.net/lists/listinfo/basedb-devel
> 



Dr David Martin
Bioinformatics Scientific Officer
Post-Genomics and Molecular Interactions Centre
University of Dundee
+44 1382 348704

p.lijnzaad@... writes:

> even though we may not be doing much on BASE anytime soon, I still thought it
> might be useful to share a relatively recent patch of ours, which is fairly
> minimal, but nonetheless important. It concerns the ubiquitous use of
> integers which we think really ought to be reals. The context diff is
> included. If anything, it might provoke some debate on the
> issue.

It'll cost about 24 bytes per row in RawBioAssayData, so the table will
grow by about 20-30%. Not too bad. Some changes might be needed in the
code to reflect that it's float rather than int, but I'm not sure.
I know that there will be some changes needed when we make the
extracted/labeled quantities floats, but since I'm doing lots of
changes there anyway (for hiding, groups, general cleanup), I can
just as well do that at the same time.

//Carl

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se

> I start tackling the problem of group-level access now.
> The solution we have in mind is to make user a subclass
> of group, and have a fixed number of BIGINT columns for
> each Item, where each bit corresponds to read access for
> a user/group. As the number of users+groups is limited
> by the number of such columns, there will be a script that
> can change the number of columns in all sharable subclasses
> of Item.

Mmmm ... this doesn't strike me as the easiest (nor most scalable) solution
(that would be to go postgres all the way and make use of its permission
system on tables ... but I post^H^H^H^Hdigress). I think implementing a
Unix-like permission system on Items would not be so difficult, and much more
flexible. I.e., you have groups and users, and each Item has an ownership and
and a groupownership, as well as permissions (read, write, execute?).

Just a thought,

                                                                      Philip
-- 
Philip Lijnzaad
Genomics Laboratory
Dept. of Biomedical Genetics
University Medical Center (UMC), Utrecht
Stratenum, STR 3.223
P.O. Box 85060, 3508 AB Utrecht
(Universiteitsweg 100, 3584 CG Utrecht)
The Netherlands
tel: +31 (0)30 253 8464
fax: +31 (0)30 253 9035

p.lijnzaad@... writes:

> Mmmm ... this doesn't strike me as the easiest (nor most scalable) solution
> (that would be to go postgres all the way and make use of its permission
> system on tables ... but I post^H^H^H^Hdigress).

Now that BASE works with both My and Postgres, I think it'd be
a slight bit stupid to give up on that. But I guess we could discuss
the pros and cons of different RDBMSs for the rest of eternity
(how long would that be, anyway?) and still not get anywhere. :-0

> I think implementing a
> Unix-like permission system on Items would not be so difficult, and much more
> flexible. I.e., you have groups and users, and each Item has an ownership and
> and a groupownership, as well as permissions (read, write, execute?).

It's equally flexible if you with the unix-like system can aggregate
groups when you want to share something with two groups. If you can't,
the system I proposed is more flexible (if we add write/execute(?)
access). But then there's the point that a unix-like system may be
easier to understand (or explain) and to work with.
I think that if it's likely that users will want to share things with
the people they usually work with and nobody else, a unix-like system
works best. If, on the other hand, people will want to share their
samples/uploads/whatnot with specific people, it doesn't seem enough.
If you and the person you want to share something with aren't in the
same group, you'll have to share with everybody.

Looking at the SQL needed, the unix-like system certainly looks
better. To find things that you have read access to, you'd have
something like
...FROM Item i WHERE (i.owner=$userid OR i.gid IN (<groups the user
  is in>) AND i.groupRead <> 0) ...
as opposed to something like
...FROM Item i WHERE (i.readAccess1 & $mask[0] <> 0 OR
  i.readAccess2 & $mask[1] <> 0 OR ...) ...

The ORs will probably be enough to prevent much optmization in
either case, so it's mostly a question of how complicated/flexible
a system we want.

//Carl

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se

> Now that BASE works with both My and Postgres, I think it'd be
> a slight bit stupid to give up on that. But I guess we could discuss
> the pros and cons of different RDBMSs for the rest of eternity
> (how long would that be, anyway?) and still not get anywhere. :-0

agreed (hence the parentheses :-)

>> I think implementing a
>> Unix-like permission system on Items would not be so difficult, and much more
>> flexible. I.e., you have groups and users, and each Item has an ownership and
>> and a groupownership, as well as permissions (read, write, execute?).

> It's equally flexible if you with the unix-like system can aggregate
> groups when you want to share something with two groups. If you can't,

If two groups need to share stuff, then you have to create a third group
containing the members of both. You can't organize groups hierarchically
(unless you do this explicitly and manually), but in practice that never
seems to be needed in Unix.

> But then there's the point that a unix-like system may be easier to
> understand (or explain) and to work with.  I think that if it's likely that
> users will want to share things with the people they usually work with and
> nobody else, a unix-like system works best. If, on the other hand, people
> will want to share their samples/uploads/whatnot with specific people, it
> doesn't seem enough.  

yes; a special little group containing their collaborators would have to be
created, and the items that are to be shared have to be assigned to that
group. In other words, you'll need to have the equivalent chown and/or chgrp
and chmod in the system. This would not be so bad anyway, because changing
ownership is a request I have seen flitting by in these parts as
well. Cheers,

                                                                      Philip
-- 
Philip Lijnzaad
Genomics Laboratory
Dept. of Biomedical Genetics
University Medical Center (UMC), Utrecht
Stratenum, STR 3.223
P.O. Box 85060, 3508 AB Utrecht
(Universiteitsweg 100, 3584 CG Utrecht)
The Netherlands
tel: +31 (0)30 253 8464
fax: +31 (0)30 253 9035

Quoting p.lijnzaad@...:

> 
> >> I think implementing a
> >> Unix-like permission system on Items would not be so difficult, and
> much more
> >> flexible. I.e., you have groups and users, and each Item has an
> ownership and
> >> and a groupownership, as well as permissions (read, write,
> execute?).
> 
> > It's equally flexible if you with the unix-like system can aggregate
> > groups when you want to share something with two groups. If you
> can't,
> 
> If two groups need to share stuff, then you have to create a third
> group
> containing the members of both. You can't organize groups
> hierarchically
> (unless you do this explicitly and manually), but in practice that
> never
> seems to be needed in Unix.
> 
> > But then there's the point that a unix-like system may be easier to
> > understand (or explain) and to work with.  I think that if it's likely
> that
> > users will want to share things with the people they usually work with
> and
> > nobody else, a unix-like system works best. If, on the other hand,
> people
> > will want to share their samples/uploads/whatnot with specific people,
> it
> > doesn't seem enough.  
> 
> yes; a special little group containing their collaborators would have to
> be
> created, and the items that are to be shared have to be assigned to
> that
> group. In other words, you'll need to have the equivalent chown and/or
> chgrp
> and chmod in the system. This would not be so bad anyway, because
> changing
> ownership is a request I have seen flitting by in these parts as
> well. Cheers,

Heirarchical groups won't work as they soon become impossible.

The alternative is for each item to track its group/user access in a separate
table. Any access rights then have to be looked up in the table if the user is
not the owner.

It does require a join or two but shouldn't be too hard (and is almost
infinitely extendable).

I really don't like the bitmask approach.

..d


Dr David Martin
Bioinformatics Scientific Officer
Post-Genomics and Molecular Interactions Centre
University of Dundee
+44 1382 348704

David Martin writes:

> Heirarchical groups won't work as they soon become impossible.

From an administration point of view or otherwise?
They would give at least one extra database call per request,
as we'd need to find out what parent groups the user's groups
belong to, and then another if any rows were returned and so
on... If people don't use the feature, we'll have a minimum
of 5 rather than 4 db calls per request. (4 calls: kill old
sessions, get session, get user, get groups).
But anyway, since there seems to be little interest in hierarchical
groups, I'll won't bother to implement them.

> The alternative is for each item to track its group/user access in a separate
> table. Any access rights then have to be looked up in the table if the user is
> not the owner.

I don't like the idea of an external table for this, for two reasons.
One is the lack of transactions in MySQL 3 and that even if we did
have them, it'd be more complicated. Also, we'd always need DISTINCT
(or a subselect, I guess) if we allow an item/user pair to be matched
by more than one row.

> I really don't like the bitmask approach.

Neither do I, to be honest. I used it in my photo album, and that
was definitely a design error. I'll try to learn from my past
mistakes, and choose the unix approach instead. I will have to change
some of the code I've written, but that shouldn't take very long
compared to the time it'll take to change the rest of the code
for the new system.

//Carl

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se

p.lijnzaad@... writes:

> If two groups need to share stuff, then you have to create a third group
> containing the members of both. You can't organize groups hierarchically
> (unless you do this explicitly and manually), but in practice that never
> seems to be needed in Unix.

It could be that it's so easy to just email files that you don't
want to make public. Which I don't suppose you'll do here, since
we're not talking very emailable things.

> group. In other words, you'll need to have the equivalent chown and/or chgrp
> and chmod in the system. This would not be so bad anyway, because changing
> ownership is a request I have seen flitting by in these parts as
> well. Cheers,

One thought - it makes no sense to let a user share something with
a group he's not a member of. So if there is a group for every user
(like some linux distros are set up be default), and there is a
group where everyone's always a member, we just need to assign two
groups (reading/writing) to each item. This is really the simplest
possible solution I can think of, and it does seem powerful enough.
Thoughts?

//Carl

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se

(flicking on the supercite bit again) 

On Mon, 16 Sep 2002 15:59:28 +0000 (GMT), 
"Carl" == Carl Troein <carl@...> wrote:


Carl> p.lijnzaad@... writes:

>> If two groups need to share stuff, then you have to create a third group
>> containing the members of both. You can't organize groups hierarchically
>> (unless you do this explicitly and manually), but in practice that never
>> seems to be needed in Unix.

Carl> It could be that it's so easy to just email files that you don't
Carl> want to make public. Which I don't suppose you'll do here, since
Carl> we're not talking very emailable things.


You've lost me here ... 


>> group. In other words, you'll need to have the equivalent chown and/or chgrp
>> and chmod in the system. This would not be so bad anyway, because changing
>> ownership is a request I have seen flitting by in these parts as
>> well. Cheers,

Carl> One thought - it makes no sense to let a user share something with
Carl> a group he's not a member of. So if there is a group for every user
Carl> (like some linux distros are set up be default), and there is a
Carl> group where everyone's always a member, we just need to assign two
Carl> groups (reading/writing) to each item. This is really the simplest
Carl> possible solution I can think of, and it does seem powerful enough.

That does sound right (or at least interesting:-) Some thoughts:
 - It's unusual (might be an issue for users)
 - It seems a richer system than the Unix permissions. E.g., the readers and
   writers of something can be entirely different. Which leads me to:
 - You still need a real Owner (s/he's the one that can decide about the
   read/write group memberships); this real Owner, will it have read/write
   access? 
 - I'm not sure whether it's easier or harder than Unix-style permissions:
   'read-group + write-group' vs. 'group-ownership + access permissions'
 - The number of groups could be come biggish
 - Do you need special 'Unix directory style' permissions for container-like
   things?

Cheers,
                                                                      Philip

-- 
Philip Lijnzaad
Genomics Laboratory
Dept. of Biomedical Genetics
University Medical Center (UMC), Utrecht
Stratenum, STR 3.223
P.O. Box 85060, 3508 AB Utrecht
(Universiteitsweg 100, 3584 CG Utrecht)
The Netherlands
tel: +31 (0)30 253 8464
fax: +31 (0)30 253 9035

p.lijnzaad@... writes:

> Carl> It could be that it's so easy to just email files that you don't
> Carl> want to make public. Which I don't suppose you'll do here, since
> Carl> we're not talking very emailable things.
> 
> You've lost me here ... 

And the reply I just typed. I so dislike this web-based email client
I'm currently using. :-P
Anyway, I was just pondering why you can do without sharing files with
just one user on unix systems: You can transfer the file contents some
other way.

> That does sound right (or at least interesting:-) Some thoughts:
>  - It's unusual (might be an issue for users)

A good enough point, although it would be possible (with a bit
of magic on the application side) to make the systems act almost
the same way, at least for sensible permissions (rw-----w- might
be tricky, but it'd be rather silly to use in real life).

>  - Do you need special 'Unix directory style' permissions for container-like
>    things?

Like the x bit? Hmm...

//Carl

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se

James F. Reid writes:

> I had in mind
> the standard GNU approach using automake, autoconf etc. for getting the
> famous: 
> unpack
> ./configure
> make
> make install
> 
> Two phases seem necessary: 
> 	1) Fix the current package as best I can by writing the appropriate
> (minimal) configure.in and Makefile.am, i.e. get it up and running just
> how the current Makefiles work for X86 machines running linux.
> 	2) The next step would probably take more time and would aim for a more
> standard distribution (i.e. src/ config/ directories etc.) thereby
> affecting (probably) the current layout of the source code, adding more
> hardware platforms and options for the install.
> 
> I think that I could manage 1) whereas 2) will need a lot of
> collaboration.
> 
> Who would be interested in this?
> 
> Could we draw up a list of the platform we could support?  ( I can test
> on Pentium (RH linux), Sparc (Solaris 7) and Alpha (RH Linux).)

1) is an excellent idea, and something I have been missing (as I have
three different setups here). I'd be delighted to see automake in the
next relase.
2) is certainly more work, and most likely worth it. It's something
we'll have to discuss further.

//Carl

-- 
 Carl Troein - carl@...
 http://www.thep.lu.se/~carl/
 BASE developer - http://base.thep.lu.se