exist-development — eXist Developer's List

Re: [Exist-development] missing jars in eXist eclipse project

[Dannes W. <da...@ex...>]

2012/2/16 Chris Tomlinson <chr...@gm...>

> I should have included in the list the ojdbc14.jar - I have that lying
> about and put it in place - downloading that one can not be usefully
> automated, but it should also not be required by the default .classpath
> (there's a note about the jar file in the build.properites so anyone
> wanting to enable the option can deal with updating their local .classpath).
>

Chris , thank you for figuring out which files are missing. I realize that
eclipse cannot deal well with missing JAR files, but it does not provide a
way to really deal with it. In the end, when we switch to Maven, thngs
would be much better.

cheers

Dannes
(netbeans user :-) )

-- 
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] missing jars in eXist eclipse project

[Chris T. <chr...@gm...>]

Hello again,

I should have included in the list the ojdbc14.jar - I have that lying about and put it in place - downloading that one can not be usefully automated, but it should also not be required by the default .classpath (there's a note about the jar file in the build.properites so anyone wanting to enable the option can deal with updating their local .classpath).

Regards,
Chris


On Feb 16, 2012, at 1:48 PM, Chris Tomlinson wrote:

> Hello,
> 
> After enabling all extensions in eXist-2.0.x/extensions/build.properties (but not the eXist-2.0.x/extensions/indexes/build.properties - so the spatial jars didn't get downloaded (if they would have)) and performing:
> 
> ./build.sh rebuild
> 
> various jars are downloaded leaving the following 18 missing jar files:
> 
>> tools/aspectj/lib:
>> 
>> aspectjrt-1.6.12.jar
>> 
>> 
>> extensions/indexes/spatial/lib:
>> 
>> geoapi-nogenerics-2.1-M3.jar
>> gt2*   (7)
>> jsr108-0.01.jar
>> jts-1.8.jar
>> vecmath-1.3.1.jar
>> 
>> 
>> extensions/security/oauth/lib
>> 
>> scribe-1.2.3.jar
>> 
>> 
>> lib/optional:
>> 
>> commons-codec-1.5.jar
>> 
>> 
>> lib/user:
>> 
>> avalon-framework-api-4.3.jar
>> avalon-framework-impl-4.3.jar
>> icu4j-4.2.1.jar
>> saxon9he.jar
> 
> The icu4j-4.2.1.jar is likely a leftover since icu4j-4_8_1_1.jar is present.
> 
> It seems like the jars for the various options such as the spatial index that aren't enabled by default ought probably not be present in the .classpath.
> 
> Thanks,
> Chris
> 
> 
> 
> 
> On Feb 16, 2012, at 1:35 PM, Dannes Wessels wrote:
> 
>> Hi
>> 
>> On Thu, Feb 16, 2012 at 7:21 AM, Dmitriy Shabanov <sha...@gm...> wrote:
>> BTW, some jars was updated to new version without updateting eclipse file.
>> 
>> Dmitriy Shabanov
>> 
>> 16.02.2012 11:05 пользователь "Chris Tomlinson" <chr...@gm...> написал:
>> 
>> Hello,
>> 
>> I just did a fresh check out the stable/eXist-2.0.x branch into an eclipse workspace and see around 30 jars referenced in the .classpath which are not present in the checkout hence showing up as missing items in the buildpath.
>> 
>> The project must be rather complete *but* you might need to run the build.sh script with the right modules enabled in extensions/build.properties once to have all optional jars downloaded.
>> 
>> Please could you provide the list of missing JAR files?
>> 
>> I'll update the build documentation accordingly....
>> 
>> cheers
>> 
>> Dannes
>> 
>> 
>> -- 
>> eXist-db Native XML Database - http://exist-db.org
>> Join us on linked-in: http://www.linkedin.com/groups?gid=35624
> 

Re: [Exist-development] missing jars in eXist eclipse project

[Chris T. <chr...@gm...>]

Hello,

After enabling all extensions in eXist-2.0.x/extensions/build.properties (but not the eXist-2.0.x/extensions/indexes/build.properties - so the spatial jars didn't get downloaded (if they would have)) and performing:

./build.sh rebuild

various jars are downloaded leaving the following 18 missing jar files:

> tools/aspectj/lib:
> 
> aspectjrt-1.6.12.jar
> 
> 
> extensions/indexes/spatial/lib:
> 
> geoapi-nogenerics-2.1-M3.jar
> gt2*   (7)
> jsr108-0.01.jar
> jts-1.8.jar
> vecmath-1.3.1.jar
> 
> 
> extensions/security/oauth/lib
> 
> scribe-1.2.3.jar
> 
> 
> lib/optional:
> 
> commons-codec-1.5.jar
> 
> 
> lib/user:
> 
> avalon-framework-api-4.3.jar
> avalon-framework-impl-4.3.jar
> icu4j-4.2.1.jar
> saxon9he.jar

The icu4j-4.2.1.jar is likely a leftover since icu4j-4_8_1_1.jar is present.

It seems like the jars for the various options such as the spatial index that aren't enabled by default ought probably not be present in the .classpath.

Thanks,
Chris




On Feb 16, 2012, at 1:35 PM, Dannes Wessels wrote:

> Hi
> 
> On Thu, Feb 16, 2012 at 7:21 AM, Dmitriy Shabanov <sha...@gm...> wrote:
> BTW, some jars was updated to new version without updateting eclipse file.
> 
> Dmitriy Shabanov
> 
> 16.02.2012 11:05 пользователь "Chris Tomlinson" <chr...@gm...> написал:
> 
> Hello,
> 
> I just did a fresh check out the stable/eXist-2.0.x branch into an eclipse workspace and see around 30 jars referenced in the .classpath which are not present in the checkout hence showing up as missing items in the buildpath.
> 
> The project must be rather complete *but* you might need to run the build.sh script with the right modules enabled in extensions/build.properties once to have all optional jars downloaded.
> 
> Please could you provide the list of missing JAR files?
> 
> I'll update the build documentation accordingly....
> 
> cheers
> 
> Dannes
> 
> 
> -- 
> eXist-db Native XML Database - http://exist-db.org
> Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] missing jars in eXist eclipse project

[Dannes W. <da...@ex...>]

Hi

On Thu, Feb 16, 2012 at 7:21 AM, Dmitriy Shabanov <sha...@gm...>wrote:

> BTW, some jars was updated to new version without updateting eclipse file.
>
> Dmitriy Shabanov
> 16.02.2012 11:05 пользователь "Chris Tomlinson" <
> chr...@gm...> написал:
>
>  Hello,
>>
>> I just did a fresh check out the stable/eXist-2.0.x branch into an
>> eclipse workspace and see around 30 jars referenced in the .classpath which
>> are not present in the checkout hence showing up as missing items in the
>> buildpath.
>>
>
The project must be rather complete *but* you might need to run the
build.sh script with the right modules enabled in
extensions/build.properties once to have all optional jars downloaded.

Please could you provide the list of missing JAR files?

I'll update the build documentation accordingly....

cheers

Dannes


-- 
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] missing jars in eXist eclipse project

[Dmitriy S. <sha...@gm...>]

BTW, some jars was updated to new version without updateting eclipse file.

Dmitriy Shabanov
16.02.2012 11:05 пользователь "Chris Tomlinson" <chr...@gm...>
написал:

> Hello,
>
> I just did a fresh check out the stable/eXist-2.0.x branch into an eclipse
> workspace and see around 30 jars referenced in the .classpath which are not
> present in the checkout hence showing up as missing items in the buildpath.
>
> In the past I've scrounged around on the internet collecting the various
> jar files via googling and so on.
>
> Is there another location where the referenced versions of the jars is to
> be found or a list of URLs to these jars?
>
> Thanks,
> Chris
>
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>

[Exist-development] missing jars in eXist eclipse project

[Chris T. <chr...@gm...>]

Hello,

I just did a fresh check out the stable/eXist-2.0.x branch into an eclipse workspace and see around 30 jars referenced in the .classpath which are not present in the checkout hence showing up as missing items in the buildpath.

In the past I've scrounged around on the internet collecting the various jar files via googling and so on.

Is there another location where the referenced versions of the jars is to be found or a list of URLs to these jars?

Thanks,
Chris

[Exist-development] New stable branch: 2.0.x

[Wolfgang M. <wol...@ex...>]

To all committers and users: to prepare for the final 2.0 release and
to continue with active development, we have created a new stable
branch in SVN, which will replace the 1.4.x branch:
stable/eXist-2.0.x.

Developers: starting with rev 15855, please decide which commits have
to be ported to 2.0.x and merge them yourselves. It is difficult for
us if we have to go through the commit list months later to find the
changes which have to be merged. As a general guideline, changes
should be ported to the stable branch if they: 1) fix a bug which
affects many users and the impact of the bug fix on other areas of
eXist is limited, 2) fix an issue or improve a feature in code you are
maintaining and which is sufficiently separate from eXist core, e.g.
in an extension, 3) documentation updates, web site fixes. If in
doubt, please send a message to the development list.

Wolfgang

[Exist-development] Wrapper code from Tanuki Software is now at 3.5.13 should it be upgraded from 3.5.11?

[Loren C. <lor...@gm...>]

Does the changes in the wrapper warrant an upgrade from 3.5.11 to 3.5.13 or do we want to wait for 3.5.14?

http://wrapper.tanukisoftware.com/doc/english/release-notes.html

There is an old file wrapper.conf-3.3.9 in tools/wrapper/conf/ that looks like it is referring to the 3.3.9 version.

Begin forwarded message:

> From: Anders Bruun Olsen <ab...@ds...>
> Subject: Re: [Exist-open] Problems installing service on Linux Ubuntu server
> Date: February 9, 2012 9:45:46 AM GMT+01:00
> To: Efraim Feinstein <efr...@gm...>
> Cc: exi...@li...
> 
> I usually grab the newest (community) version of the wrapper from wrapper.tanukisoftware.com since I need a proper 64-bit version, which the included version does not seem to be.
> The newer versions also have the added advantage of supporting upstart (used in Ubuntu instead of sysv init).
> 
> Short instructions for switching to newer wrapper:
> 
> 1. download from wrapper.tanukisoftware.com
> 2. mv $EXIST/tools/wrapper $EXIST/tools/wrapper.old
> 3. unpack wrapper to tools/wrapper
> 4. cp -a tools/wrapper.old/{classes,work} tools/wrapper/
> 5. cp tools/wrapper.old/conf/{log4j.xml,wrapper.conf} tools/wrapper/conf/
> 6. cp tools/wrapper/bin/testwrapper tools/wrapper/bin/exist
> 7. edit tools/wrapper/conf/wrapper.conf and set wrapper.java.command to the correct java bin
> 8. edit tools/wrapper/conf/wrapper.conf and set set.EXIST_HOME to the path to your exist install
> 9. edit tools/wrapper/bin/exist and set APP_NAME="exist" and APP_LONG_NAME="eXist server"
> 9. edit tools/wrapper/bin/exist and set USE_UPSTART="yes" and RUN_AS_USER="yourusername"
> 10. run "tools/wrapper/bin/exist install" to add exist as a service that will start at boot and can be controlled by the service command (it installs a config-file in /etc/init)
> 
> 2012/2/7 Efraim Feinstein <efr...@gm...>
> Hi,
> 
> Debian (and Ubuntu, which is a Debian derivative) have recently switched
> to a newer system of update scripts that use a Makefile-like dependency
> graph to determine which services need to be started first. The eXist
> startup scripts don't have the information.
> 
> The command line I've used to initialize /etc/init.d on Debian is:
> /usr/sbin/update-rc.d existdb start 20 2 3 4 5 . stop 80 0 1 6 .
> 
> I'm pretty sure the scripts work without the LSB header, although you
> will be given the warning you reproduced below until they have an LSB
> header....
> 
> On 02/07/2012 03:27 PM, Pallant, Julie wrote:
> >
> > root@hfdev:f# update-rc.d existdb defaults
> > update-rc.d: warning: /etc/init.d/existdb missing LSB information
> > update-rc.d: see <http://wiki.debian.org/LSBInitScripts>
> >  Adding system startup for /etc/init.d/existdb ...
> >    /etc/rc0.d/K20existdb -> ../init.d/existdb
> >    /etc/rc1.d/K20existdb -> ../init.d/existdb
> >    /etc/rc6.d/K20existdb -> ../init.d/existdb
> >    /etc/rc2.d/S20existdb -> ../init.d/existdb
> >    /etc/rc3.d/S20existdb -> ../init.d/existdb
> >    /etc/rc4.d/S20existdb -> ../init.d/existdb
> >    /etc/rc5.d/S20existdb -> ../init.d/existdb
> >
> > I have created a symbolic link from
> > $EXIST_HOME/tools/wrapper/bin/exist.sh /etc/init.d/existdb
> 
> Another little pointer: I've found that on some systems, the wrapper
> executable ($EXIST_HOME/tools/wrapper/bin/wrapper) needs to be linked to
> the appropriate executable in the same directory, which also needs to be
> chmod-ded +x. (True on 64 bit Debian where you need to symlink
> $EXIST_HOME/tools/wrapper/bin/wrapper-linux-x86-64 to wrapper *or*
> change the wrapper command in exist.sh )
> 
> >
> > Thank you for any insight or pointing to documentation on this problem
> 
> The documentation is http://wiki.debian.org/LSBInitScripts :-)
> 
> Hope this helps,
> 
> --
> ---
> Efraim Feinstein
> Lead Developer
> Open Siddur Project
> http://opensiddur.net
> http://wiki.jewishliturgy.org
> ... and Harvard Physics Department :-)
> 
> 
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
> 
> 
> 
> -- 
> Anders Bruun Olsen
> It-ansvarlig
> Det Danske Sprog- og Litteraturselskab
> (Society for Danish Language and Literature)
> 
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing 
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/_______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Adam R. <ad...@ex...>]

> XQuery processor have three stagers: read, compile & evaluation. My proposal
> to run first stage (read) as SYSTEM after 'x' check & 2nd/3rd as
> authenticated/guest user.
>
> Can we trust to XQuery processor? ... well .... hmmmm .... why not? -)

No No No. User rights escalation in eXist-db is evil, just as it is in
any system, and can easily lead to privilege escalation attacks, and
bugs (we have seen some of those in the past, users magically becoming
DBA/system etc.) I have been phasing out the use of the SYSTEM user in
many places, and now we have the new security model, I think this can
be removed from *almost* everywhere.

I think there is a better way to do this, see my last email...


-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Adam R. <ad...@ex...>]

> Can the xquery processor (the agent) be trusted, to not allow literal output
> of some resource, that the calling user (the princpal) must not see? The way
> I see it, the devil is in /including modules/. Then there are xquery
> functions, that are probably safe - eg. import() when constructing the parse
> tree - and xquery functions that are not safe.

Exactly! If a .xqy main module script just needs 'x' permissions. Well
what does a .xqm library module need to be included in the first.
Probably 'r-x', but this would need to be changed to just 'x' as well.

> Won't this put the xquery processor in charge of enforcing permissions?

I dont want to do that. I want permissions to be enforced by the
database. So I have to refactor so that the database knows what an
XQuery module is and that permissions for execution of that are
handled differently.

> This just my outside view, please excuse, if I am off.
>
> --
> peter



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] Merge in of Security Branch

[Andrzej J. T. <an...@ch...>]

Loren:

> We have been running XQuery jobs by the scheduler.  What were/are the problems that you encountered?

This was a while back....when Adam first deployed the first pieces of the new Security
implementation.  XQuery jobs specified in conf.xml would not run due to a permissions problem, which
if memory serves, had something to do with them being run using guest.

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Adam R. <ad...@ex...>]

> In eXist speak, the request servlet would map to the shell, the xquery
> servlet to the interpreter. A very clear analogy. Java'd be the native
> executable format. Is it like that? If so, it should not be lightly broken.

I think its more likely that we consider the XQuery Parser/Compiler to
be part of the kernel rather than an interpreter, so the Native
executable format could be considered XQuery, as you previously
suggested.

-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Dmitriy S. <sha...@gm...>]

On Thu, Feb 9, 2012 at 5:38 PM, Hungerburg <pc...@my...> wrote:

> Am 2012-02-09 04:32, schrieb Dmitriy Shabanov:
>
>
>> C program for linux is same as xquery script for eXist (visa versa), IMHO.
>>
>>  > So I am now open to the idea of just requiring the 'x' bit to execute
>>  > an XQuery script and not the 'r' bit, however the implementation of
>>  > this is incredibly hard without sacrificing security and seperation of
>>  > concerns.
>>
>> It simple if interpretator check 'x' bit and read script as SYSTEM
>> (including modules)
>>
>
> Dmitriy, as your case is to keep some users of the system from seeing the
> source of certain scripts: it is about confidentiality in a local context,
> unlike Joe, who seems to care about remote visibility only.
>
> Can the xquery processor (the agent) be trusted, to not allow literal
> output of some resource, that the calling user (the princpal) must not see?
> The way I see it, the devil is in /including modules/. Then there are
> xquery functions, that are probably safe - eg. import() when constructing
> the parse tree - and xquery functions that are not safe.
>

XQuery processor have three stagers: read, compile & evaluation. My
proposal to run first stage (read) as SYSTEM after 'x' check & 2nd/3rd as
authenticated/guest user.

Can we trust to XQuery processor? ... well .... hmmmm .... why not? -)

-- 
Dmitriy Shabanov

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Hungerburg <pc...@my...>]

Am 2012-02-09 04:32, schrieb Dmitriy Shabanov:
>
> C program for linux is same as xquery script for eXist (visa versa), IMHO.
>
>  > So I am now open to the idea of just requiring the 'x' bit to execute
>  > an XQuery script and not the 'r' bit, however the implementation of
>  > this is incredibly hard without sacrificing security and seperation of
>  > concerns.
>
> It simple if interpretator check 'x' bit and read script as SYSTEM
> (including modules)

Dmitriy, as your case is to keep some users of the system from seeing 
the source of certain scripts: it is about confidentiality in a local 
context, unlike Joe, who seems to care about remote visibility only.

Can the xquery processor (the agent) be trusted, to not allow literal 
output of some resource, that the calling user (the princpal) must not 
see? The way I see it, the devil is in /including modules/. Then there 
are xquery functions, that are probably safe - eg. import() when 
constructing the parse tree - and xquery functions that are not safe.

Won't this put the xquery processor in charge of enforcing permissions?

This just my outside view, please excuse, if I am off.

-- 
peter

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Hungerburg <pc...@my...>]

Am 2012-02-08 23:03, schrieb Adam Retter:
>
> I just wrote a small bash script and a C program on my Macbook and
> compared the Unix permissions required to execute each, to check.
> The bash script requires both read and execute bits when executed as
> '$ ./hello.sh' and bash cmd requires execute, BUT only requires read
> when executed as '$ bash hello.sh', whilst bash cmd requires execute.
> The C program, only requires the execute bit to execute.

May I explore the unix model? I do not know every detail and simplify a 
lot...

First there is the shell, it wraps the kernel, you use it to operate the 
computer. If you enter "ls" at the prompt, it will interpret this as a 
command and will search PATH for a file, that is called "ls". It will 
then find "/bin/ls", and if this is marked executable it will call to 
the kernel to execute that. The kernel I suppose now has to find out, 
what kind of contents there are in the file. It will have to read the 
first N bytes or the first line and do its magic. If the signature 
matches a process image, it will call the linker to complete it, change 
its persona to the users and hand over execution to it. If the file 
starts with a hashbang #! the kernel will read the name of the 
interpreter from the line, execute that one passing the file name as an 
argument. While the script is running, you will see in "ps" the 
interpreter and the script name. So "./hello.sh" is just a convenient 
shorthand for "/bin/sh hello.sh". One will understand, why a user can 
only run scripts, that she can read.

Only recently, there was this local exploit on linux kernels, where the 
attacker would read some suid binary to determine return addresses of 
functions in order to overwrite them with shell code. Making the command 
NOT readable by the user proved insufficient to prevent the exploit, see 
http://blog.zx2c4.com/749 Update 3.

In eXist speak, the request servlet would map to the shell, the xquery 
servlet to the interpreter. A very clear analogy. Java'd be the native 
executable format. Is it like that? If so, it should not be lightly broken.

-- 
peter

Re: [Exist-development] Merge in of Security Branch

[Loren C. <lor...@gm...>]

We have been running XQuery jobs by the scheduler.  What were/are the problems that you encountered?

On Feb 9, 2012, at 6:34 AM, Andrzej Jan Taramina wrote:

> Adam said:
> 
>> Okay so I have been weighing up the pros and cons of the 'r-x' vs.
>> '--x' requirements on stored XQuery main modules in eXist-db. I have
>> foremost my security hat on, and want to adhere to the Unix security
>> model as that is what eXist-db attempts to implement, and it is a very
>> good model.
> <snip lots of other good stuff>
> 
> What does an XQuery need to be executed by the Quartz Scheduler?  That was broken some time ago (and
> prevented me from moving over to newer releases of trunk).  Is that fixed now?
> 
> Can you set permissions through the ant tasks or when you create/update an xquery in the DB using an
> ANT task?
> 
> Thanks!
> 
> -- 
> Andrzej Taramina
> Chaeron Corporation: Enterprise System Solutions
> http://www.chaeron.com
> 
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing 
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development

Re: [Exist-development] Merge in of Security Branch

[Andrzej J. T. <an...@ch...>]

Adam said:

> Okay so I have been weighing up the pros and cons of the 'r-x' vs.
> '--x' requirements on stored XQuery main modules in eXist-db. I have
> foremost my security hat on, and want to adhere to the Unix security
> model as that is what eXist-db attempts to implement, and it is a very
> good model.
<snip lots of other good stuff>

What does an XQuery need to be executed by the Quartz Scheduler?  That was broken some time ago (and
prevented me from moving over to newer releases of trunk).  Is that fixed now?

Can you set permissions through the ant tasks or when you create/update an xquery in the DB using an
ANT task?

Thanks!

-- 
Andrzej Taramina
Chaeron Corporation: Enterprise System Solutions
http://www.chaeron.com

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Dmitriy S. <sha...@gm...>]

09.02.2012 3:03 пользователь "Adam Retter" <ad...@ex...> написал:
>
> Okay so I have been weighing up the pros and cons of the 'r-x' vs.
> '--x' requirements on stored XQuery main modules in eXist-db. I have
> foremost my security hat on, and want to adhere to the Unix security
> model as that is what eXist-db attempts to implement, and it is a very
> good model.
>
> The argument or not having the 'r' flag on XQuery scripts because they
> may contain sensitive information like usernames and passwords seems
> invalid to me, because sensitive information probably should not be in
> these scripts anyway.
> Arguably there was a time when you had to do this because the eXist-db
> authentication and user management system was not flexible enough; so
> perhaps because you created your own username/password system which
> mapped onto a few simple eXist-db users. This has changed, eXist-db
> now supports ACL's and multiple authentication realms. In fact it is
> this very use-case that prompted the start of all the security changes
> in eXist-db by me.
>
> > If one can consider xquery /the native binary format/ in eXist-db, the
> > model would look a lot more, like what you are used to.
>
> However, the above argument suggested by Peter actually almost
> convinced me that maybe we should just require '--x' for execution of
> XQuery scripts and not 'r-x'.
> However, we would have to be willing to apply the same rule to XSLT
> and XProc, which I think is not perhaps a problem?

It simple to get ever with current messy :-)

> I just wrote a small bash script and a C program on my Macbook and
> compared the Unix permissions required to execute each, to check.
> The bash script requires both read and execute bits when executed as
> '$ ./hello.sh' and bash cmd requires execute, BUT only requires read
> when executed as '$ bash hello.sh', whilst bash cmd requires execute.
> The C program, only requires the execute bit to execute.

C program for linux is same as xquery script for eXist (visa versa), IMHO.

> So I am now open to the idea of just requiring the 'x' bit to execute
> an XQuery script and not the 'r' bit, however the implementation of
> this is incredibly hard without sacrificing security and seperation of
> concerns.

It simple if interpretator check 'x' bit and read script as SYSTEM
(including modules)

> The problem is that eXist-db's internals are somewhat messy,
> and to know if a document is an XQuery document you have to read it
> from the database, reading from the database requires the 'r' flag.

Note: Permissions to read metadata required only.

> So what am I saying, I think this is doable and I will change it to
> just require the 'x' bit, but it will take time to do this correctly
> as much refactoring of eXist-db will have to happen. So please be
> patient...

Ok ;-)

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Joe W. <jo...@gm...>]

Hi Peter,

Thanks for your email.

> Joe, for you the problem only will arise, if your guests have the means
> of printing the source code and, if I understand you fully, if these
> means are readily available over the network, where a webbrowser can
> leach it.

That's right.  I'm satisfied that now, I know how to achieve my goal -
of ensuring users can view .xq-driven webpages without seeing the
source.

My original point was to let Adam know that I had encountered a
problem due to my misunderstanding.  At first I interpreted his
email/the docs to mean that --x was the only permission needed to let
world users execute .xq files, but when I tested this and found .xq
files would not execute unless r-x was set, I was unsure whether this
was intended or not.  I worried (out loud) about the implications of
giving world users read access when I didn't intend for them to be
able to read.  Now I understand the purpose of the +r is not so much
letting the user read a file as letting the system access the file in
order to be able to execute it.

So far smarter people than me have now heard my misunderstanding and
thought process!  My misunderstanding might be an indication that some
additional or more explicit documentation could help.

> Billions of mod-perl, mod-php etc. programmers live in the very same
> situation. If for some reason the web-server is misconfigured, instead
> of a nice webpage the users see source code gibberish. This only rarely
> happens though.

Good to know!

> Is there a way, in a stock installation, apart of exide, of getting eg.
> /apps/myApp/controller.xql served as plain text? That would be bad
> indeed. The rest servlet will not do it, it will try to run the
> procedure, and maybe the error message will leak some secrets...

Not that I know of.  So I think we're okay.

> If I understand Adam correctly, at the moment there is no "native
> executable" format in eXist (java maybe?), everything is requiring an
> interpreter and therefore read access.

Interesting.

> Just like you, I am a little curious, on how ACLs can solve this. Do
> they override unix-like permissions?

I look forward to learning more too about this, as well as umask,
setUID, and setGID.

Cheers,
Joe

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Hungerburg <pc...@my...>]

Am 2012-02-09 00:28, schrieb Joe Wicentowski:
>
> No, it's not that - I don't store usernames or passwords in XQuery
> files.  It's the fact that I simply have no intention of exposing the
> source code of my site to guest users.  My intention is to let guest
> users view webpages generated by my XQuery files.  In my mind the
> permission to view a webpage generated by an XQuery files should be
> distinct from the permission to view the XQuery source code.

Joe, for you the problem only will arise, if your guests have the means 
of printing the source code and, if I understand you fully, if these 
means are readily available over the network, where a webbrowser can 
leach it.

Billions of mod-perl, mod-php etc. programmers live in the very same 
situation. If for some reason the web-server is misconfigured, instead 
of a nice webpage the users see source code gibberish. This only rarely 
happens though.

Is there a way, in a stock installation, apart of exide, of getting eg. 
/apps/myApp/controller.xql served as plain text? That would be bad 
indeed. The rest servlet will not do it, it will try to run the 
procedure, and maybe the error message will leak some secrets...

If I understand Adam correctly, at the moment there is no "native 
executable" format in eXist (java maybe?), everything is requiring an 
interpreter and therefore read access.

Just like you, I am a little curious, on how ACLs can solve this. Do 
they override unix-like permissions?

-- 
peter

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Joe W. <jo...@gm...>]

Hi Adam,

>> Giving guest users access to eXide is unwise.  This issue isn't
>> specific to eXide; it applies equally to letting users execute any
>> code which could run doc() or util:binary-doc() on a w+r resource.
>
> Yes and Yes :-)

:)

> I think my concern is this - your use case if I understand, is that
> you want to keep secure information in XQuery files e.g. usernames and
> passwords. I dont understand why you would do that, perhaps you can
> explain what you are trying to do in general terms, I think there may
> be other solutions than the approach that you took maybe.

No, it's not that - I don't store usernames or passwords in XQuery
files.  It's the fact that I simply have no intention of exposing the
source code of my site to guest users.  My intention is to let guest
users view webpages generated by my XQuery files.  In my mind the
permission to view a webpage generated by an XQuery files should be
distinct from the permission to view the XQuery source code.

I'm absolutely in favor of your security-minded approach, and I think
it's wise to model eXist's security on best practices.  I also am not
an expert in the unix security model or ACLs, masks, etc.  If there's
a way to achieve what I described above, I'll be happy!

Joe

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Adam R. <ad...@ex...>]

Okay so I have been weighing up the pros and cons of the 'r-x' vs.
'--x' requirements on stored XQuery main modules in eXist-db. I have
foremost my security hat on, and want to adhere to the Unix security
model as that is what eXist-db attempts to implement, and it is a very
good model.

The argument or not having the 'r' flag on XQuery scripts because they
may contain sensitive information like usernames and passwords seems
invalid to me, because sensitive information probably should not be in
these scripts anyway.
Arguably there was a time when you had to do this because the eXist-db
authentication and user management system was not flexible enough; so
perhaps because you created your own username/password system which
mapped onto a few simple eXist-db users. This has changed, eXist-db
now supports ACL's and multiple authentication realms. In fact it is
this very use-case that prompted the start of all the security changes
in eXist-db by me.

> If one can consider xquery /the native binary format/ in eXist-db, the
> model would look a lot more, like what you are used to.

However, the above argument suggested by Peter actually almost
convinced me that maybe we should just require '--x' for execution of
XQuery scripts and not 'r-x'.
However, we would have to be willing to apply the same rule to XSLT
and XProc, which I think is not perhaps a problem?

I just wrote a small bash script and a C program on my Macbook and
compared the Unix permissions required to execute each, to check.
The bash script requires both read and execute bits when executed as
'$ ./hello.sh' and bash cmd requires execute, BUT only requires read
when executed as '$ bash hello.sh', whilst bash cmd requires execute.
The C program, only requires the execute bit to execute.


So I am now open to the idea of just requiring the 'x' bit to execute
an XQuery script and not the 'r' bit, however the implementation of
this is incredibly hard without sacrificing security and seperation of
concerns.The problem is that eXist-db's internals are somewhat messy,
and to know if a document is an XQuery document you have to read it
from the database, reading from the database requires the 'r' flag.

So what am I saying, I think this is doable and I will change it to
just require the 'x' bit, but it will take time to do this correctly
as much refactoring of eXist-db will have to happen. So please be
patient...

And before someone points out that you 'only used to' need 'x' (or
'u') in eXist-db to execute a script, so solving the problem should
not be hard, I should pre-empt and point out that eXist-db used to
have several security holes; most of which I hope are now plugged ;-)





-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Adam R. <ad...@ex...>]

>> I don't think changing the security concerns here is a good idea. Rather I think the issue lies elsewhere, let me think about this...
>
> Right, I see your point.  You're saying that the issue here could be seen as:
>
> Giving guest users access to eXide is unwise.  This issue isn't
> specific to eXide; it applies equally to letting users execute any
> code which could run doc() or util:binary-doc() on a w+r resource.

Yes and Yes :-)

> You're saying that we need to start realizing that permissions now
> dictate whether the the *system* can read/write/execute resources on a
> given user's behalf, not whether the *user* can read/write/execute
> resources.  The system is now an explicit intermediary between the
> user and resources.  The system is the user's agent in
> reading/writing/executing resources.
>
> Is this a correct way of articulating the new permissions framework?

Kind of, but maybe too strict.

I think my concern is this - your use case if I understand, is that
you want to keep secure information in XQuery files e.g. usernames and
passwords. I dont understand why you would do that, perhaps you can
explain what you are trying to do in general terms, I think there may
be other solutions than the approach that you took maybe.

> Joe



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Casey J. <cas...@jo...>]

So I am confused, does that mean a user would be able to execute a script
without read access because the system/interpreter would be executing so it
would have read access?

Regardless I tend to agree with Dimitriy, in the fact that forcing the
source of an XQuery to be exposed just for execution is certainly a
security vulnerability. It makes sense in Unix but not really in eXist.

Cheers,

Casey

On Wed, Feb 8, 2012 at 1:19 PM, Hungerburg <pc...@my...> wrote:

> Am 08.02.2012 16:40, schrieb Joe Wicentowski:
> >
> > You're saying that we need to start realizing that permissions now
> > dictate whether the the *system* can read/write/execute resources on a
> > given user's behalf, not whether the *user* can read/write/execute
> > resources.  The system is now an explicit intermediary between the
> > user and resources.  The system is the user's agent in
> > reading/writing/executing resources.
>
> How about that: The system, when executing/acting on behalf of a user,
> becomes her agent. The agents permissions are restricted by the
> principals, ie. users permissions.
>
> Therefore, in the unix model, interpreted scripts have to be readable,
> because they are not executed, instead the interpreter is executed,
> which then processes the text of the script.
>
> The exception are set-uid and set-gid executables, where the system dons
> the permissons of a specified other user/group, when acting for some
> user. It is NEVER a good idea to make something interpreted setuid!
>
> If one can consider xquery /the native binary format/ in eXist-db, the
> model would look a lot more, like what you are used to.
>
> --
> peter
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>



-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

Re: [Exist-development] [HEADS-UP] Merge in of Security Branch

[Hungerburg <pc...@my...>]

Am 08.02.2012 16:40, schrieb Joe Wicentowski:
>
> You're saying that we need to start realizing that permissions now
> dictate whether the the *system* can read/write/execute resources on a
> given user's behalf, not whether the *user* can read/write/execute
> resources.  The system is now an explicit intermediary between the
> user and resources.  The system is the user's agent in
> reading/writing/executing resources.

How about that: The system, when executing/acting on behalf of a user, 
becomes her agent. The agents permissions are restricted by the 
principals, ie. users permissions.

Therefore, in the unix model, interpreted scripts have to be readable, 
because they are not executed, instead the interpreter is executed, 
which then processes the text of the script.

The exception are set-uid and set-gid executables, where the system dons 
the permissons of a specified other user/group, when acting for some 
user. It is NEVER a good idea to make something interpreted setuid!

If one can consider xquery /the native binary format/ in eXist-db, the 
model would look a lot more, like what you are used to.

-- 
peter